Mend Remediate and Renovate

Overview

Mend Remediate automatically opens fix Pull Requests for vulnerable open-source components, upgrading them to the lowest non-vulnerable version. Mend Remediate is part of Mend Developer Integrations and integrated with Bitbucket Cloud, Bitbucket Data Center, Bitbucket Server, Github.com, GitHub Enterprise, GitLab and Azure Devops. Providing additional capabilities for project dependency health, Remediate is integrated with Mend Renovate (see below for details). Renovate enables you to save time and reduce risk by automating dependency updates in software projects. 

Mend Remediate provides the following capabilities:

• Real-time monitoring - Runs continuously to detect the latest available fixes.

• Single-click fix - Generate automated fix Pull Requests on-demand or using custom workflow rules.

• Custom workflow rules - Open fix Pull Requests based on vulnerability severity or CVSS score.

• Annotated - Changelogs and commit histories are extracted and included when available.

• Dependency file discovery - Discovers and processes all dependency files in a repository.

• Lock-file support - Update lock files seamlessly in the same commit, including immediately resolving conflicts whenever PRs are merged.

NOTE: A Pull Request fix is only generated for security vulnerabilities discovered on your repo's base branches, and it includes transitive dependencies in Maven, Gradle, npm and pip.

Supported Package Managers

NOTE: In Mend for GitLab, mirrored repositories are not supported.

Remediate Configuration

Starting from version 24.10.1, Remediate jobs use all the same configuration options that are available to the Renovate-CLI.

Note: Previous versions of Remediate parsed only a handful of selected Renovate-CLI configuration options.

Priority and Behavior of Configuration options

Values found at each level will override values found at the previous level.

1. Default config

2. Global config - i.e. all container configurations (environment variables, config.js, CLI arguments)

3. Remediate / Repo Integrations configuration (see docs)

Note: These values are passed to the renovate-cli via the RENOVATE_CONFIG envornment variable and will override any user-defined value for RENOVATE_CONFIG.

4. Inherited config - i.e. Org, Group (see docs)

5. Repository config - i.e. renovate.json file inside the repo

Architecture

Remediate workers run the Renovate-CLI as a separate sub-process for each repository scan (job) inheriting all environment variables of the parent process.

Mend Workflow Rules

Mend Remediate is configured via custom Workflow Rules. These can be set in one of two ways,

Method 1: Set up Workflow Rules for all or a selected list of Mend Products/Projects in the Web Application

1. Open the Mend web application, and click the Integrate tab.

2. Expand the Developer Integrations option.

3. From within the relevant repo integration settings, click Manage Workflow Rules. The Workflow Rules page is displayed.

4. Click Add Rule. The Add Rule dialog box is displayed.
   

   

5. Select a Product and/or Project scope from the Scope area or leave at the default (applies to all of your Mend Products and Projects).

6. Select a rule type from the Type dropdown menu.

7. Click OK to create the rule.

Method 2: Set Workflow Rules in the repo integration configuration file

1. Each repo integration has configurations in the .whitesource file.

2. One of the sections in the configuration file contains the remediateSettings parameters.

3. As an example of this section, see the Bitbucket Cloud configuration.

Once you set up Workflow Rules, Mend Remediate will begin monitoring your repositories for vulnerable dependencies and generate corresponding fix Pull Requests.

Configuration of GitHub.com Credentials

If you are running Remediate against GitHub.com already, or making use of Mend for github.com, then you don’t need to provision credentials explicitly.

Regardless of which platform you are running on (GitHub.com or GitHub Server), Remediate can locate and embed open source Release Notes from projects hosted on github.com into Pull Requests. When Renovate functionality is enabled, Remediate may also need to fetch lists of tags and releases from GitHub.com for other reasons (such as to check if a particular open-source package has any newer versions).

However, GitHub heavily rate limits any unauthenticated API requests to github.com, so if credentials are not available then Remediate will disable Release Notes fetching as well as github-tags, github-releases, and go datasources to ensure predictable behavior. Remediate will log a WARN message on startup if such credentials are missing.

For other use cases (including self-hosted GitHub Enterprise Server), to provide such credentials to the Remediate container, you should configure a github.com Personal Access Token into the environment variable GITHUB_COM_TOKEN. This token can belong to any github.com account and does needs only public read-only permissions. For details on creating such a token, click here. 

Mend cannot provide customers with a token for accessing github.com as that would go against that platform’s Terms of Service.

Providing the Integration Activation Key

Mend Remediate can be configured to work with the integration activation key using either the prop.json configuration file or using environment variables.

NOTE: The environment variable takes precedence over an activation key value specified inside a prop.json file. 

Providing the Integration Activation Key in prop.json

You can provide the integration's activation key to the Remediate container using a prop.json file.

Providing the Integration Activation Key using Environment Variables

You can provide the integration activation key by using the WS_ACTIVATION_KEY environment variable inside the Remediate container.

Proxy Support

Mend Remediate can be configured to work with proxy servers using either the prop.json configuration file or using environment variables.

Proxy Configuration in prop.json

When configuring proxies using prop.json, the "no proxy" list will be automatically configured to match the VCS endpoint in use (i.e. the GitHub, GitLab or Bitbucket Server endpoint). Internally Remediate will convert these to HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables and pass them to child processes such as npm and composer.

Proxy Configuration Using Environment Variables

Mend Remediate also supports the industry convention of HTTP_PROXY, HTTPS_PROXY and NO_PROXY. This provides more flexibility if you need to also configure any internal/private registries in the no proxy list so is the recommended configuration approach. Such variables will be passed transparently to child processes.

Automated Dependency Updates for Improved Dependency Health

Although remediating vulnerabilities should be the highest priority for dependency updating, it is recommended to adopt a proactive dependency update approach using Remediate’s “Renovate” capabilities.

mend’s Renovate capabilities bring automated dependency updates to mend’s repository integrations. By enabling Renovate, more than 50 package manager formats can be detected automatically and dependencies within updated, including numerous Infrastructure as Code managers such as Terraform and Ansible.

The key advantages of automating dependency updates are:

• Vulnerability fixes are often discretely released days or weeks prior to public disclosure, and you may be lucky enough to frequently update using Renovate prior to disclosure and avoid a vulnerability notification altogether

• Even if projects are not completely up-to-date, being reasonably up-to-date means that applying vulnerability remediations is much lower risk and therefore can be done more quickly, lowering the average time-to-resolution for CVEs

• For highly nested dependencies, such as Containers and Infrastructure as Code where vulnerable components may be deeply transitive, staying up-to-date (e.g. with Docker base image updates) is one of the best ways to stay secure anyway

Integration with Mend Renovate

Mend Renovate functionality can be enabled in Mend Remediate via an option in the .whitesource configuration file.

With Renovate functionality enabled, Remediate will then raise PRs/MRs not only for vulnerable dependencies but also for outdated dependencies too.

Renovate is designed for users who aim to keep their dependencies up-to-date, primarily for the reasons of:

• Keeping up with all bug-fixes from open source libraries

• Reducing the risk of updating vulnerable versions in a hurry, because small jumps in versions are easier to test than large

Renovate itself is well-documented via its own documentation site. The Renovate docs are automatically generated based off the open source Renovate repository, so are unnecessary to duplicate here in full, however examples and clarifications will be provided below specifically for users of Mend Remediate.

Renovate Configuration

Renovate supports a range of different filenames, however for Mend Remediate integration only the .whitesource configuration file is used. Renovate is disabled by default so enabled by setting remediateSettings.enableRenovate = true like so:

{
  "remediateSettings": {
    "enableRenovate": true
  }
}

Any configuration you place within the remediateSettings object will be used by Remediate similarly to how Renovate usually uses a renovate.json configuration file. Therefore, if you have an existing renovate.json file in a repository but are migrating to Mend Remediate then you'd copy paste its contents inside remediateSettings before removing the renovate.json file.

Recommended Starting Config

If you are new to Renovate, we recommend you start with the config:recommended preset config, which enables functionality such as maximum two dependency update PRs per hour as well as grouping well-known monorepo dependencies together:

{
  "remediateSettings": {
    "enableRenovate": true,
    "extends": ["config:recommended"]
  }
}

What Happens When Remediations Are Available

It's useful to understand how Remediate works if you have Renovate enabled with updates available while vulnerability fixes are also required at the same time. If a package dependency needs vulnerability remediation, then Remediate reads the list of fixed versions and determines the minimum version that solves all known CVEs. It then uses a package rule to enforce that the remediation PR/MR for that package does not propose to update to the latest and instead updates to the lowest version that can resolve all open CVEs for the package. If an existing PR/MR exists proposing an update to the latest version but the minimum fix is lower, the existing PR/MR may be auto-closed by Remediate and replaced with the minimum fix PR/MR instead. This is intentional so that (a) teams can focus on the high-priority remediation PR/MR, and (b) remediations cause the least changes possible so that they are simpler to verify and test.

Reasons Remediate PR cannot be automatically created despite remediation being available

Reasons Remediation isn’t possible for a vulnerability

Common Configuration Options and Examples

Renovate is highly configurable with many options, so here are the most commonly used ones to take a look at first:

• schedule: use this is you want to raise dependency update PRs during limited times, such as weeknights and weekends. Combine with timezone so that schedules can be written in your local time instead of UTC.

• packageRules with groupName: matching multiple packages and assigning a groupName will mean they will be grouped into a single PR. This can be useful if certain packages must always be upgraded together, or if you wish to reduce the total number of PRs.

• assignees and reviewers: use these fields to configure who is assigned or requested to review PRs (note: will apply to remediation PRs/MRs too)

• automerge: enable this (ideally using packageRules) on dependencies you trust to be merged automatically by Remediate once they pass tests.

If you have questions about any particular configuration option or are looking for examples of its use, you may find it useful to search existing Issues in Renovate's Config Help Repository on GitHub. 

Merge Confidence

For more information on Merge Confidence, see here.

Merge Confidence is a beta feature for Renovate which embeds visual "badges" in Pull Requests to show statistics on upgrade age, adoption, compatibility and ultimately "confidence". 

Supported Languages are listed here.

To enable Merge Confidence for Renovate within Mend Remediate, add the configuration within the remediateSettings section of your .whitesource file, as specified here.

Note: The above configuration is supported only in version 21.2.2 or later of the Repo integration.

If you are using version 21.2.1 or earlier of the Repo integration, add the following configuration within the remediateSettings section of your .mend file:

{
  "remediateSettings": {
    "enableRenovate": true,
    "prBodyDefinitions": {
      "Age": "![age](https://badges.renovateapi.com/packages/npm/{{replace '/' '%2f' depName}}/{{{toVersion}}}/age-slim)",
      "Adoption": "![adoption](https://badges.renovateapi.com/packages/npm/{{replace '/' '%2f' depName}}/{{{toVersion}}}/adoption-slim)",
      "Passing": "![passing](https://badges.renovateapi.com/packages/npm/{{replace '/' '%2f' depName}}/{{{toVersion}}}/compatibility-slim/{{{fromVersion}}})",
      "Confidence": "![confidence](https://badges.renovateapi.com/packages/npm/{{replace '/' '%2f' depName}}/{{{toVersion}}}/confidence-slim/{{{fromVersion}}})"
    },
    "packageRules": [
      {
        "datasources": [
          "maven", "npm", "pypi"
        ],
        "updateTypes": [
          "patch",
          "minor",
          "major"
        ],
        "prBodyColumns": [
          "Package",
          "Change",
          "Age",
          "Adoption",
          "Passing",
          "Confidence"
        ]
      }
    ]
  }
}

Remediate Worker Horizontal Scalability

To scale Remediate to allow it to utilize additional containers to process multiple repositories concurrently, you can enable Remediate Worker Horizontal Scalability. In this mode, the Remediate “worker” logic (which processes repositories) is separated from the Remediate “server” logic (scheduler, job queue, and webhook handling) in a many-to-one relationship.

The same Remediate Docker image is used for both Server and Worker functionality, as they are differentiated/configured using environment variables.

If you have already been running Remediate, you can keep that existing node as the new “Remediate Server”. All that is needed is to pass it the environment variable REMEDIATE_SERVER_ONLY: 'true', and that will be enough for it to know it should be in server-only mode and not run any worker jobs itself. This container will still be the one that the W4D controller needs to reach, e.g. to pass on webhooks.

Next, you should configure one or more Remediate Worker containers. To do so, start up a Remediate image and configure the environment variable REMEDIATE_SERVER_URL to point to the above Remediate Server’s API.

Here is a simple example of two Worker containers and one Server container using Docker Compose syntax:

remediate-server:
  image: wss-remediate
  restart: always
  ports:
    - '8080:8080'
  volumes:
    - './conf/:/etc/usr/local/mend/conf/'
  environment:
    REMEDIATE_SERVER_ONLY: 'true'
remediate-worker:
  image: wss-remediate
  scale: 2
  depends_on:
    - remediate-server
  restart: always
  volumes:
    - './conf/:/etc/usr/local/mend/conf/'
  environment:
    REMEDIATE_SERVER_URL: http://remediate-server:8080

Mend recommends using the R/R status API to monitor the jobs.queueLength property over time to determine and track R/R worker scaling efficiency.

curl http://localhost:8080/status | jq

Use the following jq command to extract just the queueLength value from the status API response:

curl http://localhost:8080/status | jq .status.jobs.queueLength

Server Response Example:

{
    "status": {
        "bootDate": "2023-11-28T11:27:22.808Z",
        "scheduler": {
            "platform": "github.com",
            "lastScheduling": null
        },
        "webhooks": {
            "lastWebhookReceived": "2023-11-29T13:01:15.842Z"
        },
        "jobs": {
            "lastEnqueueDate": "2023-11-29T13:00:49.821Z",
            "lastJobDispatchDate": "2023-11-29T13:00:49.873Z",
            "lastJob": {
                "reason": "remediate",
                "repository": "TriarqHealthOTC/QInsightFlaskRest",
                "uniqueKey": "TriarqHealthOTC/QInsightFlaskRest",
                "priority": 8,
                "logContext": "5cb408c45bb4492796d6b6df2b5f44ff",
                "params": {
                    "collectionId": "14642459",
                    "repoId": "193862761"
                },
                "installationId": "14642459"
            },
            "lastJobFinished": {
                "reason": "remediate",
                "repository": "TriarqHealthOTC/QInsightFlaskRest",
                "uniqueKey": "TriarqHealthOTC/QInsightFlaskRest",
                "priority": 8,
                "logContext": "5cb408c45bb4492796d6b6df2b5f44ff",
                "params": {
                    "collectionId": "14642459",
                    "repoId": "193862761"
                },
                "installationId": "14642459",
                "finished": "2023-11-29T13:01:01.850Z"
            },
            "queueLength": 0,
            "totalJobsProcessed": 57220
        },
        "jobsInProgress": [],
        "worker": {
            "remediateServerEnabled": false,
            "previousJob": null,
            "previousJobStart": null,
            "currentJob": null,
            "currentJobStart": null
        }
    }
}

Note: The json response from the R/R status API from the server container will have values for the scheduler, webhooks, and jobs blocks but will have nulls for the worker block.

Worker Response Example:

{
    "status": {
        "bootDate": "2023-11-28T11:28:41.957Z",
        "scheduler": {
            "lastScheduling": null
        },
        "webhooks": {
            "lastWebhookReceived": null
        },
        "jobs": {
            "lastEnqueueDate": null,
            "lastJobDispatchDate": null,
            "lastJob": null,
            "lastJobFinished": null,
            "queueLength": 0,
            "totalJobsProcessed": 0
        },
        "jobsInProgress": [],
        "worker": {
            "remediateServerEnabled": true,
            "remediateServerUrl": "http://ghc-remediate-server:8080",
            "previousJob": {
                "reason": "remediate",
                "repository": "kryonsystems/node-monorepo",
                "uniqueKey": "kryonsystems/node-monorepo",
                "priority": 8,
                "logContext": "7acba7d03ae04b1c912e93a8151920ee",
                "params": {
                    "collectionId": "13190091",
                    "repoId": "279922596"
                },
                "installationId": "13190091"
            },
            "previousJobStart": "2023-11-29T12:59:36.176Z",
            "currentJob": {
                "reason": "master-commit",
                "repository": "id-fabrik/tidy-frontend-shared",
                "uniqueKey": "id-fabrik/tidy-frontend-shared",
                "priority": 70,
                "logContext": "27d439341dc1415ab0c7e092b5548895",
                "params": {},
                "installationId": 31502381
            },
            "currentJobStart": "2023-11-29T13:02:14.228Z"
        }
    }
}

Note: The json response from the R/R status API from the worker container will have nulls for the scheduler, webhooks, and jobs blocks but will have values for the worker block.