Renovate EE - Job processing in Renovate

Job processing in Renovate is based on a queueing mechanism. Jobs are picked up by Renovate workers from a centralized queue and processed in a serial manner by each worker. The queue is kept in a database managed by the Renovate server(s). For single-server deployments, Renovate defaults to using an in-memory SQLite database; however, for increased reliability, it is recommended to set up a PostgreSQL database for this purpose. Note that this is a requirement for multi-server deployments. It is also recommended that this database be provisioned as a Relational Database Service (RDS)

Jobs

Jobs are generated by various sources:

Webhooks

These are event messages triggered from the SCM platform whenever an action is taken. Renovate reacts to each of the following:

Meaningful modifications to package manager manifest files (e.g., package.json, pom.xml, etc.)
Meaningful modifications to Renovate’s repo configuration files (e.g., renovate.json)
Merging a Renovate PR
Closing Renovate PRs or deleting Renovate branches
Adding the Renovate app (bot) to a repo
Manual creation or rebasing of PRs from the Dependency Dashboard
A manual run triggered from the Dependency Dashboard

Webhook message delivery is not guaranteed in most SCM systems. They work in a “fire-and-forget” manner

Renovate scheduled tasks

These are job-generating actions proactively driven by Renovate and controlled by a set of configured schedules. Due to the unreliable nature of webhooks, Renovate scheduled tasks allow it to behave as a self-healing solution – I.e., even if a webhook is missed for any reason (e.g., Renovate wasn’t running at the time, etc.), Renovate guarantees that repos are eventually processed. There are two different types of scheduled tasks:

Platform synchronization

In charge of generating jobs to ensure Renovate is up to date with the SCM platform as it relates to Renovate app (bot) installations – i.e., the listing of repos that the Renovate bot has been given access to.

It is controlled by the MEND_RNV_CRON_APP_SYNC environment variable (defaults to every four hours) [doc ref]. The variable needs to be set on the Renovate server(s).

Job synchronization

In charge of generating jobs to ensure Renovate is up to date with repo-related tasks. These can include:

Updating PRs if newer artifact versions are available
Processing meaningful changes to package manager manifest files
Processing meaningful changes in Renovate configuration files or presets
Performing automatic rebasing of PRs if needed
Creating new PRs when hourly limits (prHourlyLimit) were priorly reached
Creating new PRs if the number of concurrent PRs drops below the configured limit (prConcurrentLimit)

The job synchronization task is driven by four separate schedules. For each schedule, only repos that match a given state or activity level are processed. This allows for an overall reduced level of Renovate activity. The schedules are controlled by setting the environment variables below on the Renovate server(s):

Hot

MEND_RNV_CRON_JOB_SCHEDULER_HOT (Defaults to every one hour)
This is the higher frequency schedule and applies to repos that are more active. A repo’s activity is assessed by evaluating if the repo has accepted (merged) a Renovate suggested update PR.

Cold

MEND_RNV_CRON_JOB_SCHEDULER_COLD (Defaults to once a day)
This schedule applies to:
- Repos that have been onboarded (accepted onboarding PR or have a manually-added renovate.json config) but have not yet accepted (merged) any Renovate suggested update PRs
- Repos that have an onboarding PR waiting to be merged
- Repos that have failed a job execution for anything other than resource limits (see Capped schedule below)

Capped

MEND_RNV_CRON_JOB_SCHEDULER_CAPPED (Defaults to once a week)
This schedule applies to:
- Repos (likely large ones) where Renovate encounters compute resource constraints that lead to no results. Examples of these constraints include repos that, while being processed, may result in out-of-memory errors, insufficient disk-space errors, or long-time processing leading to timeouts.
- A repo falling in this group will be moved back to a more active schedule (Cold or Hot) once it runs once without encountering resource constraint limits.
- Blocked repos
Note that resource-related job failures can be addressed by making adjustments to infrastructure resources or by modifying the repo Renovate configuration to reduce the scope of the scan

All

MEND_RNV_CRON_JOB_SCHEDULER_ALL (Defaults to once a month)
This schedule applies to all onboarded repos
As part of the self-healing behavior of Renovate, this schedule ensures that there is “catch-all” guaranteed time when all relevant repos will be processed

API calls

Renovate’s API provides the following endpoints that will result in jobs being added to the queue:

POST /api/sync: Force platform synchronization task
POST /api/job/add: Job enqueuing

Throttling

For busy repositories, the processing of jobs can result in many PRs being created by Renovate at one time (e.g., throughout a developer’s workday), which can be overwhelming. To address this, Renovate provides the following job throttling mechanisms:

Concurrency limits

A limit can be configured via prConcurrentLimit (defaults to 10) to keep at most a given number of concurrent active Renovate PRs. This is essential to prevent a buildup of active PRs whenever a repo is processed by Renovate

Once this limit is reached, no additional PRs will be created until existing PRs are accepted (merged), renamed, or deleted. When the PR count drops below the limit, new PRs will be created only when:

One of the schedules described above triggers and the repo is part of it, therefore getting processed
A webhook event is sent (as described above) that triggers the processing of the repo
PR creation is manually triggered from the Dependency Dashboard

New PRs are created until the hourly limit (see below) is hit, or the concurrency limit is hit, whichever comes first

Time limits

Hourly limits

A limit can be configured via prHourlyLimit (defaults to 2) to create at most a given number of Renovate PRs within the hour during which the repo is being processed. This will effectively result in drip-feeding of PRs

Once the hourly limit is reached, no additional PRs are created within the hour. At the next hour, if the given repo is part of one of the schedules described above and if the schedule is matched to run during that hour, then additional PRs will be created until the hourly limit (or the concurrency limit) is reached again.

If no schedule is matched at the next hour, no additional PRs will be created unless:

There is a webhook event (as described above) that triggers the processing of the repo
PR creation is manually triggered from the Dependency Dashboard

Schedules

Frequently, developers find it convenient to configure Renovate to generate updates only during less disruptive times. Renovate provides this capability via the schedule configuration directive.

For example:

CODE

{
  "schedule": ["* 22-23,0-4 * * 1-5"]
}

(this schedule triggers after 10pm and before 5am every weekday)

A schedule can be defined within the context of a packageRule. This gives great flexibility to fine-tune the type of updates that should be constrained by it.

A defined schedule is honored only if the repo that defines it is processed within the time window defined by the schedule. This means that any jobs originated via webhooks (which match the context of the schedule) will be placed on hold (schedule pending in the Dependency Dashboard) until the time arrives that matches the defined schedule (or until the checkbox in the Dependency Dashboard is clicked to force the generation of the updates)

When the schedule time window arrives, the processing of the generated jobs is not guaranteed unless there is a task that results in the repo being processed during the schedule window. Specifically, if any of the following events trigger during that time frame:

A job synchronization schedule (that the repo is part of)
An SCM webhook that is related to the repo

Despite one of the events above triggering, a defined schedule may not be fully honored by Renovate – i.e., no PRs at all or not all expected number of PR may get created – due to one of the following reasons:

If the number of repos to be processed by Renovate is large enough that a given repo never gets a chance to be reviewed during the window of time defined in the job synchronization schedules
If PR concurrency limits have been reached during the schedule time
If PR hourly limits are reached during the schedule time, the logic described above applies. If the defined schedule spans multiple hours, new PRs will be created each hour provided that at least one of the events mentioned above triggers every hour
A used dependency has recently gotten a new update that results in many new PRs, thereby causing PR limits to be exceeded