Self-Managed Repository Integrations - Graceful Container Shutdown

While using Mend’s self-hosted repo integrations, you may need to shut down a container to perform maintenance such as prop.json changes, version updates, or more. One of the main concerns when planning for this is whether the containers will finish scheduled jobs before shutting down.

Container behavior

The controller and scanner containers will shut down immediately upon receiving SIGTERM to the main process or when receiving docker stop/docker rm -f. Even though there is no graceful shutdown for the controller and scanner containers when receiving SIGTERM, they will gracefully restart. This means the controller and scanner containers understand where they need to pick up again to continue working.

The Remediate container will shut down gracefully after receiving SIGTERM. It is recommended to use the command: kill <process ID> inside of the container itself on the node server in order to make sure that it exits gracefully. If you issue a docker stop or docker rm -f to the container, the Remediate container will not shut down gracefully. This can cause the container to not create any remaining Remediate PRs.

Integration jobs

Every job created by our integration is stored in a queue on Mend servers. When our repo integration containers are restarted, the job may not be picked up where it was left off and will most likely restart the job from the beginning. This includes jobs such as:

1. Onboarding large amounts of repositories with the parameter “nowhitesourceFile” set in the global configuration, in order to trigger a scan upon installation to a repo. (More info here: https://docs.mend.io/bundle/integrations/page/global_repo_configuration.html#global-config.json)

2. Running a normal scan

3. Onboarding large amounts of repositories where onboarding pull requests (PRs) are created

Note: When onboarding large amounts of repositories where the Mend integration will create pull requests, the original onboarding PR in a repository will get closed out and a new onboarding PR will be created.