Caching your Repositories with an Amazon S3 Bucket - Self-Managed Repository Integrations

Introduction

Self-hosted integrations with Remediate/Renovate implemented on a large scale (over 1,000 repositories) can strain your SCM environment or package registry with rate limits. Constantly downloading the same data is also inefficient. The Self-Hosted Mend repository integrations offer two caching solutions to optimize performance: repository caching on Amazon S3 and package caching on Redis.

Prerequisites

For both forms of caching, there are some prerequisites that you will need.

Access to IAM for your Amazon Account to create Policies and Roles
Access to the EC2 instance where the self-hosted integration runs

NOTE: Customers running the integration on Amazon EKS (Elastic Kubernetes Service) should be aware that IAM (Identity and Access Management) in EKS doesn't support Resource-based policies, the method used in this documentation. If this is the case, then customers will need to provision a Service User with appropriate permissions, and then log in with that user in the AWS CLI when deploying their K8s resources.

Once these prerequisites have been fulfilled, then you will need to create a role. This role will provide permissions to the EC2 instance to access the appropriate services.

Go to your IAM console → Roles → Create role
1. Trusted Entity Type: AWS Service
  Service or use case: EC2
  Use Case: EC2
  Add Permissions: Keep this blank for the moment
  Role Name: remediate-worker-role
  Click Create Role

Once this role has been created, then assign your EC2 instance to this role with the following steps:

Go to your EC2 instance and click Actions → Security → Modify IAM role
Select the remediate-worker-role that was created in the previous step

Caching Repositories with Amazon S3

One way of improving Remediate/Renovate performance is to cache repositories using Amazon S3. Once this has been implemented, a directory structure will be created in the bucket with the format: <SCM Environment>/<Organization>/<Repository>/cache.json. The cache.json file includes an encoded version of the repository with a few other values in JSON format.

Prerequisites

The AWS CLI installed on your EC2 Instance.

Instructions

Create a S3 Bucket for caching purposes

Here are the steps and initial permissions required for this bucket:

Name: remediate-repository-cache
Block Public Access settings for this bucket: Block all public access
Bucket Versioning: disable
Default encryption: Server-side encryption with Amazon S3 managed keys (SSE-S3)

Set up appropriate permissions

Go to IAM and create a policy for your S3 bucket. Use the JSON editor and add the following:

JSON

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RemediateS3Access",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>",
        "arn:aws:s3:::<bucket-name>/*"
      ]
    }
  ]
}

Name this policy: remediate-s3-repository-cache-access.
Assign this policy to the role created above by following these steps:
1. IAM Console → Roles
2. remediate-worker-role → Add Permissions → Attach Policies
3. Select remediate-s3-repository-cache-access → Add Permissions
Log into the AWS CLI on your EC2 instance with aws configure, and then test the permissions with: aws s3 ls s3://<bucket-name>. If the permissions work, and the bucket is empty, then nothing will be printed out.

Connect the Repository Integration to the S3 Bucket

In your Repository Integration, configure the following environment variable for the remediate workers only:
RENOVATE_REPOSITORY_CACHE: "enabled"
RENOVATE_REPOSITORY_CACHE_TYPE: "s3://<s3-bucket-name>"
AWS_REGION: "<your region>" (e.g. “us-east-1”)
Restart the integration and trigger a Renovate/Remediate Process on an on-boarded repository.

If all works well, then you should be able to see the directory structure described above inside the Amazon S3 Bucket.

Caching Downloaded Packages with AWS ElastiCache (Redis)

Remediate/Renovate can optimize performance by caching downloaded dependencies from public or private package registries. It uses a Redis cache, which stores information in key-value pairs. Each key is the name of a package, and the corresponding value holds the downloaded data.

Prerequisites

The redis-cli installed on your EC2 Instance.
Port 6379 opened on the Security Group where your EC2 Instance resides.

Instructions

To set up caching packages with AWS ElastiCache, you will need to follow these instructions:

Create an Amazon ElastiCache

Go to the Amazon ElastiCache console → Redis caches
Create Redis Cache:
Deployment option: Serverless
Creation method: New cache
Name: remediate-package-cache
Click Create

Set up the appropriate permissions

Create a user in AWS ElastiCache:
1. Go to Amazon ElastiCache console → User management
2. Create User
3. User ID: remediate
  User Name: remediate
  User authentication settings: IAM Authentication
  Access String: on ~* +@all
4. Click Create
Create a user group for this user:
1. Go to Amazon ElastiCache Console → User group management
2. Create User Group
3. User Group ID: remediate-elasticache-group
  Selected users: default, remediate
4. Click Create
Add the User Group you just created to your ElastiCache resource, and add the resource to your EC2 security group:
1. Go to Amazon ElastiCache Console → Redis Caches
2. Select your Redis Cache → Modify
3. Set Access Control under the Security Tab: User group access control list
4. User Group: remediate-elasticache-group
5. Selected Security Groups → Manage
6. Add your EC2 Instance’s Security Group
7. Preview Changes → Save Changes

Go to IAM and create a policy for your ElastiCache. Use the JSON Editor and enter the following:

JSON

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "RemediateElasticacheAccess",
            "Effect": "Allow",
            "Action": [
                "elasticache:Connect"
            ],
            "Resource": [
                "arn:aws:elasticache:<region>:<account-id>:serverlesscache:<serverless-cache-name>",
                "arn:aws:elasticache:<region>:<account-id>:user:remediate"
            ]
        }
    ]
}

This policy should be named: remediate-elasticache-access
Add your remediate-elasticache-access policy to your remediate-worker-role Role in IAM.

Connect the Repository Integration to the Redis Cache

Get your Redis Cache Domain:
1. Go to Amazon ElastiCache Dashboard → Redis Caches → Select your Redis Cache.
2. Copy the Endpoint under the Connectivity section.
Go to your EC2 Instance and use the redis-cli to test connection to the server:
redis-cli --tls -h <your-endpoint-without-port>
If all goes well, then you should not be disconnected after a few seconds. After this you can type the command: ACL LIST to see the users and their access on the Redis Cache.
Set the following environment variables on both your remediate server and remediate worker containers:
RENOVATE_REDIS_URL: "rediss://<your-endpoint>:6379"
Restart your integration and test some remediate processes to make sure that your cache is working properly. One simple way of doing this is to check a box in a Dependency Dashboard and look at logs. If connection to the Redis server fails, you should see: Socket Closed Unexpectedly errors.

NOTE: The “rediss://” in Step 4 is important as it tells the connection to use TLS when connecting to your Redis cache.

If all works well, then you should be able to see the graphs on the front page of your ElastiCache resource moving with the Remediate Server/Workers adding data to the cache. To list all of the keys stored in your Redis Cache you can use the redis-cli:
redis-cli --tls -h <your-endpoint> --scan --pattern '*'