Splitting Unified Agent Scans for Large or Mono Repos

Summary

If you have ever scanned an extremely large project with the Unified Agent, then you know that it doesn’t accept more than 1 million dependencies. This can cause an issue as sometimes there are projects where the amount of direct dependencies and transitive dependencies is larger than 1 million, and in that case any normal scan would fail.

Example:

Number of dependencies: 42801682 exceeded the maximum supported: 1000000
Process finished with exit code SERVER_FAILURE (Number of dependencies exceeded the maximum supported)

NOTE: The limitation of 1 million dependencies is both for libraries resolved by package managers, as well as source files.

Methodology

You can benefit greatly when splitting up your scans if you know the Unified Agent Scan Flow. Usually, the process is as follows:

1. Load the Configuration File

2. Scan all the Package Manager Resolutions that are configured

3. Scan the filesystem if configured

4. Aggregate the results and send them to Mend servers

5. Display results

With this information, we can define two different way to scan applications with a large amount of dependencies.

Scanning on Application Boundary

This approach is useful for mono-repos where there could be a large amounts of dependencies in multiple projects. To do this, you can split the application by folder at the top level of the mono-repo using the projectPerFolder variable. This approach allows each individual folder in the mono-repo to exist in the Mend UI as separate projects that are named based on the subfolder.

Example:

export WS_APIKEY=<your api key>
export WS_USERKEY=<your user key>
export WS_PRODUCTNAME=<your product name>
export WS_PROJECTPERFOLDER=true
export WS_EXCLUDES=<exclude any unnecessary folders here>
java -jar ./wss-unified-agent.jar

Scanning with OVERRIDE/APPEND

This method involves initiating multiple scans to upload all information for a project into one project in the Mend UI. You should order your scans as follows, with different configurations per scan:

1. Package Manager Resolutions First:

   1. Your first scan should have updateType=OVERRIDE

   2. Set fileSystemScan=false to turn off binary and source file scanning

   3. Exclude unnecessary folders using the excludes parameter

2. File System Scan Second:

   1. Set resolveAllDependencies=false to turn off package manager resolution scanning

   2. Set updateType=APPEND to add the result of the second scan to the originally scanned project

   3. Set fileSystemScan=true to turn on binary and source file scanning

Example:

curl -LJO https://unified-agent.s3.amazonaws.com/wss-unified-agent.jar

export WS_APIKEY=<your api key>
export WS_USERKEY=<your user key>
export WS_PRODUCTNAME=<your product name>
export WS_PROJECTNAME=<your project name>
export WS_FILESYSTEMSCAN=false
export WS_EXCLUDES=<exclude any unnecessary folders here>
java -jar ./wss-unified-agent.jar

export WS_RESOLVEALLDEPENDENCIES=false
export WS_FILESYSTEMSCAN=true
export WS_UPDATETYPE=APPEND
java -jar ./wss-unified-agent.jar