Configuring the Unified Agent for NPM Yarn HTML
JavaScript is one of the most popular programming languages for building web pages and applications. However, due to the massive extent of open-source sharing around the world, managing your JavaScript open-source dependencies can be challenging.
Mend offers the ability to match your “loose” JavaScript source files to their most likely origin, but obtaining a high level of accuracy can be challenging when performing this matching. Instead of the true origin library, you may see source files matched to libraries that your development team is not using.
To increase the accuracy of the matching, we always recommend using a package manager to manage server-side dependencies. If you do deviate from this practice and scan your JavaScript artifacts not via a package manager, please note that Mend only covers artifacts that are obtainable through https://cdnjs.com.
This article demonstrates the best practices for scanning JavaScript projects with the three main resolvers that are supported by Mend: NPM, Yarn, and HTML (client-side). With each package manager, Mend will be able to obtain the most accurate information such as license, copyright, and vulnerabilities that are impacting your project.
NPM
Getting Started
Before running the scan, we recommend installing the dependencies on your machine first by running the npm install
command. This will download your dependencies and generate the node_modules folder and the package-lock.json file.
Alternatively, this command can also be run by the Unified Agent (UA) during the scan by enabling this parameter in the config file:
npm.runPrestep=true
Upon completion of the step above, you should have the following folder structure:
/scan/directory/package.json
/scan/directory/package-lock.json
/scan/directory/node_modules/
Unified Agent Configuration
There are two different approaches when resolving NPM dependencies.
First Approach
The Unified Agent (UA) can resolve dependencies by parsing the package.json and package-lock.json files - this is the recommended best practice.
This approach is the optimal option as it not only improves the scanning time of the project, but also produces more accurate results. In addition, this mechanism has the ability to detect unmet optional and/or peer dependencies when they are found in the lock file.
npm.resolveDependencies=true (default)
npm.ignoreSourceFiles=true (default)
npm.resolveLockFile=true (default)
Note: when npm dependencies are detected, npm.ignoreSourceFiles=true
will ignore files with extension .js, .coffee, .ts, and .tsx.
With these parameters, the package-lock.json and package.json are required, but the node_modules folder is NOT required.
Second Approach
The Unified Agent (UA) runs the npm ls
command during scanning, and resolves dependencies based on the output of this command.
npm.resolveDependencies=true (default)
npm.ignoreSourceFiles=true (default)
npm.resolveLockFile=false
npm.ignoreNpmLsErrors=true
With these parameters, the package.json and node_modules is required, the package-lock.json is NOT required.
YARN
Getting Started
Similar to NPM, we recommend running the command yarn install
to download the dependencies prior to the scan:
Alternatively, this command can also be run during the Unified Agent (UA) scan when you enable the following parameters in the config file:
npm.yarnProject=true
npm.runPrestep=true
Note: In case both yarn.lock and package-lock.json files are found, npm.yarnProject parameter will be used to determine which resolution to perform.
Upon completion of the above step, you should have the following folder structure:
/scan/directory/package.json
/scan/directory/yarn.lock
/scan/directory/node_modules/
Unified Agent Configuration
Similar to NPM resolver, there are also two different approaches:
First Approach
The Unified Agent (UA) resolves dependencies by parsing the package.json and yarn.lock files - this is the recommended best practice.
npm.resolveDependencies=true (default)
npm.ignoreSourceFiles=true (default)
npm.resolveLockFile=true (default)
npm.yarnProject=true
With these parameters, the yarn.lock and package.json are required, the node_modules folder is NOT required.
Second Approach
The Unified Agent (UA) resolves dependencies by scanning the node_modules and parsing all package.json files.
npm.resolveDependencies=true (default)
npm.ignoreSourceFiles=true (default)
npm.resolveLockFile=false
npm.yarnProject=true
With these parameters, the package.json and node_modules is required, the package-lock.json is NOT required.
HTML
The HTML resolver can be used to resolve JavaScript source files that are used by src attribute in HTML files. However, unlike NPM or Yarn, the HTML resolver uses the source file matching algorithm to map the resolved .js file to the library that we have determined is most likely the origin library.
Unified Agent Configuration
There is only 1 parameter for HTML resolver:
html.resolveDependencies=true (default)
Potential Issue for HTML Resolver
The HTML resolver can cause some inaccuracies since it might also pick up JavaScript source files inside of the node_modules folder. For this scenario, we have two recommendations below:
If you’re running the scan for projects in Node.js server-side, and HTML is not relevant to your project, we can recommend disabling it:
html.resolveDependencies=false
If you’re running the scan on the client-side and intend to enable the HTML resolver, we recommend adding the node_modules to the excludes list:
html.resolveDependencies=true (default)
excludes=**/node_modules/** (default starting from v21.8.1)
The parameters discussed above should cover the general use of scanning JavaScript. However, we understand that you may manage your dependencies differently and you are the person who knows your project best. Therefore we always recommend reviewing all the parameters and editing the settings to fit your project. Details about the parameters can be found here: Unified Agent Configuration Parameters
If you still have a specific question or issue regarding your scan, feel free to contact Mend Support, or post your questions in the Discussions group on our Mend Community website.
FAQ
Q: I have over 100 packages installed in the node_modules folder, however when I run the scan, I only see 30 libraries in my inventory, why is this happening?
A: In the node_modules folder, you will also have devDependencies installed, and by default, the Unified Agent(UA) will not resolve devDependencies because they are not directly used in production.
However, if you would like to include the devDependencies in your inventory, please enable this parameter:
npm.includeDevDependencies=true
Q: I’m getting lots of javascript source files and they’re mapped to an incorrect library. How can I correct this?
A: You can manually change the origin library from the UI following the steps in this document:
Changing the Origin Library for Source Files