Preparing Scan Logs for Side-by-Side Comparison - Unified Agent

When running more than one scan with the Unified Agent against the same project or code base, it is often needed to compare the resulting scan logs against each other (side by side) in order to investigate a given problem or behavior. Although there are tools designed to compare files, they fail to show precisely what areas of the logs are different due to the fact that the time stamps in the logs do not match, resulting in the compare tool identifying almost every line in the logs as different. See left bar in screenshot below for an example. It is showing solid blue which indicates almost every line in the first log is different from the second log:

By performing some easy data manipulation in the logs, it is possible to remove the timestamps from both logs which will allow the compare tool to show more meaningful results. See screenshot below where the true differences are now highlighted:

Instructions

In order to remove the timestamps, you’ll need access to a Bash shell on a Linux machine, Linux Subsystem for Windows or a Linux Docker container. Proceed as follows (Docker steps can be skipped if you already have access to a Linux machine):

1. Pull a copy of a Linux image to use (in case you don’t have one already) and start a container for it:

   CODE

   C:\docker pull centos:latest
   C:\docker run -it centos:latest

2. Copy the log files into the container (for example, log_A.txt and log_B.txt):

   CODE

   C:\docker cp log_A.txt <container ID>:\home
   C:\docker cp log_B.txt <container ID>:\home

3. From within the container “\home” directory execute the following (this removes unnecessary non-printable characters introduced by Windows):

   CODE

   %>strings -1 log_A.txt > log_A_STR.txt
   %>strings -1 log_B.txt > log_B_STR.txt

4. Perform the data manipulation:

   CODE

   %>cat log_A_STR.txt | sed -r "{s/\[202.*\]/[---]/}" | sed -r "{s/\[202.*\[/[---]/}" >log_A_FLTR.txt
   %>cat log_B_STR.txt | sed -r "{s/\[202.*\]/[---]/}" | sed -r "{s/\[202.*\[/[---]/}" >log_B_FLTR.txt

5. Copy the logs back to your local machine:

   CODE

   C:>docker cp <container ID>:\home\log_A_FLTR.txt .
   C:>docker cp <container ID>:\home\log_B_FLTR.txt .

The newly created files can now be loaded into the compare tool for analysis.

Note that the Unified Agent scan logs contain many lines indicating a progress bar of the scan, they look like this:

[INFO] | [###################              ] 59% - 12,098 of 20,301 files
[INFO] | [###################              ] 59% - 12,099 of 20,301 files
[INFO] / [###################              ] 59% - 12,100 of 20,301 files
[INFO] / [###################              ] 59% - 12,101 of 20,301 files
[INFO] / [###################              ] 59% - 12,102 of 20,301 files

Such lines add noise to the log analysis and since they display the total number of files being scanned, chances are, that number is different between the 2 scans which is going to cause the compare tool to flag each line as different. For investigation purposes, those lines can be either removed or modified to eliminate the numbers that will cause them to be flagged as different.

Option 1: Remove the lines altogether (recommended)

In step 4 above, replace with the following command instead:

%>cat log_A-STR.txt | sed -r "{s/\[202.*\]/[---]/}" | sed -r "{s/\[202.*\[/[---]/}" | grep -E -v -e "] [0-9]?[0-9]% - " >log_A_FLTR.txt
%>cat log_B-STR.txt | sed -r "{s/\[202.*\]/[---]/}" | sed -r "{s/\[202.*\[/[---]/}" | grep -E -v -e "] [0-9]?[0-9]% - " >log_B_FLTR.txt

Option 2: Modify the lines to only remove the percentage numbers

In step 4 above, replace with the following command instead:

%>cat log_A_STR.txt | sed -r "{s/\[202.*\]/[---]/}" | sed -r "{s/\[202.*\[/[---]/}" | sed -r "{s/\] [0-9]?[0-9]% - .*/]/}" > log_A_FLTR.txt
%>cat log_B_STR.txt | sed -r "{s/\[202.*\]/[---]/}" | sed -r "{s/\[202.*\[/[---]/}" | sed -r "{s/\] [0-9]?[0-9]% - .*/]/}" > log_B_FLTR.txt