Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance issue with 6.1.2 #3183

Closed
laurentgo opened this issue Mar 9, 2021 · 5 comments
Closed

Performance issue with 6.1.2 #3183

laurentgo opened this issue Mar 9, 2021 · 5 comments
Labels
Milestone

Comments

@laurentgo
Copy link

laurentgo commented Mar 9, 2021

Describe the bug
On a maven project with hundreds of modules, generating an aggregated report with 6.1.2 is way slower than when using 6.1.1 version. When CVE databases are already cached, it takes around 1 or 2 minutes to generate the whole report with 6.1.1 version, but more than 30min when using 6.1.2.

When doing some profile, it looks like most of the time is spent inside method CveDB#simpleCPESearch()

Version of dependency-check used
The problem occurs using version 6.1.2 of the maven plugin

Log file
Here's the plugin output when using version 6.1.1:

[INFO] Check for updates complete (152 ms)
[INFO] Analysis Started
[INFO] Finished Archive Analyzer (4 seconds)
[INFO] Finished File Name Analyzer (0 seconds)
[INFO] Finished Jar Analyzer (2 seconds)
[INFO] Finished Node.js Package Analyzer (2 seconds)
[INFO] Finished Dependency Merging Analyzer (5 seconds)
[INFO] Finished Version Filter Analyzer (0 seconds)
[INFO] Finished Hint Analyzer (0 seconds)
[INFO] Created CPE Index (1 seconds)
[INFO] Finished CPE Analyzer (18 seconds)
[INFO] Finished False Positive Analyzer (0 seconds)
[INFO] Finished NVD CVE Analyzer (1 seconds)
[INFO] Finished Node Audit Analyzer (0 seconds)
[INFO] Finished RetireJS Analyzer (0 seconds)
[INFO] Finished Sonatype OSS Index Analyzer (2 seconds)
[INFO] Finished Vulnerability Suppression Analyzer (2 seconds)
[INFO] Finished Dependency Bundling Analyzer (43 seconds)
[INFO] Analysis Complete (88 seconds)

And the output for 6.1.2:

[INFO] Check for updates complete (83 ms)
[INFO] Analysis Started
[INFO] Finished Archive Analyzer (4 seconds)
[INFO] Finished File Name Analyzer (0 seconds)
[INFO] Finished Jar Analyzer (3 seconds)
[INFO] Finished Node.js Package Analyzer (2 seconds)
[INFO] Finished Dependency Merging Analyzer (5 seconds)
[INFO] Finished Version Filter Analyzer (0 seconds)
[INFO] Finished Hint Analyzer (0 seconds)
[INFO] Created CPE Index (1 seconds)
[INFO] Finished CPE Analyzer (1524 seconds)
[INFO] Finished False Positive Analyzer (0 seconds)
[INFO] Finished NVD CVE Analyzer (1 seconds)
[INFO] Finished Node Audit Analyzer (0 seconds)
[INFO] Finished RetireJS Analyzer (0 seconds)
[INFO] Finished Sonatype OSS Index Analyzer (2 seconds)
[INFO] Finished Vulnerability Suppression Analyzer (2 seconds)
[INFO] Finished Dependency Bundling Analyzer (46 seconds)
[INFO] Analysis Complete (1597 seconds)

To Reproduce
mvn org.owasp:dependency-check-maven:6.1.2:aggregate

Expected behavior
Performance should be reasonably similar to the previous version.

@laurentgo laurentgo added the bug label Mar 9, 2021
@cwatzl
Copy link

cwatzl commented Mar 12, 2021

I am having the same problem with the Gradle plugin. Downgraded to 6.1.1 for now to get reasonable execution times.

@sellersj
Copy link
Contributor

I have a test app. Between 6.1.1 and 6.1.2 the times jump from 57.101 s to 2:24 min
https://github.com/sellersj/check-owasp-slow

@sellersj
Copy link
Contributor

The SELECT_SIMPLE_CPE_SEARCH query seems to be very unhappy in h2 with all the "Replace" calls.

Running EXPLAIN ANALYZE on the original query with a dummy value I get
scanCount: 209373 and reads: 6423, and take 305 ms
When I remove the Replace values, it's 1 scan and takes 0-1 ms.

@sellersj
Copy link
Contributor

sellersj commented Mar 19, 2021

My guess from what I read, the REPLACE in the WHERE clause are causing a full table scan since the indexes can't be used. A possible solution might be to have a new view with the values changed and indexes on the view.

jeremylong added a commit that referenced this issue Mar 21, 2021
…egy to account for the fact that CPE product names may contain the major version. This also resolves #3193 and #3183
@jeremylong jeremylong added this to the 6.1.3 milestone Mar 21, 2021
@jeremylong
Copy link
Owner

The fix will be released in 6.1.3 very soon. Additional performance enhancements will be in 6.2.0 with #3206.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants