[ERROR] Error updating the NVD Data; the NVD returned a 403 or 404 error ( Using 10.0.2 )

[snake121314spbr]

Hi,

After upgrade my plugin for the lastest version 10.0.2, I'm still receiving this error below.:

Installing dependency check from /var/lib/jenkins/caches/dependency-check/LINUX/10.0.2.tar.gz to /var/lib/jenkins/tools/org.jenkinsci.plugins.DependencyCheck.tools.DependencyCheckInstallation/OWASP-July-2024 on Jenkins_node01
[INFO] Checking for updates
[ERROR] Error updating the NVD Data; the NVD returned a 403 or 404 error

Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

If your NVD API Key is valid try increasing the NVD API Delay.

If this is ocurring in a CI environment
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data; the NVD returned a 403 or 404 error

Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

If your NVD API Key is valid try increasing the NVD API Delay.

If this is ocurring in a CI environment
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:387)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427)
at org.owasp.dependencycheck.App.run(App.java:172)
at org.owasp.dependencycheck.App.main(App.java:89)
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[INFO] Begin database defrag
[INFO] End database defrag (256 ms)
[ERROR] Error updating the NVD Data; the NVD returned a 403 or 404 error

Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

If your NVD API Key is valid try increasing the NVD API Delay.

If this is ocurring in a CI environment
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data; the NVD returned a 403 or 404 error

Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

If your NVD API Key is valid try increasing the NVD API Delay.

If this is ocurring in a CI environment
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:387)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427)
at org.owasp.dependencycheck.App.run(App.java:172)
at org.owasp.dependencycheck.App.main(App.java:89)
ERROR: Mark build as failed because of exit code 8

---

I'm already checked the API Key.. everything OK and the communication with the SITE NVD ok too..

But I'm still receiving this... start only after upgrade from 9.0 to 10.0.2 because these last problems from NVD.. before this.. using 9.0 everthing ok.

[aikebah]

Most likely cause for a 403 would be too many requests for your API key. Do you have an API-key specific to your project, or a shared API key that is used across all/many projects?

Best setup is to have your projects run in no-update mode with a dedicated schedule-triggered job that maintains the CVE database locally (empty project/job invoking dependencycheck in update-only mode)

Second-best setup is only having a shared/cached location for the data, but each job configured to update this central data whenever it builds.

Best guarantees for disasters like this are jobs that each and every time have to build the CVE database from scratch (which needs to get 200k+ entries from the NVD each and every build)

[aikebah]

(speculation mode on) It may very well be that in addition to blocking 'old clients' that NVD has also put measures in place that will block APIKeys when they observe abusive behaviour of the last type of project setup on a certain APIKey (speculation mode off).

NVD is hosting a valuable database with limited resources. The best thing the community can do is to use it in a gentle manner and therefor cache results locally in their CI infrastructure rather than re-retrieving the entire dataset on every CI-build.

[chadlwilson]

As this also seems to be the jenkins plugin I guess it's additionally difficult to tell if the key is being supplied properly? Not sure.

If it works fine locally using ODC CLI with a different API key then the problem is clearly the Jenkins plugin or some additional throttling like @aikebah alludes to.

It is a good question as to how the throttling behaves from NVD. If they don't do it already, they really should be replying with HTTP 429 (or 503 if they really have to) for throttling situations to distinguish from 403 for (bad API key). Preferably they'd give a retry-after header and even concurrent clients could back off to the required time and "co-exist".

I assume from @aikebah's response above that he's pretty sure they dont use 429s for these cases where an individual API key is making too many requests though. And nothing is mentioned at https://nvd.nist.gov/developers/start-here.

the rate limit with an API key is 50 requests in a rolling 30 second window

Is actually quite a lot, especially when bulk retrieving the database. In theory you could retrieve the entire database (~250,000 entries) in parallel in batches of 10,000 and be under that limit - but yeah, probably not their desired usage, and likely retrying 50xs also count against ones rate limit. :-)

[aikebah]

Don't know whether they use the 429, but can't recall having seen 429s, which is why I went to speculation mode that maybe they put 403s for that case as well.

[chadlwilson]

Yeah it's fair speculation. Maybe I'll try an empirical test at some point. Hmm.

[snake121314spbr]

@jeremylong hi how are you?

I would like to ask you connected with this problem above.. because I identified another limitation, only 1000 registers are authorized and the last 10.0.2 trying 2000 ( [WARN] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=0 : 2 time )

• TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
  { [1 bytes data]

• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  { [265 bytes data]

• TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
  { [1 bytes data]

• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  { [265 bytes data]

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* TLSv1.3 (IN), TLS Unknown, Unknown (23):
  { [1 bytes data]
  < HTTP/1.1 404
  < message: resultsPerPage parameter cannot exceed 1000.

It's possible create a new version ( 10.0.3 ) only to return 1000 registers per page?

[aikebah]

Hmmm.. their response message contradicts their documentation as https://nvd.nist.gov/developers/vulnerabilities documents both default and upper limit as 2000

[aikebah]

And it must be over 1k, as I just successfully updated my local CVE database with 1580 new records.
Did you properly configure an API key for NVD?

[snake121314spbr]

Hi @aikebah yes everthing ok .. I tested with Curl and works very well if I change from 2000 to 1000.

If have any argument to use in the jenkins pipeline to make the Owasp Dependence consider only 1000, because the --updateonly doesn't work, I my tests .. I purged my cache before @jeremylong released 10.0.2.

I have another test environment in AWS and from there I don't have any problem to use 2000 registers, but in my official environment on Azure, only works the Curl if I change to 1000. My pipeline returns only these results above.

So strange.. this behavior

[aikebah]

Both with and without an API key NIST NVD responds normally on a resultsPerPage of 2000

So my suspicion would go to an intercepting firewall/proxy in your infrastructure that has some generic policy activated for a resultsPerPage query-parameter

[aikebah]

Quite a badly configured one even... as a request parameter violation should be accompanied by an HTTP 400 (Bad Request) response code rather than 404 (Not Found)

[snake121314spbr]

Hi @aikebah , no I considered this too and my access is direct to internet, no filter or proxy, but still returning this error.

I imagined some WAF in the NVD side.. maybe some rule there.. for the range IP used by me.

[snake121314spbr]

my all log when I tried use 2000 registers by Curl.

pdate OWASP Dependency Check Cache)
Shell Script -- curl -H "Accept: application/json" -H "apiKey: XXXX" -v "https://services.nvd.nist.gov/rest/json/source/2.0?resultsPerPage=2000" (self time 898ms)

• curl -H Accept: application/json -H apiKey: XXXXX -v https://services.nvd.nist.gov/rest/json/source/2.0?resultsPerPage=2000
  % Total % Received % Xferd Average Speed Time Time Time Current
  Dload Upload Total Spent Left Speed

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 18.235.227.114:443...

• Connected to services.nvd.nist.gov (18.235.227.114) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• TLSv1.0 (OUT), TLS header, Certificate Status (22):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
  } [512 bytes data]
• TLSv1.2 (IN), TLS header, Certificate Status (22):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Server hello (2):
  { [122 bytes data]
• TLSv1.2 (IN), TLS header, Finished (20):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  { [25 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Certificate (11):
  { [2586 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
  { [264 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Finished (20):
  { [52 bytes data]
• TLSv1.2 (OUT), TLS header, Finished (20):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  } [1 bytes data]
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Finished (20):
  } [52 bytes data]
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN, server accepted to use http/1.1
• Server certificate:
• subject: CN=*.nvd.nist.gov
• start date: May 31 01:13:03 2024 GMT
• expire date: Aug 29 01:13:02 2024 GMT
• subjectAltName: host "services.nvd.nist.gov" matched cert's "*.nvd.nist.gov"
• issuer: C=US; O=Let's Encrypt; CN=R3
• SSL certificate verify ok.
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
  } [5 bytes data]

GET /rest/json/source/2.0?resultsPerPage=2000 HTTP/1.1
Host: services.nvd.nist.gov
User-Agent: curl/7.81.0
Accept: application/json
apiKey: XXXXX

• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]

• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  { [265 bytes data]

• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]

• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  { [265 bytes data]

• old SSL session ID is stale, removing

• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]

• Mark bundle as not supporting multiuse
  < HTTP/1.1 404
  < message: resultsPerPage parameter cannot exceed 1000.
  < x-frame-options: SAMEORIGIN
  < access-control-allow-origin: *
  < access-control-allow-headers: accept, apiKey, content-type, origin, x-requested-with
  < access-control-allow-methods: GET, HEAD, OPTIONS
  < access-control-allow-credentials: false
  < date: Mon, 15 Jul 2024 17:24:44 GMT
  < content-length: 0
  < apikey: Yes
  < strict-transport-security: max-age=31536000
  <

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0

• Connection #0 to host services.nvd.nist.gov left intact

[aikebah]

sounds like a genuine direct connection towards NVD infrastructure indeed, same IP shown for me on nslookup of the host

[aikebah]

@snake121314spbr For future: don't forget to delete your initial comment when redacting sensitive data (I already took care if it for you this time) - see https://docs.github.com/en/communities/moderating-comments-and-conversations/tracking-changes-in-a-comment

[aikebah]

@jeremylong I've started to work on a PR to allow configuration of a lower resultsPerPage for cases like mentioned in the comments.

[snake121314spbr]

Thank you very much!!!!

[jeremylong]

We should surface this error better as well... Let me look at the client code.

[jeremylong]

With release 10.0.3 the client should automatically adjust if the NVD requests a lower resultsPerPage. For other errors with the NVD API 10.0.3 improves the error reporting so it will be easier to determine exactly what is wrong.