Trivy DB rate limit issues while downloading DBs

[simar7]

Please see updated post #8009

Description

Recently Trivy started to experience rate limit issues while downloading trivy-db and trivy-java-db (here on referred collectively as "DBs") that are required for vulnerability and java scanning. Our users have seen this both while using the Trivy CLI and also when using the trivy-action on GitHub.

We have raised this issue with GitHub, you can track the progress here

Steps to Remediate

In the meantime, we've added several ways to improve the situation for our users.

Trivy CLI

• We've added the ability for users to specify the container registry where the DBs are pulled from via the --db-repository and --java-db-repository flags. You can read more on the options here
• In order to use the above flags, users are able to request the DBs from several public sources namely
  • For Trivy DB:
    • "ghcr.io/aquasecurity/trivy-db:2"
    • "public.ecr.aws/aquasecurity/trivy-db:2"
    • "aquasec/trivy-db:2"
  • For the Java DB:
    • "ghcr.io/aquasecurity/trivy-java-db:1"
    • "public.ecr.aws/aquasecurity/trivy-java-db:1"
    • "aquasec/trivy-java-db:1"
  • For misconfig checks bundle:
    • "ghcr.io/aquasecurity/trivy-checks:1"
    • At the moment we don't offer any other mirrors for the checks bundle. This is because Trivy already has the capability to fallback to checks that are embedded within Trivy. As a result, the scans will run as expected, with the checks that are embedded within Trivy.
  • Keep in mind unauthenticated requests to the above two sources are acceptable but they will be limited to the each of the service's limits. It is highly recommended to use them authenticated by obtaining a token from each of the registries.
• In addition, we have added the ability within Trivy to automatically fallback to known registry sources when one fails. Please see this discussion for details.
• We're currently testing out the rollout of the public ECR mirror as a default fallback option. Please track the progress via this PR: feat: Update registry fallbacks #7679

Trivy GitHub Action

Trivy Action uses Trivy under the hood so the above mentioned improvements are also available to it. Furthermore, we've implemented the following:

• Caching: Previously Trivy Action would download a new DB artifact on each run. We've improved this so that DB artifacts are cached within repositories, thereby reducing the number of times artifacts are fetched from upstream. You can read more on this here: https://github.com/aquasecurity/trivy-action?tab=readme-ov-file#cache
• For further discussions, please see the pinned issue within the trivy-action repo here

[irbull]

I have created a private ECR Pull-Through-Cache four user users, with the idea that we could use a cached copy and only D/L a new DB once whenever it changes. Even with this, the ECR Pull Through Cache is hitting rate limits when pulling from ghcr, which it is only doing once every 24h (at most).

[TicianoH]

yep it's a matter of retries as GitHub Registry (GHCR) doesn't seem to care where the request is running from. You could just add more retries or more updates per day
But the above fix is more definitive of course (I've had no issues when pulling from Aquasec or AWS registries instead)

[ivan-sre]

I'm not sure it's related, but it works properly on ubuntu-22.04 github runners. Please aware that ubuntu-latest is pointing now to ubuntu-24.04

[ChrisWestcottUK]

This has made no difference in my case

[anantM001]

Above for the misconfig checks bundle it is mentioned that a mirror is not being provided for misconfig checks bundle and Trivy scans will run as expected because of the embedded checks bundle.

But I'm facing an issue where I'm getting 'AVD-DS-0031 (CRITICAL): Possible exposure of secret' misconfig occurrences for some of my dockerfiles when the latest misconfig checks bundle download is successful, and not when the download of misconfig checks gets failed. I'm using trivy version 0.56.1 on Ubuntu 24.04.

I think that providing an additional mirror for misconfig checks bundle just like the ones provided for vulnerability and java db, and incorporating --checks-bundle-repository flag (which is used to specify OCI registry for misconfig checks bundle) with fallback to other registry source feature just like the one implemented for vuln and java db, can fix this scan results inconsistency issue.

[knqyf263]

v0.57.1 with mirror.gcr.io is out. See #7938 for details. Please bump Trivy up to v0.57.1+ and let us know if you still face the same issue.

[tstraley]

Can we get similar fixes in place for the misconfig policy bundle (trivy-checks image)?

[knqyf263]

yes
#7953

[laupse]

Hello, we upgraded to 0.57.1. But now the oci image is like corrupted
2024-11-25T13:25:21Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: OCI artifact error: failed to download vulnerability DB: failed to download artifact from mirror.gcr.io/aquasec/trivy-db:2: oci download error: failed to fetch the layer: GET https://mirror.gcr.io/v2/aquasec/trivy-db/blobs/sha256:d0c7b5542a48933d63f0d6742ad2792aa24c77b4771530bff2dcc667177be253: BLOB_UNKNOWN: Unknown blob: sha256:d0c7b5542a48933d63f0d6742ad2792aa24c77b4771530bff2dcc667177be253

[knqyf263]

I think that is a cache issue and is temporary. At least in my environment, it downloads successfully. Could you please try again?
Currently, we only try other registries in case of rate-limiting errors, but if this error happens regularly, it might be better to fall back to other registries even when this kind of error occurs.

[etaormm]

I think that is a cache issue and is temporary. At least in my environment, it downloads successfully. Could you please try again? Currently, we only try other registries in case of rate-limiting errors, but if this error happens regularly, it might be better to fall back to other registries even when this kind of error occurs.

yeah this happens for me as well... fall back to other registries even with other errors seems reasonable.

[knqyf263]

Closing. See #8009