Trivy Config Fatal Scan Error - Failed to initialize built-in policies (OPA bundle error, OCI repository error)

[AErmie]

Description

When running Trivy config (using the Docker container method), to scan a Dockerfile, a fatal error is thrown about the OPA policies.

Command Used

docker run --rm --volume /var/run/docker.sock:/var/run/docker.sock \
            --volume $(pwd)/pipelines/.trivyignore:/.trivyignore \
            --volume $HOME/Library/Caches:/root/.cache/ \
            --volume $(pwd):/code \
            --name Trivy aquasec/trivy:latest config ./Dockerfile

What did you expect to happen?

Trivy to complete the scan of the Dockerfile.

What happened instead?

The following error is thrown:

FATAL scan error: failed to initialize built-in policies: failed to download built-in policies: OPA bundle error: OCI repository error: Get "https://ghcr.io/v2/": net/http: TLS handshake timeout

Output of run with -debug:

2021-09-27T02:30:40.089Z	DEBUG	Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2021-09-27T02:30:40.092Z	DEBUG	cache dir:  /root/.cache/trivy
2021-09-27T02:30:40.092Z	DEBUG	Vulnerability type:  []
2021-09-27T02:30:40.092Z	DEBUG	Failed to open the policy metadata: open /root/.cache/trivy/policy/metadata.json: no such file or directory
2021-09-27T02:30:40.092Z	INFO	Need to update the built-in policies
2021-09-27T02:30:40.092Z	INFO	Downloading the built-in policies...
2021-09-27T02:31:34.373Z	FATAL	scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.runWithTimeout
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:69
  - failed to initialize built-in policies:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:164
  - failed to download built-in policies:
    github.com/aquasecurity/trivy/pkg/commands/operation.InitBuiltinPolicies
        /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:125
  - OPA bundle error:
    github.com/aquasecurity/trivy/pkg/policy.(*Client).DownloadBuiltinPolicies
        /home/runner/work/trivy/trivy/pkg/policy/policy.go:169
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/policy.(*Client).populateImage
        /home/runner/work/trivy/trivy/pkg/policy/policy.go:160
  - Get "https://ghcr.io/v2/": net/http: TLS handshake timeout
##[debug]Exit code 1 received from tool '/usr/bin/bash'
##[debug]STDIO streams have closed for tool '/usr/bin/bash'
##[error]Bash exited with code '1'.

Output of trivy -v:

Version: 0.19.2
Vulnerability DB:
  Type: Light
  Version: 1
  UpdatedAt: 2021-09-24 12:05:50.408035299 +0000 UTC
  NextUpdate: 2021-09-24 18:05:50.408034999 +0000 UTC
  DownloadedAt: 2021-09-24 13:09:29.089610356 +0000 UTC

Additional details (base image name, container registry info...):

[knqyf263]

Thanks for the report. It looks like https://ghcr.io was down or unstable. Are you still facing the same issue?

[AErmie]

Hi @knqyf263, yes I'm still seeing the same error. Here's the latest debug output...

Status: Downloaded newer image for aquasec/trivy:latest
2021-09-29T12:03:53.326Z	DEBUG	Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2021-09-29T12:03:53.330Z	DEBUG	cache dir:  /root/.cache/trivy
2021-09-29T12:03:53.330Z	DEBUG	Vulnerability type:  []
2021-09-29T12:03:53.330Z	DEBUG	Failed to open the policy metadata: open /root/.cache/trivy/policy/metadata.json: no such file or directory
2021-09-29T12:03:53.330Z	INFO	Need to update the built-in policies
2021-09-29T12:03:53.330Z	INFO	Downloading the built-in policies...
2021-09-29T12:04:47.619Z	FATAL	scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.runWithTimeout
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:69
  - failed to initialize built-in policies:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:164
  - failed to download built-in policies:
    github.com/aquasecurity/trivy/pkg/commands/operation.InitBuiltinPolicies
        /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:125
  - OPA bundle error:
    github.com/aquasecurity/trivy/pkg/policy.(*Client).DownloadBuiltinPolicies
        /home/runner/work/trivy/trivy/pkg/policy/policy.go:169
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/policy.(*Client).populateImage
        /home/runner/work/trivy/trivy/pkg/policy/policy.go:160
  - Get "https://ghcr.io/v2/": net/http: TLS handshake timeout
##[debug]Exit code 1 received from tool '/usr/bin/bash'
##[debug]STDIO streams have closed for tool '/usr/bin/bash'
##[error]Bash exited with code '1'.

[AErmie]

Hi @knqyf263, so I've made another discovery that may/may not help determine the root cause.

I've run trivy config in 2 different Azure DevOps pipelines. One worked successfully, the other, threw this error. The difference? One was using a Microsoft-Hosted build agent, whereas the other was using a self-hosted build agent.

That's the only difference. Interestingly, the Microsoft-hosted agent did not have any issues, whereas the self-hosted agent did. The latest run attempt threw this error:

2021-10-05T00:06:22.974Z	FATAL	scan error: failed to initialize built-in policies: failed to download built-in policies: OPA bundle error: OCI repository error: Get "https://ghcr.io/v2/": net/http: TLS handshake timeout

Any thoughts as to why that would be? Any specifics that might need to be accounted for with self-hosted agents?

[knqyf263]

The error seems to be irrelevant to Trivy. Again, looks like the network issue. Does the following command work in your environment?

$ curl https://ghcr.io/v2/

[AErmie]

Hey @knqyf263, it doesn't look like the curl command works in the target environment. However, even when I try it from my local system, I encounter this error:

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]}

[knqyf263]

UNAUTHORIZED is ok since it means it is reachable. If you cannot reach GHCR from your env, you should investigate your network.

[jpinkham]

@knqyf263 Can you tell us the exact URL (or even a glob) that trivy is trying to reach out to for policy updates? I'm having a similar problem but I need to whitelist this URL in my firewall.

[AErmie]

@jpinkham, I'm in the same situation (RE: firewall).

Based on the Built-In Policies documentation...

AppShield policies are destributed as OPA bundle on GitHub Container Registry (GHCR). When misconfiguration detection is enabled, Trivy pulls OPA bundle from GHCR as OCI artifact and stores it in the cache. Then, those policies are loaded into Trivy OPA engine and used for detecting misconfigurations.

I'm just not sure if there's more to it. Appreciate any clarity/guidance you can provide to us, @knqyf263

[github-actions]

This issue is stale because it has been labeled with inactivity.

[knqyf263]

I'm sorry I missed your messages. We have a list here, but this is for the vulnerability database. As for builtin policies, ghcr.io is needed. I'll update the doc.
https://aquasecurity.github.io/trivy/v0.21.2/getting-started/troubleshooting/#error-downloading-vulnerability-db

[jpinkham]

@AErmie : You may find this very recent update helpful in https://github.com/aquasecurity/trivy/blob/main/docs/advanced/air-gap.md . It now contains info on downloading policies for offline use. Also https://github.com/aquasecurity/trivy/blob/main/docs/getting-started/troubleshooting.md has a new additional site that needs whitelisting (i think it's only needed if you're on the most recent version of trivy).

The PR #1539 has a bunch of stuff related to the DBs, including the document updates. Specific to doing offline scans in a fully air-gapped environment, there are changes in #1511 and #1512

I verified that using the "--offline-scan" option works beautifully for java (though it of course cannot detect anything for java as a result). Combined with "--debug", you can see what dependencies are trying to be resolved and then skipped because they couldn't be found locally nor online....and no delay waiting for a timeout.

Thanks so much @knqyf263!!!!!

[hron84]

@jpinkham the mentioned documentations are not available anymore and there is an issue accessing with gchr.io again.

[jpinkham]

@hron84 Try https://aquasecurity.github.io/trivy/v0.56/docs/advanced/air-gap/