Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to download vulnerability DB in trivy 0.32.0+ #3146

Closed
ryandens opened this issue Nov 8, 2022 · 24 comments
Closed

failed to download vulnerability DB in trivy 0.32.0+ #3146

ryandens opened this issue Nov 8, 2022 · 24 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and will be auto-closed.

Comments

@ryandens
Copy link

ryandens commented Nov 8, 2022

Description

I've been having trouble using trivy versions 0.32.0 and higher for several weeks. Trivy cli 0.32.0 and higher cannot seem to download the vulnerability DB from github. I can use these versions of trivy successfully if i first scan an image with trivy version 0.31.3, which downloads the DB and then scan the image again with 0.32.0+ which uses the cached db.

ryandens@Ryans-MacBook-Pro ge % TRIVY_TEMP_DIR=$(mktemp -d)
ryandens@Ryans-MacBook-Pro ge % ~/Downloads/trivy_0.34.0_macOS-ARM64/trivy --cache-dir $TRIVY_TEMP_DIR image --download-db-only 
2022-11-07T17:04:36.630-0800	INFO	Need to update DB
2022-11-07T17:04:36.630-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-07T17:04:36.630-0800	INFO	Downloading DB...

2022-11-07T17:05:05.739-0800	FATAL	init error: DB error: failed to download vulnerability DB: OCI artifact error: OCI artifact error: OCI repository error: Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: i/o timeout
ryandens@Ryans-MacBook-Pro ge % 
ryandens@Ryans-MacBook-Pro ge % ~/Downloads/trivy_0.31.3_macOS-ARM64/trivy --cache-dir $TRIVY_TEMP_DIR image --download-db-only 
2022-11-07T17:05:23.780-0800	INFO	Need to update DB
2022-11-07T17:05:23.780-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-07T17:05:23.780-0800	INFO	Downloading DB...
34.84 MiB / 34.84 MiB [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 7.98 MiB p/s 4.6s
ryandens@Ryans-MacBook-Pro ge % rm -rf $TRIVY_TEMP_DIR

What did you expect to happen?

Any version of trivy should be able to download the trivy db

What happened instead?

An older version of trivy must be used to scan an image first to cache the DB, which is burdensome.

Output of run with -debug:

(paste your output here)

ryandens@Ryans-MacBook-Pro ge % ~/Downloads/trivy_0.34.0_macOS-ARM64/trivy --debug --cache-dir $TRIVY_TEMP_DIR image --download-db-only
2022-11-07T17:11:22.072-0800	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2022-11-07T17:11:22.099-0800	DEBUG	cache dir:  /var/folders/c0/r4nk44951y14twd7j25r6h9h0000gn/T/tmp.i6E0Etc1
2022-11-07T17:11:22.099-0800	DEBUG	There is no valid metadata file: unable to open a file: open /var/folders/c0/r4nk44951y14twd7j25r6h9h0000gn/T/tmp.i6E0Etc1/db/metadata.json: no such file or directory
2022-11-07T17:11:22.099-0800	INFO	Need to update DB
2022-11-07T17:11:22.099-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-07T17:11:22.099-0800	INFO	Downloading DB...
2022-11-07T17:11:22.099-0800	DEBUG	no metadata file
2022-11-07T17:11:51.289-0800	FATAL	init error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:362
  - DB error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.NewRunner
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:121
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/commands/operation.DownloadDB
        /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:117
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).Download
        /home/runner/work/trivy/trivy/pkg/db/db.go:154
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).initOCIArtifact
        /home/runner/work/trivy/trivy/pkg/db/db.go:194
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/oci.NewArtifact
        /home/runner/work/trivy/trivy/pkg/oci/artifact.go:69
  - Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: i/o timeout

Output of trivy -v:

ryandens@Ryans-MacBook-Pro ge % ~/Downloads/trivy_0.34.0_macOS-ARM64/trivy -v     
Version: 0.34.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2022-11-07 18:07:31.473970035 +0000 UTC
  NextUpdate: 2022-11-08 00:07:31.473969535 +0000 UTC
  DownloadedAt: 2022-11-08 00:09:19.37135 +0000 UTC

Additional details (base image name, container registry info...):
@ryandens ryandens added the kind/bug Categorizes issue or PR as related to a bug. label Nov 8, 2022
@DmitriyLewen
Copy link
Contributor

DmitriyLewen commented Nov 11, 2022 •
edited
Loading

Hello @ryandens
Thanks for your report!

Are you using a proxy or something similar?
Looks like #2816 same issue.
Can you check this?

Regards, Dmitriy

@ryandens
Copy link
Author

👋 Hi @DmitriyLewen I am not using a proxy in this case - both of these runs (using 0.31.3 and 0.34.0) were run on my personal computer. However, I can confirm adding the insecure flag for 0.34.0 enables it to download the DB. This flag isn't necessary when using 0.31.3 on the same machine.

Note the following three experiments, running trivy image on 0.34.0 fails, while running trivy image --insecure succeeds. On 0.31.3 trivy image without the --insecure flag succeeds.

ryandens@Ryans-MacBook-Pro ge % ~/Downloads/trivy_0.34.0_macOS-ARM64/trivy image --debug hello-world
2022-11-11T10:47:24.626-0800	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2022-11-11T10:47:24.657-0800	DEBUG	cache dir:  /Users/ryandens/Library/Caches/trivy
2022-11-11T10:47:24.657-0800	DEBUG	There is no valid metadata file: unable to open a file: open /Users/ryandens/Library/Caches/trivy/db/metadata.json: no such file or directory
2022-11-11T10:47:24.657-0800	INFO	Need to update DB
2022-11-11T10:47:24.657-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-11T10:47:24.657-0800	INFO	Downloading DB...
2022-11-11T10:47:24.657-0800	DEBUG	no metadata file
2022-11-11T10:47:53.939-0800	FATAL	init error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:362
  - DB error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.NewRunner
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:121
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/commands/operation.DownloadDB
        /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:117
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).Download
        /home/runner/work/trivy/trivy/pkg/db/db.go:154
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).initOCIArtifact
        /home/runner/work/trivy/trivy/pkg/db/db.go:194
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/oci.NewArtifact
        /home/runner/work/trivy/trivy/pkg/oci/artifact.go:69
  - Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: i/o timeout
ryandens@Ryans-MacBook-Pro ge % ~/Downloads/trivy_0.34.0_macOS-ARM64/trivy image --debug --insecure hello-world 
2022-11-11T10:48:28.281-0800	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2022-11-11T10:48:28.311-0800	DEBUG	cache dir:  /Users/ryandens/Library/Caches/trivy
2022-11-11T10:48:28.311-0800	DEBUG	There is no valid metadata file: unable to open a file: open /Users/ryandens/Library/Caches/trivy/db/metadata.json: no such file or directory
2022-11-11T10:48:28.311-0800	INFO	Need to update DB
2022-11-11T10:48:28.311-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-11T10:48:28.311-0800	INFO	Downloading DB...
2022-11-11T10:48:28.311-0800	DEBUG	no metadata file
34.93 MiB / 34.93 MiB [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 7.04 MiB p/s 5.2s
2022-11-11T10:48:44.241-0800	DEBUG	Updating database metadata...
2022-11-11T10:48:44.242-0800	DEBUG	DB Schema: 2, UpdatedAt: 2022-11-11 18:07:45.792218079 +0000 UTC, NextUpdate: 2022-11-12 00:07:45.792217479 +0000 UTC, DownloadedAt: 2022-11-11 18:48:44.242011 +0000 UTC
2022-11-11T10:48:44.245-0800	INFO	Vulnerability scanning is enabled
2022-11-11T10:48:44.245-0800	DEBUG	Vulnerability type:  [os library]
2022-11-11T10:48:44.245-0800	INFO	Secret scanning is enabled
2022-11-11T10:48:44.245-0800	INFO	If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-11-11T10:48:44.245-0800	INFO	Please see also https://aquasecurity.github.io/trivy/v0.34/docs/secret/scanning/#recommendation for faster secret detection
2022-11-11T10:48:55.549-0800	DEBUG	No secret config detected: trivy-secret.yaml
2022-11-11T10:48:55.709-0800	DEBUG	Image ID: sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
2022-11-11T10:48:55.709-0800	DEBUG	Diff IDs: [sha256:e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359]
2022-11-11T10:48:55.709-0800	DEBUG	Base Layers: []
2022-11-11T10:48:55.714-0800	DEBUG	OS is not detected.
2022-11-11T10:48:55.714-0800	DEBUG	Detected OS: unknown
2022-11-11T10:48:55.714-0800	INFO	Number of language-specific files: 0
ryandens@Ryans-MacBook-Pro ge % rm -rf ~/Library/Caches/trivy/db/                                              
ryandens@Ryans-MacBook-Pro ge % ~/Downloads/trivy_0.31.3_macOS-ARM64/trivy image --debug hello-world        
2022-11-11T10:49:14.257-0800	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2022-11-11T10:49:14.283-0800	DEBUG	cache dir:  /Users/ryandens/Library/Caches/trivy
2022-11-11T10:49:14.284-0800	DEBUG	There is no valid metadata file: unable to open a file: open /Users/ryandens/Library/Caches/trivy/db/metadata.json: no such file or directory
2022-11-11T10:49:14.284-0800	INFO	Need to update DB
2022-11-11T10:49:14.284-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-11T10:49:14.284-0800	INFO	Downloading DB...
2022-11-11T10:49:14.284-0800	DEBUG	no metadata file
34.93 MiB / 34.93 MiB [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 6.86 MiB p/s 5.3s
2022-11-11T10:49:20.441-0800	DEBUG	Updating database metadata...
2022-11-11T10:49:20.441-0800	DEBUG	DB Schema: 2, UpdatedAt: 2022-11-11 18:07:45.792218079 +0000 UTC, NextUpdate: 2022-11-12 00:07:45.792217479 +0000 UTC, DownloadedAt: 2022-11-11 18:49:20.441135 +0000 UTC
2022-11-11T10:49:20.446-0800	INFO	Vulnerability scanning is enabled
2022-11-11T10:49:20.447-0800	DEBUG	Vulnerability type:  [os library]
2022-11-11T10:49:20.447-0800	INFO	Secret scanning is enabled
2022-11-11T10:49:20.447-0800	INFO	If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-11-11T10:49:20.447-0800	INFO	Please see also https://aquasecurity.github.io/trivy/v0.31.3/docs/secret/scanning/#recommendation for faster secret detection
2022-11-11T10:49:21.520-0800	DEBUG	No secret config detected: trivy-secret.yaml
2022-11-11T10:49:21.671-0800	DEBUG	Image ID: sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
2022-11-11T10:49:21.671-0800	DEBUG	Diff IDs: [sha256:e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359]
2022-11-11T10:49:21.671-0800	DEBUG	Base Layers: []
2022-11-11T10:49:21.674-0800	DEBUG	OS is not detected.
2022-11-11T10:49:21.674-0800	DEBUG	Detected OS: unknown
2022-11-11T10:49:21.674-0800	INFO	Number of language-specific files: 0

@DmitriyLewen
Copy link
Contributor

Hello @ryandens
Thanks for your examples.

Looks like problem in update of go-containerregistry.
Can you install v0.32.0 and try to load the db please?

@ryandens
Copy link
Author

v0.32.0 presents the same problems as 0.34.0

ryandens@Ryans-MacBook-Pro ~ % ~/Downloads/trivy_0.32.0_macOS-ARM64/trivy image --debug hello-world
2022-11-14T14:59:20.303-0800	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2022-11-14T14:59:20.317-0800	DEBUG	cache dir:  /Users/ryandens/Library/Caches/trivy
2022-11-14T14:59:20.317-0800	DEBUG	There is no valid metadata file: unable to open a file: open /Users/ryandens/Library/Caches/trivy/db/metadata.json: no such file or directory
2022-11-14T14:59:20.317-0800	INFO	Need to update DB
2022-11-14T14:59:20.317-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-14T14:59:20.317-0800	INFO	Downloading DB...
2022-11-14T14:59:20.317-0800	DEBUG	no metadata file
2022-11-14T14:59:49.860-0800	FATAL	init error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:362
  - DB error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.NewRunner
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:121
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/commands/operation.DownloadDB
        /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:117
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).Download
        /home/runner/work/trivy/trivy/pkg/db/db.go:154
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).initOCIArtifact
        /home/runner/work/trivy/trivy/pkg/db/db.go:194
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/oci.NewArtifact
        /home/runner/work/trivy/trivy/pkg/oci/artifact.go:69
  - Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: i/o timeout
ryandens@Ryans-MacBook-Pro ~ % ~/Downloads/trivy_0.32.0_macOS-ARM64/trivy image --debug --insecure hello-world
2022-11-14T15:00:01.985-0800	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2022-11-14T15:00:02.000-0800	DEBUG	cache dir:  /Users/ryandens/Library/Caches/trivy
2022-11-14T15:00:02.000-0800	DEBUG	There is no valid metadata file: unable to open a file: open /Users/ryandens/Library/Caches/trivy/db/metadata.json: no such file or directory
2022-11-14T15:00:02.000-0800	INFO	Need to update DB
2022-11-14T15:00:02.000-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-14T15:00:02.000-0800	INFO	Downloading DB...
2022-11-14T15:00:02.000-0800	DEBUG	no metadata file
34.97 MiB / 34.97 MiB [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 5.29 MiB p/s 6.8s
2022-11-14T15:00:19.860-0800	DEBUG	Updating database metadata...
2022-11-14T15:00:19.861-0800	DEBUG	DB Schema: 2, UpdatedAt: 2022-11-14 18:07:12.342610381 +0000 UTC, NextUpdate: 2022-11-15 00:07:12.342609981 +0000 UTC, DownloadedAt: 2022-11-14 23:00:19.860883 +0000 UTC
2022-11-14T15:00:19.861-0800	INFO	Vulnerability scanning is enabled
2022-11-14T15:00:19.861-0800	DEBUG	Vulnerability type:  [os library]
2022-11-14T15:00:19.861-0800	INFO	Secret scanning is enabled
2022-11-14T15:00:19.861-0800	INFO	If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-11-14T15:00:19.861-0800	INFO	Please see also https://aquasecurity.github.io/trivy/v0.32/docs/secret/scanning/#recommendation for faster secret detection
2022-11-14T15:00:31.062-0800	DEBUG	No secret config detected: trivy-secret.yaml
2022-11-14T15:00:31.260-0800	DEBUG	Image ID: sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
2022-11-14T15:00:31.260-0800	DEBUG	Diff IDs: [sha256:e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359]
2022-11-14T15:00:31.260-0800	DEBUG	Base Layers: []
2022-11-14T15:00:31.262-0800	DEBUG	OS is not detected.
2022-11-14T15:00:31.262-0800	DEBUG	Detected OS: unknown
2022-11-14T15:00:31.262-0800	INFO	Number of language-specific files: 0

@DmitriyLewen
Copy link
Contributor

@ryandens can you check this? #1717

@jtougas
Copy link

jtougas commented Nov 16, 2022

I'm experiencing the same issue on my computer, but not when used from a github action however...
Using the 0.31.3 image as a workaround does not work for me.

$ time docker run -i --rm  aquasec/trivy:0.34.0 image --download-db-only --no-progress --insecure
2022-11-16T13:27:40.351Z    INFO    Need to update DB
2022-11-16T13:27:40.351Z    INFO    DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-16T13:27:40.351Z    INFO    Downloading DB...

real    0m28.526s
user    0m0.019s
sys 0m0.015s
$ time docker run -i --rm  aquasec/trivy:0.34.0 image --download-db-only --no-progress
2022-11-16T13:28:13.286Z    INFO    Need to update DB
2022-11-16T13:28:13.286Z    INFO    DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-16T13:28:13.286Z    INFO    Downloading DB...
2022-11-16T13:28:42.419Z    FATAL   init error: DB error: failed to download vulnerability DB: OCI artifact error: OCI artifact error: OCI repository error: Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: i/o timeout

real    0m30.302s
user    0m0.021s
sys 0m0.010s

$ time docker run -i --rm  aquasec/trivy:0.31.3 image --download-db-only --no-progress
2022-11-16T13:31:29.290Z    INFO    Need to update DB
2022-11-16T13:31:29.290Z    INFO    DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-16T13:31:29.290Z    INFO    Downloading DB...
2022-11-16T13:31:58.571Z    FATAL   init error: DB error: failed to download vulnerability DB: OCI artifact error: OCI artifact error: OCI repository error: Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: i/o timeout

real    0m30.319s
user    0m0.028s
sys 0m0.021s

@DmitriyLewen
Copy link
Contributor

Hello @jtougas
There is Trivy-action to use from GitHub action.
Have you tried this?

@ryandens
Copy link
Author

Hi @DmitriyLewen, regarding this

@ryandens can you check this? #1717

I verified with netcat that i can connect to ghcr as suggested in #1717

ryandens@Ryans-MacBook-Pro ~ % nc -z ghcr.io 443
Connection to ghcr.io port 443 [tcp/https] succeeded!

I'm fairly certain it's not a network or authentication issue, given that versions prior to 0.32.0 can pull from ghcr perfectly fine on my machine as demonstrated by my last comment.

As another experiment, I checked out the 0.34.0 tag and ran make build to build the trivy executable. The locally built executable was able to download the db

ryandens@Ryans-MacBook-Pro trivy % ./trivy image --debug hello-world                                  
2022-11-21T14:11:39.716-0800	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2022-11-21T14:11:39.729-0800	DEBUG	cache dir:  /Users/ryandens/Library/Caches/trivy
2022-11-21T14:11:39.729-0800	DEBUG	There is no valid metadata file: unable to open a file: open /Users/ryandens/Library/Caches/trivy/db/metadata.json: no such file or directory
2022-11-21T14:11:39.729-0800	INFO	Need to update DB
2022-11-21T14:11:39.729-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-11-21T14:11:39.729-0800	INFO	Downloading DB...
2022-11-21T14:11:39.729-0800	DEBUG	no metadata file
35.22 MiB / 35.22 MiB [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 7.62 MiB p/s 4.8s
2022-11-21T14:11:45.713-0800	DEBUG	Updating database metadata...
2022-11-21T14:11:45.714-0800	DEBUG	DB Schema: 2, UpdatedAt: 2022-11-21 18:07:19.204675007 +0000 UTC, NextUpdate: 2022-11-22 00:07:19.204674607 +0000 UTC, DownloadedAt: 2022-11-21 22:11:45.714034 +0000 UTC
2022-11-21T14:11:45.714-0800	INFO	Vulnerability scanning is enabled
2022-11-21T14:11:45.714-0800	DEBUG	Vulnerability type:  [os library]
2022-11-21T14:11:45.714-0800	INFO	Secret scanning is enabled
2022-11-21T14:11:45.714-0800	INFO	If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-11-21T14:11:45.714-0800	INFO	Please see also https://aquasecurity.github.io/trivy/v0.34/docs/secret/scanning/#recommendation for faster secret detection
2022-11-21T14:11:46.927-0800	DEBUG	No secret config detected: trivy-secret.yaml
2022-11-21T14:11:47.096-0800	DEBUG	Image ID: sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
2022-11-21T14:11:47.096-0800	DEBUG	Diff IDs: [sha256:e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359]
2022-11-21T14:11:47.096-0800	DEBUG	Base Layers: []
2022-11-21T14:11:47.099-0800	DEBUG	OS is not detected.
2022-11-21T14:11:47.099-0800	DEBUG	Detected OS: unknown
2022-11-21T14:11:47.099-0800	INFO	Number of language-specific files: 0
ryandens@Ryans-MacBook-Pro trivy % ./trivy image --debug hello-world --vers
ryandens@Ryans-MacBook-Pro trivy % ./trivy version
Version: 0.34.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2022-11-21 18:07:19.204675007 +0000 UTC
  NextUpdate: 2022-11-22 00:07:19.204674607 +0000 UTC
  DownloadedAt: 2022-11-21 22:11:45.714034 +0000 UTC

@DmitriyLewen
Copy link
Contributor

Hello @ryandens

Thanks for your investigation!

The locally built executable was able to download the db

This is so strange.
Did you install "wrong" Trivy with brew?
Can you download the binary and try it?

@ryandens
Copy link
Author

👋 Hi @DmitriyLewen, the releases were all tested from downloads of the binary off the github release page! You can see this in my copy-pasted shell outputs that the trivy executables are in folders corresponding to their appropriate version in my downloads folder. I unpacked each tar and left it in the downloads folder. Additionally, the output of trivy -v shows the appropriate information for each installation

@DmitriyLewen
Copy link
Contributor

DmitriyLewen commented Nov 23, 2022 •
edited
Loading

Hi @ryandens , thanks!

1 more question: have you logged into ghcr.io in Docker?
There was problem with ghcr.io credentials.

This will not solve your problem, but we collect all the factors to find the problem.

@afdesk afdesk mentioned this issue Nov 24, 2022
Closed
@DmitriyLewen
Copy link
Contributor

@ryandens can I ask one more thing:
Can you build Trivy with this command - CGO_ENABLED=0 go build -ldflags "-s -w -X=main.version=test -extldflags '-static'" ./cmd/trivy and try to load the db?

@lucacome
Copy link

lucacome commented Dec 6, 2022

I've been getting a slightly different message:

- GET https://ghcr.io/token?scope=repository%3Aaquasecurity%2Ftrivy-db%3Apull&service=ghcr.io: DENIED: denied

full output

❯ trivy image --debug --insecure hello-world
2022-12-06T11:36:05.576-0800	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2022-12-06T11:36:05.587-0800	DEBUG	cache dir:  /Users/luca/Library/Caches/trivy
2022-12-06T11:36:05.588-0800	DEBUG	There is no valid metadata file: unable to open a file: open /Users/luca/Library/Caches/trivy/db/metadata.json: no such file or directory
2022-12-06T11:36:05.588-0800	INFO	Need to update DB
2022-12-06T11:36:05.588-0800	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-12-06T11:36:05.588-0800	INFO	Downloading DB...
2022-12-06T11:36:05.588-0800	DEBUG	no metadata file
2022-12-06T11:36:06.037-0800	FATAL	init error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        github.com/aquasecurity/trivy/pkg/commands/artifact/run.go:381
  - DB error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.NewRunner
        github.com/aquasecurity/trivy/pkg/commands/artifact/run.go:124
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/commands/operation.DownloadDB
        github.com/aquasecurity/trivy/pkg/commands/operation/operation.go:117
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).Download
        github.com/aquasecurity/trivy/pkg/db/db.go:154
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).initOCIArtifact
        github.com/aquasecurity/trivy/pkg/db/db.go:194
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/oci.NewArtifact
        github.com/aquasecurity/trivy/pkg/oci/artifact.go:69
  - GET https://ghcr.io/token?scope=repository%3Aaquasecurity%2Ftrivy-db%3Apull&service=ghcr.io: DENIED: denied

@afdesk
Copy link
Contributor

afdesk commented Dec 6, 2022 •
edited
Loading

DENIED: denied

@lucacome it appears when a customer uses an expired Github token, please, check you environment and GITHUB_TOKEN variable

@lucacome
Copy link

lucacome commented Dec 6, 2022 •
edited
Loading

@afdesk I don't remember ever setting a token and I don't usually have it exported it as a variable unless I need it, is it something new?

But even with GITHUB_TOKEN exported it keeps failing... (and it's not expired)

@afdesk
Copy link
Contributor

afdesk commented Dec 6, 2022

@afdesk I don't remember ever setting a token and I don't usually have it exported it as a variable unless I need it, is it something new?

But even with GITHUB_TOKEN exported it keeps failing... (and it's not expired)

Trivy uses default docker permissions to download the database. so if you login to ghcr.io and your token is expired the error will appears...

@lucacome
Copy link

lucacome commented Dec 7, 2022

thanks @afdesk docker login ghcr.io fixed it, but I didn't think I would have to be logged in to pull from a public registry (and I don't remember doing this before), maybe it's worth adding it to the docs?

@afdesk
Copy link
Contributor

afdesk commented Dec 7, 2022

thanks @afdesk docker login ghcr.io fixed it, but I didn't think I would have to be logged in to pull from a public registry (and I don't remember doing this before), maybe it's worth adding it to the docs?

@lucacome You don't have to login especially, it's not necessary. But if you're already logged, your token can't be expired.
I hope it's correct )

@wilddeuces
Copy link

wilddeuces commented Dec 13, 2022 •
edited
Loading

I am experiencing the same issue with Harbor Trivy Adapter.

goharbor/trivy-adapter-photon:v2.6.2

That image uses TRIVY_VERSION=v0.32.1

Same problem persists with

goharbor/trivy-adapter-photon:v2.7.0-dev which uses TRIVY_VERSION=v0.35.0

2022-12-13T13:09:03.651Z	�[34mINFO�[0m	Need to update DB
2022-12-13T13:09:03.651Z	�[34mINFO�[0m	DB Repository: ghcr.io/aquasecurity/trivy-db
2022-12-13T13:09:03.651Z	�[34mINFO�[0m	Downloading DB...
2022-12-13T13:20:03.284Z	�[31mFATAL�[0m	init error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:362
  - DB error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.NewRunner
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:121
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/commands/operation.DownloadDB
        /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:117
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).Download
        /home/runner/work/trivy/trivy/pkg/db/db.go:154
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).initOCIArtifact
        /home/runner/work/trivy/trivy/pkg/db/db.go:194
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/oci.NewArtifact
        /home/runner/work/trivy/trivy/pkg/oci/artifact.go:69
  - Get "https://ghcr.io/v2/": dial tcp 140.82.121.33:443: connect: connection timed out
: general response handler: unexpected status code: 500, expected: 200

Curling the endpoint is working fine.
Also trying to manually pull that image in another container is working fine.

The problem seems to be --insecure option.
When disabling "SCANNER_TRIVY_INSECURE" environment variable the download works fine!

@github-actions
Copy link

This issue is stale because it has been labeled with inactivity.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and will be auto-closed. label Feb 12, 2023
@rams3sh
Copy link

rams3sh commented Feb 12, 2023 •
edited
Loading

I am having the same issue. My execution log below :-

root@5b033ec9a1db:/app# ./scanner/trivy version
Version: 0.36.1


root@5b033ec9a1db:/app# ./scanner/trivy image --download-db-only -d
2023-02-12T15:49:47.319Z	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-02-12T15:49:47.328Z	DEBUG	cache dir:  /app/scanner/cache/
2023-02-12T15:49:47.328Z	DEBUG	There is no valid metadata file: unable to open a file: open /app/scanner/cache/db/metadata.json: no such file or directory
2023-02-12T15:49:47.328Z	INFO	Need to update DB
2023-02-12T15:49:47.328Z	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2023-02-12T15:49:47.328Z	INFO	Downloading DB...
2023-02-12T15:49:47.328Z	DEBUG	no metadata file
2023-02-12T15:50:16.440Z	FATAL	init error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:389
  - DB error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.NewRunner
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:124
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/commands/operation.DownloadDB
        /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:117
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).Download
        /home/runner/work/trivy/trivy/pkg/db/db.go:154
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).initOCIArtifact
        /home/runner/work/trivy/trivy/pkg/db/db.go:194
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/oci.NewArtifact
        /home/runner/work/trivy/trivy/pkg/oci/artifact.go:69
  - Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io: i/o timeout


root@5b033ec9a1db:/app# ./scanner/trivy image --download-db-only -d --insecure
2023-02-12T15:50:33.097Z	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-02-12T15:50:33.106Z	DEBUG	cache dir:  /app/scanner/cache/
2023-02-12T15:50:33.106Z	DEBUG	There is no valid metadata file: unable to open a file: open /app/scanner/cache/db/metadata.json: no such file or directory
2023-02-12T15:50:33.106Z	INFO	Need to update DB
2023-02-12T15:50:33.106Z	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2023-02-12T15:50:33.106Z	INFO	Downloading DB...
2023-02-12T15:50:33.106Z	DEBUG	no metadata file
35.62 MiB / 35.62 MiB [-----------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 4.26 MiB p/s 8.6s
2023-02-12T15:50:54.195Z	DEBUG	Updating database metadata...
2023-02-12T15:50:54.197Z	DEBUG	DB Schema: 2, UpdatedAt: 2023-02-12 12:07:58.149900128 +0000 UTC, NextUpdate: 2023-02-12 18:07:58.149899828 +0000 UTC, DownloadedAt: 2023-02-12 15:50:54.195223719 +0000 UTC


root@5b033ec9a1db:/app# env 
HOSTNAME=5b033ec9a1db
PYTHON_VERSION=3.11.2
PWD=/app
PYTHON_SETUPTOOLS_VERSION=65.5.1
TRIVY_CACHE_DIR=/app/scanner/cache/
HOME=/root
LANG=C.UTF-8
GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D
TERM=xterm
SHLVL=1
PYTHON_PIP_VERSION=22.3.1
PYTHON_GET_PIP_SHA256=d1d09b0f9e745610657a528689ba3ea44a73bd19c60f4c954271b790c71c2653
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/1a96dc5acd0303c4700e02655aefd3bc68c78958/public/get-pip.py
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
OLDPWD=/

I am currently using trivy 0.36.1 within a container. The container has no proxy setup.
Using --insecure works , but without that argument it times out.

I did curl to the ghcr.io and observed that it takes around 15 seconds to connect from inside the container.

root@5b033ec9a1db:/app# time curl https://ghcr.io -v 
*   Trying 20.207.73.86:443...
* Connected to ghcr.io (20.207.73.86) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.ghcr.io
*  start date: Jul 19 00:00:00 2022 GMT
*  expire date: Jul 19 23:59:59 2023 GMT
*  subjectAltName: host "ghcr.io" matched cert's "ghcr.io"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5625e84a62e0)
> GET / HTTP/2
> Host: ghcr.io
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 301 
< content-type: application/json
< docker-distribution-api-version: registry/2.0
< location: https://github.com/features/packages
< date: Sun, 12 Feb 2023 15:55:11 GMT
< content-length: 0
< x-github-request-id: 2A2E:7E97:1C78D:586BE:63E90BDE
< 
* Connection #0 to host ghcr.io left intact

real	0m15.345s
user	0m0.027s
sys	0m0.005s

The DB update was working perfectly fine in my host system. So I configured the host network bridged to the container and ran the image update command. I observed that the DB was getting downloaded immediately. Also curl request finished immediately.

rams3sh@lab:~/Downloads $ docker run --network=host -it vulcon /bin/bash

root@5b033ec9a1db1:/# cd app/

root@5b033ec9a1db1:/app# ./scanner/trivy image --download-db-only
2023-02-12T15:58:35.828Z	INFO	Need to update DB
2023-02-12T15:58:35.828Z	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2023-02-12T15:58:35.828Z	INFO	Downloading DB...
35.62 MiB / 35.62 MiB [-----------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 8.83 MiB p/s 4.2s


root@5b033ec9a1db1:/app# time curl https://ghcr.io -v 
*   Trying 20.207.73.86:443...
* Connected to ghcr.io (20.207.73.86) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.ghcr.io
*  start date: Jul 19 00:00:00 2022 GMT
*  expire date: Jul 19 23:59:59 2023 GMT
*  subjectAltName: host "ghcr.io" matched cert's "ghcr.io"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55c4433df2e0)
> GET / HTTP/2
> Host: ghcr.io
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 301 
< content-type: application/json
< docker-distribution-api-version: registry/2.0
< location: https://github.com/features/packages
< date: Sun, 12 Feb 2023 16:00:05 GMT
< content-length: 0
< x-github-request-id: C890:4160:1FE16:5CB9C:63E90D04
< 
* Connection #0 to host ghcr.io left intact

real	0m0.337s
user	0m0.021s
sys	0m0.005s

root@5b033ec9a1db1:/app# env
HOSTNAME=5b033ec9a1db1
PYTHON_VERSION=3.11.2
PWD=/app
PYTHON_SETUPTOOLS_VERSION=65.5.1
TRIVY_CACHE_DIR=/app/scanner/cache/
HOME=/root
LANG=C.UTF-8
GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D
TERM=xterm
SHLVL=1
PYTHON_PIP_VERSION=22.3.1
PYTHON_GET_PIP_SHA256=d1d09b0f9e745610657a528689ba3ea44a73bd19c60f4c954271b790c71c2653
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/1a96dc5acd0303c4700e02655aefd3bc68c78958/public/get-pip.py
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
OLDPWD=/
_=/usr/bin/env

Can an option be provided along with the db update command for setting timeout as an alternative , so that one can set the timeout accordingly depending on their environment?

I am yet to find the reason for the delay of request from within the container, but seems like the failure of the DB update is because of the time delay.

@github-actions github-actions bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and will be auto-closed. label Feb 13, 2023
@github-actions
Copy link

This issue is stale because it has been labeled with inactivity.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and will be auto-closed. label Apr 15, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 12, 2023
@blackbass64
Copy link

thanks @afdesk docker login ghcr.io fixed it, but I didn't think I would have to be logged in to pull from a public registry (and I don't remember doing this before), maybe it's worth adding it to the docs?

On the other hand, simply log out of ghcr.io with docker logout ghcr.io

@knqyf263
Copy link
Collaborator

For those who reached here.
https://aquasecurity.github.io/trivy/v0.46/docs/references/troubleshooting/#denied

@bl4ko bl4ko mentioned this issue Nov 29, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and will be auto-closed.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@lucacome @knqyf263 @rams3sh @jtougas @ryandens @afdesk @wilddeuces @blackbass64 @DmitriyLewen