Harbor parses trivy scan result or image tag wrong for nginx 1-alpine(-perl) image

[SymbioticKilla]

Harbor 2.3.2
Trivy 0.17.2
Hi,

the issue is very strange...

Somehow 1.21 is detected as 1.21.0 and not as 1.21.3.
As you can see harbor shows two images but actually it is the same image.
https://hub.docker.com/_/nginx
Same tag: 1.21.3-alpine-perl, mainline-alpine-perl, 1-alpine-perl, 1.21-alpine-perl, alpine-perl

Strange thing is that the image is actually correct:
Wenn I download the image with 1.21-alpine-perl tag from harbor and check it locally, I see that it is 1.21.3 image which is correct and trivy scan shows 6 vulnerabilities and not 30+.

I don't know how harbor works but it seems that some version parser is not correct and it shows cached scan from wrong version.

Thanks!

[danielpacak]

Just for the record I tested both images with Trivy CLI v0.17.2 and I got consistent results:

• docker.io/library/nginx:1.21.3-alpine-perl@sha256:e6dc61d453c2896a0ee861d53db0abfab2b4b32270f0fb882498f32c9aaf459c - 6 vulnerbilities
• docker.io/library/nginx:1.21-alpine-perl@sha256:522494d7025b4a57cbcd92fdfe2b9720d9828b68ac15041a6c3ae1312a8eace5 - 6 vulnerabilities

$ trivy image nginx:1.21.3-alpine-perl
2021-10-10T21:20:44.606+0200	INFO	Need to update DB
2021-10-10T21:20:44.606+0200	INFO	Downloading DB...
24.21 MiB / 24.21 MiB [------------------------------------------------------------------------------------------------------] 100.00% 2.78 MiB p/s 9s
2021-10-10T21:21:05.552+0200	WARN	This OS version is not on the EOL list: alpine 3.14
2021-10-10T21:21:05.552+0200	INFO	Detecting Alpine vulnerabilities...
2021-10-10T21:21:05.553+0200	INFO	Trivy skips scanning programming language libraries because no supported file was detected
2021-10-10T21:21:05.553+0200	WARN	This OS version is no longer supported by the distribution: alpine 3.14.2
2021-10-10T21:21:05.553+0200	WARN	The vulnerability detection may be insufficient because security updates are not provided

nginx:1.21.3-alpine-perl (alpine 3.14.2)
========================================
Total: 6 (UNKNOWN: 0, LOW: 0, MEDIUM: 2, HIGH: 2, CRITICAL: 2)

+---------+------------------+----------+-------------------+---------------+---------------------------------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |                 TITLE                 |
+---------+------------------+----------+-------------------+---------------+---------------------------------------+
| curl    | CVE-2021-22945   | CRITICAL | 7.78.0-r0         | 7.79.0-r0     | curl: use-after-free and              |
|         |                  |          |                   |               | double-free in MQTT sending           |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22945 |
+         +------------------+----------+                   +               +---------------------------------------+
|         | CVE-2021-22946   | HIGH     |                   |               | curl: Requirement to use              |
|         |                  |          |                   |               | TLS not properly enforced             |
|         |                  |          |                   |               | for IMAP, POP3, and...                |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22946 |
+         +------------------+----------+                   +               +---------------------------------------+
|         | CVE-2021-22947   | MEDIUM   |                   |               | curl: Server responses                |
|         |                  |          |                   |               | received before STARTTLS              |
|         |                  |          |                   |               | processed after TLS handshake         |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22947 |
+---------+------------------+----------+                   +               +---------------------------------------+
| libcurl | CVE-2021-22945   | CRITICAL |                   |               | curl: use-after-free and              |
|         |                  |          |                   |               | double-free in MQTT sending           |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22945 |
+         +------------------+----------+                   +               +---------------------------------------+
|         | CVE-2021-22946   | HIGH     |                   |               | curl: Requirement to use              |
|         |                  |          |                   |               | TLS not properly enforced             |
|         |                  |          |                   |               | for IMAP, POP3, and...                |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22946 |
+         +------------------+----------+                   +               +---------------------------------------+
|         | CVE-2021-22947   | MEDIUM   |                   |               | curl: Server responses                |
|         |                  |          |                   |               | received before STARTTLS              |
|         |                  |          |                   |               | processed after TLS handshake         |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22947 |
+---------+------------------+----------+-------------------+---------------+---------------------------------------+

$ trivy image nginx:1.21-alpine-perl
2021-10-10T21:21:49.690+0200	WARN	This OS version is not on the EOL list: alpine 3.14
2021-10-10T21:21:49.691+0200	INFO	Detecting Alpine vulnerabilities...
2021-10-10T21:21:49.693+0200	INFO	Trivy skips scanning programming language libraries because no supported file was detected
2021-10-10T21:21:49.693+0200	WARN	This OS version is no longer supported by the distribution: alpine 3.14.2
2021-10-10T21:21:49.693+0200	WARN	The vulnerability detection may be insufficient because security updates are not provided

nginx:1.21-alpine-perl (alpine 3.14.2)
======================================
Total: 6 (UNKNOWN: 0, LOW: 0, MEDIUM: 2, HIGH: 2, CRITICAL: 2)

+---------+------------------+----------+-------------------+---------------+---------------------------------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |                 TITLE                 |
+---------+------------------+----------+-------------------+---------------+---------------------------------------+
| curl    | CVE-2021-22945   | CRITICAL | 7.78.0-r0         | 7.79.0-r0     | curl: use-after-free and              |
|         |                  |          |                   |               | double-free in MQTT sending           |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22945 |
+         +------------------+----------+                   +               +---------------------------------------+
|         | CVE-2021-22946   | HIGH     |                   |               | curl: Requirement to use              |
|         |                  |          |                   |               | TLS not properly enforced             |
|         |                  |          |                   |               | for IMAP, POP3, and...                |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22946 |
+         +------------------+----------+                   +               +---------------------------------------+
|         | CVE-2021-22947   | MEDIUM   |                   |               | curl: Server responses                |
|         |                  |          |                   |               | received before STARTTLS              |
|         |                  |          |                   |               | processed after TLS handshake         |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22947 |
+---------+------------------+----------+                   +               +---------------------------------------+
| libcurl | CVE-2021-22945   | CRITICAL |                   |               | curl: use-after-free and              |
|         |                  |          |                   |               | double-free in MQTT sending           |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22945 |
+         +------------------+----------+                   +               +---------------------------------------+
|         | CVE-2021-22946   | HIGH     |                   |               | curl: Requirement to use              |
|         |                  |          |                   |               | TLS not properly enforced             |
|         |                  |          |                   |               | for IMAP, POP3, and...                |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22946 |
+         +------------------+----------+                   +               +---------------------------------------+
|         | CVE-2021-22947   | MEDIUM   |                   |               | curl: Server responses                |
|         |                  |          |                   |               | received before STARTTLS              |
|         |                  |          |                   |               | processed after TLS handshake         |
|         |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-22947 |
+---------+------------------+----------+-------------------+---------------+---------------------------------------+

Now, I'm going to check why Harbor is showing so wrong results.

[danielpacak]

Manually pulling images down from DockerHub, tagging them and pushing to Harbor shows the following:

docker image pull nginx:1.21.3-alpine-perl
docker image tag nginx:1.21.3-alpine-perl harbor.dev.io/library/nginx:1.21.3-alpine-perl
docker image push !$

docker image pull nginx:1.21-alpine-perl
docker image tag nginx:1.21-alpine-perl harbor.dev.io/library/nginx:1.21-alpine-perl
docker image push !$

$ docker image ls --digests
REPOSITORY                      TAG                  DIGEST                                                                    IMAGE ID       CREATED         SIZE
nginx                           1.21-alpine-perl     sha256:522494d7025b4a57cbcd92fdfe2b9720d9828b68ac15041a6c3ae1312a8eace5   213a542c9862   4 weeks ago     57.6MB
nginx                           1.21-alpine-perl     sha256:e6dc61d453c2896a0ee861d53db0abfab2b4b32270f0fb882498f32c9aaf459c   213a542c9862   4 weeks ago     57.6MB
nginx                           1.21.3-alpine-perl   sha256:522494d7025b4a57cbcd92fdfe2b9720d9828b68ac15041a6c3ae1312a8eace5   213a542c9862   4 weeks ago     57.6MB
nginx                           1.21.3-alpine-perl   sha256:e6dc61d453c2896a0ee861d53db0abfab2b4b32270f0fb882498f32c9aaf459c   213a542c9862   4 weeks ago     57.6MB
harbor.dev.io/library/nginx     1.21-alpine-perl     sha256:deabdbf1c31b2650e69257b063312927cbd82974a9a5ecf689b250a46df18a64   213a542c9862   4 weeks ago     57.6MB
harbor.dev.io/library/nginx     1.21.3-alpine-perl   sha256:deabdbf1c31b2650e69257b063312927cbd82974a9a5ecf689b250a46df18a64   213a542c9862   4 weeks ago     57.6MB

@SymbioticKilla Did you push images with docker CLI or used replication jobs? In any case please share steps to push these images to your registry.

[SymbioticKilla]

@danielpacak Thank you for investigating. I'm using a replication job:

[danielpacak]

Indeed. I can reproduce now. @steven-zou @heww do you happen to know why replication is not consistent with docker CLI?

In this case I'd expect that we have only one artifact with digest sha256:deabdbf1c31b2650e69257b063312927cbd82974a9a5ecf689b250a46df18a64 and two tags 1.21-alpine-perl and 1.21.3-alpine-perl.

[steven-zou]

Replication will replicate all the manifest index into the harbor, but when using docker CLI, only the image for the specific platform is pulled to the client.

For scanning manifests index, the result is generated by simply add all the results of the underlying sub artifacts.
cc @danielpacak

I don't think Harbor will show the wrong tags, please double confirm. @SymbioticKilla

[SymbioticKilla]

@steven-zou What should I check? I have created a replication with all 1.21.x versions:

[SymbioticKilla]

Here you can see that 1-alpine-perl is same to 1.21-alpine-perl => 1.21.3 should be also same to them and scan should show 6 vulnerabilities

[danielpacak]

Okey so I was tricked by the OCI index and Harbor UX as well. There's this folder icon that you can click to see images grouped by architecture.

Then you will see the following scan results.

Apparently, Harbor sums up vulnerability summaries at OCI index level.

[SymbioticKilla]

Thank you! I haven't even know this "folder" function. I have found an opened issue on nginx. Is it completely problem on nginx or is there any possibility to maintain these cases on harbor?

[danielpacak]

We double checked with @knqyf263 and Trivy scanner works as expected. There is a big differences between nginx images built for linux/386 and linux/amd64 architectures in terms of installed packages. They even have a different base OS! Alpine 3.13.5 for linux/386 vs Alpine 3.14.2 for linux/amd64.

You can use Trivy --list-all-pkgs option to compare installed packages.

trivy image --list-all-pkgs --format json --output 386.json harbor.dev.io/library/nginx@sha256:66371f17cc61bbbed2667b0285a10981deba5eb969df9bfd4cf273706044ddcb

trivy image --list-all-pkgs -f json -o amd64.json harbor.dev.io/library/nginx@sha256:deabdbf1c31b2650e69257b063312927cbd82974a9a5ecf689b250a46df18a64