crane: index filter finds the image has a legacy format thus do not support, while manifest says opposite

[IceCodeNew]

Describe the bug

I was utilizing the crane index filter to only sync images to a private registry and filter out architectures that I did not care about.
However, I could not get it to work for image docker.io/neuvector/scanner:latest. crane reported Error: pulling docker.io/neuvector/scanner:latest: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

I dug in and found out that the manifest of neuvector image did comply with the schema V2, and this only deepens the puzzle.
It looks more likely a bug rather than a deformation of the docker images, so I reached here before reporting to the neuvector community.
Any help would be appreciated ;-)

To Reproduce

crane --verbose index filter docker.io/neuvector/scanner:latest --platform linux/amd64 -t "$dest"

[root@node ~]# echo $src
docker.io/neuvector/scanner:latest
[root@node ~]# crane --verbose index filter "$src" --platform linux/amd64 -t "$dest"
2023/06/05 19:30:28 --> GET https://index.docker.io/v2/
2023/06/05 19:30:28 GET /v2/ HTTP/1.1
Host: index.docker.io
User-Agent: crane/0.15.2 go-containerregistry/0.15.2
Accept-Encoding: gzip


2023/06/05 19:30:29 <-- 401 https://index.docker.io/v2/ (1.0539876s)
2023/06/05 19:30:29 HTTP/1.1 401 Unauthorized
Content-Length: 87
Content-Type: application/json
Date: Mon, 05 Jun 2023 11:30:29 GMT
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=31536000
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

2023/06/05 19:30:29 --> GET https://auth.docker.io/token?scope=repository%3Aneuvector%2Fscanner%3Apull&service=registry.docker.io [body redacted: basic token response contains credentials]
2023/06/05 19:30:29 GET /token?scope=repository%3Aneuvector%2Fscanner%3Apull&service=registry.docker.io HTTP/1.1
Host: auth.docker.io
User-Agent: crane/0.15.2 go-containerregistry/0.15.2
Accept-Encoding: gzip


2023/06/05 19:30:31 <-- 200 https://auth.docker.io/token?scope=repository%3Aneuvector%2Fscanner%3Apull&service=registry.docker.io (1.914007098s) [body redacted: basic token response contains credentials]
2023/06/05 19:30:31 HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Date: Mon, 05 Jun 2023 11:30:30 GMT
Strict-Transport-Security: max-age=31536000
X-Trace-Id: ce96eabe89874f5330f95231abe64d9f


2023/06/05 19:30:31 --> GET https://index.docker.io/v2/neuvector/scanner/manifests/latest
2023/06/05 19:30:31 GET /v2/neuvector/scanner/manifests/latest HTTP/1.1
Host: index.docker.io
User-Agent: crane/0.15.2 go-containerregistry/0.15.2
Accept: application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization: <redacted>
Accept-Encoding: gzip


2023/06/05 19:30:31 <-- 200 https://index.docker.io/v2/neuvector/scanner/manifests/latest (324.225312ms)
2023/06/05 19:30:31 HTTP/1.1 200 OK
Content-Length: 6218
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Date: Mon, 05 Jun 2023 11:30:31 GMT
Docker-Content-Digest: sha256:073c9341f5bd72454a63667ee19268a77317794509a693b0892b30f75c5baa6e
Docker-Distribution-Api-Version: registry/2.0
Docker-Ratelimit-Source: 117.50.122.14
Etag: "sha256:073c9341f5bd72454a63667ee19268a77317794509a693b0892b30f75c5baa6e"
Ratelimit-Limit: 100;w=21600
Ratelimit-Remaining: 70;w=21600
Strict-Transport-Security: max-age=31536000

{
   "schemaVersion": 1,
   "name": "neuvector/scanner",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:056e5a91cf035418524104eba706002cad28aab8ead43d11b073a8374905504d"
      },
      {
         "blobSum": "sha256:2703044be1479093b493ce6f6a9a70feef9ce93b347dcf104df68933d4791f45"
      },
      {
         "blobSum": "sha256:f8ab9d0286ee2e40937a52a0bbdd34e1d42bc93b5e29f6be1cb42584f88e6ba8"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:8a49fdb3b6a5ff2bd8ec6a86c05b2922a0f7454579ecc07637e94dfd1d0639b6"
      }
   ],
   "history": [
      {
         "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":null,\"Image\":\"sha256:0da4c3c58068829dbdbe9f828cb35f258b69123bc17b3ef26695e57cb4242f62\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":[\"/usr/local/bin/monitor\"],\"OnBuild\":[],\"Labels\":{\"neuvector.image\":\"neuvector/scanner\",\"neuvector.role\":\"scanner\",\"neuvector.vuln_db\":\"3.103\"}},\"container\":\"fca6fbfebfbb426547d0ae5582476b712a465a5b0fab7eb1f0e1edeaa21d5380\",\"container_config\":{\"Hostname\":\"fca6fbfebfbb\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"ENTRYPOINT [\\\"/usr/local/bin/monitor\\\"]\"],\"Image\":\"sha256:0da4c3c58068829dbdbe9f828cb35f258b69123bc17b3ef26695e57cb4242f62\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":[\"/usr/local/bin/monitor\"],\"OnBuild\":[],\"Labels\":{\"neuvector.image\":\"neuvector/scanner\",\"neuvector.role\":\"scanner\",\"neuvector.vuln_db\":\"3.103\"}},\"created\":\"2023-06-02T12:53:43.271058702Z\",\"docker_version\":\"17.09.0-ce\",\"id\":\"6a2a5427720d97d6456549394e563fc4d50a1ccb7839fc28cd79feec6b812380\",\"os\":\"linux\",\"parent\":\"8e727aa727ca79c0c33d97728ff9559f24c145f173914cb50f2bcda641030a5b\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"8e727aa727ca79c0c33d97728ff9559f24c145f173914cb50f2bcda641030a5b\",\"parent\":\"f7a05e596fb3f5d88b07ece552a871db4424a21c902df1ac74133dc27c41d1ac\",\"created\":\"2023-06-02T12:53:42.585912173Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  LABEL neuvector.image=neuvector/scanner neuvector.role=scanner neuvector.vuln_db=3.103\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"f7a05e596fb3f5d88b07ece552a871db4424a21c902df1ac74133dc27c41d1ac\",\"parent\":\"9256b5a3f74027bdd6e1d72c1cd9a91b031cd6907022069b8b353b84776c1519\",\"created\":\"2023-06-02T12:53:41.844272475Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:536f975cb41d6feaacd31968c9e960b225c5d5a68175b654d5c2e136e1d099f4 in / \"]}}"
      },
      {
         "v1Compatibility": "{\"id\":\"9256b5a3f74027bdd6e1d72c1cd9a91b031cd6907022069b8b353b84776c1519\",\"parent\":\"dfaa4fcbbad92fde1a10b7b2147c9c458f9efaff77b6d0219d061d409f1c9416\",\"created\":\"2023-05-31T21:55:50.012020821Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c apk --update add ca-certificates \\u0026\\u0026     apk add --allow-untrusted --force-overwrite /glibc-2.33-r0.apk \\u0026\\u0026 apk update \\u0026\\u0026     rm -rf /tmp/* \\u0026\\u0026 rm -rf /var/cache/apk/* /glibc-2.33-r0.apk\"]},\"author\":\"[email protected]\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"dfaa4fcbbad92fde1a10b7b2147c9c458f9efaff77b6d0219d061d409f1c9416\",\"parent\":\"d778fbceed4c84eef67647cd3c5538fd1ddb5a05b378e97a136e6bba627572be\",\"created\":\"2023-05-31T21:55:48.000087342Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:cb6719ac57225f4bc06b66e8685c880d15d2d7fb2182b5f790ee17a5e846a1a8 in / \"]},\"author\":\"[email protected]\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"d778fbceed4c84eef67647cd3c5538fd1ddb5a05b378e97a136e6bba627572be\",\"parent\":\"79428a50fdbf5ab3d67d705502e6644661fa230f39016422d1d0da51dde2754e\",\"created\":\"2023-05-31T21:55:46.676113854Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  MAINTAINER [email protected]\"]},\"author\":\"[email protected]\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"79428a50fdbf5ab3d67d705502e6644661fa230f39016422d1d0da51dde2754e\",\"parent\":\"149414ad771db6217a7e94adc1a8a85f96ba1b8a7deed38f48f22ee1b82e459b\",\"created\":\"2023-05-09T23:11:10.132147526Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  CMD [\\\"/bin/sh\\\"]\"]},\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"149414ad771db6217a7e94adc1a8a85f96ba1b8a7deed38f48f22ee1b82e459b\",\"created\":\"2023-05-09T23:11:10.007217553Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:7625ddfd589fb824ee39f1b1eb387b98f3676420ff52f26eb9d975151e889667 in / \"]}}"
      }
   ],
   "signatures": [
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "25MK:WUPN:DR2L:F7DQ:IWRD:FNMW:QXIM:WMXB:K5LB:2CR3:W65S:QTAH",
               "kty": "EC",
               "x": "Y6ogZUxj3GK6Yy5pMsztE0XAb9bDDFsSfwNM4_JYnkc",
               "y": "boRv58uUrrbhpLzY1nv3SNTKlOsU6Ggj6mWX6B46WCc"
            },
            "alg": "ES256"
         },
         "signature": "R6Yv5ZntXrFxvLS0O7oTmMnviGVr6NW64J4Mw1lwEOT4D44kjRdpweJtMthozHhTOZ41JNroriyuV0JvCqxa2A",
         "protected": "eyJmb3JtYXRMZW5ndGgiOjU1NzEsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAyMy0wNi0wNVQxMTozMDozMVoifQ"
      }
   ]
}
Error: pulling docker.io/neuvector/scanner:latest: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

Referring to issue #377, I executed the following command for troubleshooting.
The result looks fine to me, and it seems to tell the opposite, that the schema version of docker image is actually V2, not V1.

crane --verbose manifest docker.io/neuvector/scanner:latest

[root@node ~]# crane --verbose manifest "$src"
2023/06/05 19:33:20 --> GET https://index.docker.io/v2/
2023/06/05 19:33:20 GET /v2/ HTTP/1.1
Host: index.docker.io
User-Agent: crane/0.15.2 go-containerregistry/0.15.2
Accept-Encoding: gzip


2023/06/05 19:33:21 <-- 401 https://index.docker.io/v2/ (761.534481ms)
2023/06/05 19:33:21 HTTP/1.1 401 Unauthorized
Content-Length: 87
Content-Type: application/json
Date: Mon, 05 Jun 2023 11:33:20 GMT
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=31536000
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

2023/06/05 19:33:21 --> GET https://auth.docker.io/token?scope=repository%3Aneuvector%2Fscanner%3Apull&service=registry.docker.io [body redacted: basic token response contains credentials]
2023/06/05 19:33:21 GET /token?scope=repository%3Aneuvector%2Fscanner%3Apull&service=registry.docker.io HTTP/1.1
Host: auth.docker.io
User-Agent: crane/0.15.2 go-containerregistry/0.15.2
Accept-Encoding: gzip


2023/06/05 19:33:22 <-- 200 https://auth.docker.io/token?scope=repository%3Aneuvector%2Fscanner%3Apull&service=registry.docker.io (1.53910375s) [body redacted: basic token response contains credentials]
2023/06/05 19:33:22 HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Date: Mon, 05 Jun 2023 11:33:22 GMT
Strict-Transport-Security: max-age=31536000
X-Trace-Id: d53164e699edbadb7ececf46710250ed


2023/06/05 19:33:22 --> GET https://index.docker.io/v2/neuvector/scanner/manifests/latest
2023/06/05 19:33:22 GET /v2/neuvector/scanner/manifests/latest HTTP/1.1
Host: index.docker.io
User-Agent: crane/0.15.2 go-containerregistry/0.15.2
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Authorization: <redacted>
Accept-Encoding: gzip


2023/06/05 19:33:23 <-- 200 https://index.docker.io/v2/neuvector/scanner/manifests/latest (984.650335ms)
2023/06/05 19:33:23 HTTP/1.1 200 OK
Content-Length: 1163
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Mon, 05 Jun 2023 11:33:22 GMT
Docker-Content-Digest: sha256:57397b2d0e5f49447e4148c35e974ca02f06c19eebe2b7d5dcec19dcca1252aa
Docker-Distribution-Api-Version: registry/2.0
Docker-Ratelimit-Source: 117.50.122.14
Etag: "sha256:57397b2d0e5f49447e4148c35e974ca02f06c19eebe2b7d5dcec19dcca1252aa"
Ratelimit-Limit: 100;w=21600
Ratelimit-Remaining: 69;w=21600
Strict-Transport-Security: max-age=31536000

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 3097,
      "digest": "sha256:1d20924982fe9279e156044124a3e9c8ed6de6bb2fbdb89734db3a7198f22539"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 3397490,
         "digest": "sha256:8a49fdb3b6a5ff2bd8ec6a86c05b2922a0f7454579ecc07637e94dfd1d0639b6"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 4500429,
         "digest": "sha256:f8ab9d0286ee2e40937a52a0bbdd34e1d42bc93b5e29f6be1cb42584f88e6ba8"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 4781418,
         "digest": "sha256:2703044be1479093b493ce6f6a9a70feef9ce93b347dcf104df68933d4791f45"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 124409702,
         "digest": "sha256:056e5a91cf035418524104eba706002cad28aab8ead43d11b073a8374905504d"
      }
   ]
}
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 3097,
      "digest": "sha256:1d20924982fe9279e156044124a3e9c8ed6de6bb2fbdb89734db3a7198f22539"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 3397490,
         "digest": "sha256:8a49fdb3b6a5ff2bd8ec6a86c05b2922a0f7454579ecc07637e94dfd1d0639b6"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 4500429,
         "digest": "sha256:f8ab9d0286ee2e40937a52a0bbdd34e1d42bc93b5e29f6be1cb42584f88e6ba8"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 4781418,
         "digest": "sha256:2703044be1479093b493ce6f6a9a70feef9ce93b347dcf104df68933d4791f45"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 124409702,
         "digest": "sha256:056e5a91cf035418524104eba706002cad28aab8ead43d11b073a8374905504d"
      }
   ]
}

Expected behavior

crane --verbose index filter "$src" --platform linux/amd64 -t "$dest" should be able to copy any legitimate images to a private registry.
I found the command crane copy docker.io/neuvector/scanner:latest "$dest" does work for me, but I was syncing a bunch of images here. crane index filter "$src" --platform linux/amd64 --platform linux/arm64 -t "$dest" works even if the image does not have a linux/arm64 manifest, and I do not want to copy manifest of architectures that I do not need.

Additional context

• Output of crane version

0.15.2

• Registry used (e.g., GCR, ECR, Quay)

docker.io

[jonjohnsonjr]

So docker.io/neuvector/scanner:latest is not a multi-architecture image. It's just a single-architecture linux/amd64 image, which means that çrane index filter doesn't really make sense to use with it (because it's not an index).

The reason we get this weird error message is that we are asking for an index (via Accept headers), and dockerhub seems to automatically downconvert to schema 1 in that case... I can update this to make the error message a little less obtuse.

[IceCodeNew]

So docker.io/neuvector/scanner:latest is not a multi-architecture image. It's just a single-architecture linux/amd64 image, which means that çrane index filter doesn't really make sense to use with it (because it's not an index).

The reason we get this weird error message is that we are asking for an index (via Accept headers), and dockerhub seems to automatically downconvert to schema 1 in that case... I can update this to make the error message a little less obtuse.

Hi there, I appreciate your prompt response. And thank you so much for explaining the real problem to me.