Cache manifest lists can't be exported to gcr

[tonistiigi]

Related to #720 that currently has a gcr specific workaround in the pull code.

When exporting a manifest list that contains the cache metadata for a build (eg. with mode max), the upload fails in gcr with 400 error.

There was a report in slack that it also failed with 404 https://dockercommunity.slack.com/archives/C7S7A40MP/p1566224768282100 (hence similarities with #720) but couldn't repro that.

#7 ERROR: error writing manifest blob: failed commit on ref "sha256:813b455d58cf597f96c8f20d04ae670127a94cd4786f14da09fef88e97bab090": unexpected status: 400 Bad Request
------
 > exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:813b455d58cf597f96c8f20d04ae670127a94cd4786f14da09fef88e97bab090": unexpected status: 400 Bad Request

After pushing all the blobs manifest list push fails with:

request:

PUT https://gcr.io/v2/.../hello/manifests/cache HTTP/2.0
                        ← 400 application/json 190b 197ms

:authority:       gcr.io
content-type:     application/vnd.docker.distribution.manifest.list.v2+json
content-length:   935
accept-encoding:  gzip
user-agent:       Go-http-client/2.0
authorization:    Bearer ....

{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.list.v2+json","manifests":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa","size":2789742,"annotations":{"buildkit/createdat":"2019-08-14T17:52:26.789506223-07:00","containerd.io/uncompressed":"sha256:1bfeebd65323b8ddf5bd6a51cc7097b72788bc982e9ab3280d53d3c613adffa7"}},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:a8ce4bba72fb43cbaeb05eca5d6d682696b915c53a3519d0f00f5aec063c0ae9","size":219,"annotations":{"buildkit/createdat":"2019-08-19T14:39:20.018002388-07:00","containerd.io/uncompressed":"sha256:72e39bafb519d4c8b9d12597a538667085627a6b8c3c3f085d319d3ef6955b4f"}},{"mediaType":"application/vnd.buildkit.cacheconfig.v0","digest":"sha256:02656190e9941165d11c1ac96d683c3da0cec10714e01ed4de4b6ded7e8b7c49","size":565}]}


response:

docker-distribution-api-versio  registry/2.0
n:
content-type:                   application/json
content-length:                 190
content-encoding:               gzip
date:                           Mon, 19 Aug 2019 21:39:36 GMT
server:                         Docker Registry
cache-control:                  private
x-xss-protection:               0
x-frame-options:                SAMEORIGIN
alt-svc:                        quic=":443"; ma=2592000; v="46,43,39"
[decoded gzip] JSON
{
    "errors": [
        {
            "code": "MANIFEST_INVALID",
            "message": "Failed to parse manifest for request \"/v2/.../hello/manifests/cache\": Failed to deserialize application/vnd.docker.distribution.manifest.list.v2+json."
        }
    ]
}

@mattmoor @dmcgowan

[tonistiigi]

cc @jonjohnsonjr

[jonjohnsonjr]

I haven't worked on the project in a while, but, I can tell you that GCR is very strict with what it will accept in a few places... especially here.

The annotations field isn't present for manifest lists, which is where I'm guessing this is falling over.

You might have better luck if you set the mediaType to an image index, which specifies annotations, but I'm guessing that will also break because the descriptors reference mediaTypes we don't recognize ("application/vnd.buildkit.cacheconfig.v0") or expect in an index ("application/vnd.docker.image.rootfs.diff.tar.gzip").

The guidance to ignore unrecognized mediaTypes was added relatively recently (well after I added image index support to GCR). I'm pretty sure we only expect an image index to reference an OCI manifest:

This descriptor property has additional restrictions for manifests. Implementations MUST support at least the following media types:

• application/vnd.oci.image.manifest.v1+json

I'm pretty sure we'll fail for nested indexes as well, even though we SHOULD support it:

Also, implementations SHOULD support the following media types:

• application/vnd.oci.image.index.v1+json (nested index)

Image indexes concerned with portability SHOULD use one of the above media types. Future versions of the spec MAY use a different mediatype (i.e. a new versioned format). An encountered mediaType that is unknown to the implementation MUST be ignored.

cc @yolo3301

[tonistiigi]

@jonjohnsonjr

The error changed when switching to oci types(can't do this by default as would limit the number of supported registries).

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa","size":2789742,"annotations":{"buildkit/createdat":"2019-08-14T17:52:26.789506223-07:00","containerd.io/uncompressed":"sha256:1bfeebd65323b8ddf5bd6a51cc7097b72788bc982e9ab3280d53d3c613adffa7"}},{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip","digest":"sha256:3e13b0fe77d9ae8d6063010ee4c90b83a68f80667ad043511019dd8cb6555683","size":216,"annotations":{"buildkit/createdat":"2019-08-19T16:05:43.171937647-07:00","containerd.io/uncompressed":"sha256:c4750913f7411f0f9363089280b2af3699486e761ebca513f35e7d31c3b6e80f"}},{"mediaType":"application/vnd.buildkit.cacheconfig.v0","digest":"sha256:3eebdbf306be5643e5e3eba7dc4874697e2ddcba11ed3e91802b4332b8d69574","size":565}]}


{
    "errors": [
        {
            "code": "MANIFEST_INVALID",
            "message": "Failed to parse manifest for request \"/v2/.../hello/manifests/cache\": Image index child manifest not supported: application/vnd.oci.image.layer.v1.tar+gzip."
        }
    ]
}

fyi, similar manifest lists are also used in containerd checkpoints https://github.com/containerd/containerd/blob/c6da899e2fd5ec5f0863d4be5ff683ee785d155c/images/mediatypes.go#L34

[jonjohnsonjr]

The error changed when switching to oci types(can't do this by default as would limit the number of supported registries).

Do most registries work with this?

[tonistiigi]

oci support was added in version 2.7 to reference registry https://github.com/docker/distribution/releases/tag/v2.7.0 . There is also limited support among runtimes for oci mediatypes support so not only registry side issue. For buildkit side, we can allow oci mediatypes in cache export if user requests but as shown in #1143 (comment) it doesn't help with that case.

[jonjohnsonjr]

I meant do most registries allow this:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa",
      "size": 2789742,
      "annotations": {
        "buildkit/createdat": "2019-08-14T17:52:26.789506223-07:00",
        "containerd.io/uncompressed": "sha256:1bfeebd65323b8ddf5bd6a51cc7097b72788bc982e9ab3280d53d3c613adffa7"
      }
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:a8ce4bba72fb43cbaeb05eca5d6d682696b915c53a3519d0f00f5aec063c0ae9",
      "size": 219,
      "annotations": {
        "buildkit/createdat": "2019-08-19T14:39:20.018002388-07:00",
        "containerd.io/uncompressed": "sha256:72e39bafb519d4c8b9d12597a538667085627a6b8c3c3f085d319d3ef6955b4f"
      }
    },
    {
      "mediaType": "application/vnd.buildkit.cacheconfig.v0",
      "digest": "sha256:02656190e9941165d11c1ac96d683c3da0cec10714e01ed4de4b6ded7e8b7c49",
      "size": 565
    }
  ]
}

as a manifest? I'd be really surprised.

[tonistiigi]

Depends on what you mean by most. Every registry based on https://github.com/docker/distribution allows it without any updates. Docker hub didn't require any updates. I know Azure Container Registry works fine as well. ECR otoh still doesn't support any variant of manifest lists afaik. The same pattern is used for example in containerd checkpoints definition and cnab-oci (that is used by docker app for example).

[jonjohnsonjr]

It just seems to me that anyone implementing manifest list support based on reading https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list would probably reject that manifest.

[howardjohn]

I get almost the same thing but a 401 error, so maybe slightly different.

Uncahced:

[+] Building 9.6s (8/8) FINISHED
 => [internal] load .dockerignore                                                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                                                0.0s
 => [internal] load build definition from Dockerfile.pilot                                                                                                                                                     0.0s
 => => transferring dockerfile: 805B                                                                                                                                                                           0.0s
 => [internal] load metadata for docker.io/istio/base:1.5-dev.1                                                                                                                                                0.6s
 => [internal] load build context                                                                                                                                                                              0.9s
 => => transferring context: 71.21MB                                                                                                                                                                           0.9s
 => CACHED [default 1/1] FROM docker.io/istio/base:1.5-dev.1@sha256:c441777c138ea3d94de04bd115187eb122d51c9257b25e34b0c994eb7569f37f                                                                           0.0s
 => => resolve docker.io/istio/base:1.5-dev.1@sha256:c441777c138ea3d94de04bd115187eb122d51c9257b25e34b0c994eb7569f37f                                                                                          0.0s
 => [stage-2 1/2] COPY pilot-discovery /usr/local/bin/                                                                                                                                                         0.3s
 => [stage-2 2/2] COPY cacert.pem /cacert.pem                                                                                                                                                                  0.0s
 => exporting to image                                                                                                                                                                                         7.6s
 => => exporting layers                                                                                                                                                                                        2.8s
 => => exporting manifest sha256:8408e32807f8314cedbde0883ec09f3321058d17e32a7ed9a894682b79aad7bf                                                                                                              0.0s
 => => exporting config sha256:e15a45ee64eca4a8225421e627bbdbe6102b0a0430819de9cfd3f93e714a74d4                                                                                                                0.0s
 => => pushing layers                                                                                                                                                                                          3.9s
 => => pushing manifest for gcr.io/howardjohn-istio/pilot:1578078119

cached:

[+] Building 2.3s (8/8) FINISHED
 => [internal] load build definition from Dockerfile.pilot                                                                                                                                                     0.0s
 => => transferring dockerfile: 805B                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                                                0.0s
 => [internal] load metadata for docker.io/istio/base:1.5-dev.1                                                                                                                                                0.9s
 => [default 1/1] FROM docker.io/istio/base:1.5-dev.1@sha256:c441777c138ea3d94de04bd115187eb122d51c9257b25e34b0c994eb7569f37f                                                                                  0.0s
 => => resolve docker.io/istio/base:1.5-dev.1@sha256:c441777c138ea3d94de04bd115187eb122d51c9257b25e34b0c994eb7569f37f                                                                                          0.0s
 => [internal] load build context                                                                                                                                                                              0.8s
 => => transferring context: 71.21MB                                                                                                                                                                           0.7s
 => CACHED [stage-2 1/2] COPY pilot-discovery /usr/local/bin/                                                                                                                                                  0.0s
 => CACHED [stage-2 2/2] COPY cacert.pem /cacert.pem                                                                                                                                                           0.0s
 => ERROR exporting to image                                                                                                                                                                                   0.6s
 => => exporting layers                                                                                                                                                                                        0.0s
 => => exporting manifest sha256:8408e32807f8314cedbde0883ec09f3321058d17e32a7ed9a894682b79aad7bf                                                                                                              0.0s
 => => exporting config sha256:e15a45ee64eca4a8225421e627bbdbe6102b0a0430819de9cfd3f93e714a74d4                                                                                                                0.0s
 => => pushing layers                                                                                                                                                                                          0.1s
 => => pushing manifest for gcr.io/howardjohn-istio/pilot:1578078179                                                                                                                                           0.4s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:8408e32807f8314cedbde0883ec09f3321058d17e32a7ed9a894682b79aad7bf": unexpected status: 401 Unauthorized

[TBBle]

ECR now has support for manifest-lists (aws/containers-roadmap#505), but is probably also bouncing off this issue and rejecting buildkit cache manifest-lists.

[hidai]

Any update on this? I can't --export-cache type=registry,ref= to GCR neither mode=min nor max and neither OCI worker nor containerd.

error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:******": unexpected status: 400 Bad Request

[jonjohnsonjr]

Let's say theoretically I wanted to fix this :) How can I reliably reproduce this failure?

[jbielick]

For reproducing the 400 when exporting the manifest:

Prerequisites:

• Docker CLI experimental features enabled
• docker buildx create --use
• Docker credentials (or credential helper) for Google Cloud Container Registry (roles/storage.admin would do)

There are a few mentions of 401 and 403 responses—at a glance those seem like actual permission issues.

Cache export encounters error:

docker buildx build \
  --cache-to=type=registry,ref=us.gcr.io/yourproject/yourrepo:tag \
  -t buildxtest - \
  <<< $(echo "FROM alpine:latest")

You should see a 400 Bad Request when exporting cache:

[+] Building 1.4s (5/5) FINISHED                                                                                               
 => [internal] load .dockerignore                                                                                         0.0s
 => => transferring context: 2B                                                                                           0.0s
 => [internal] load build definition from Dockerfile                                                                      0.0s
 => => transferring dockerfile: 56B                                                                                       0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                          1.0s
 => CACHED [1/1] FROM docker.io/library/alpine:latest@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841  0.0s
 => => resolve docker.io/library/alpine:latest@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a    0.0s
 => ERROR exporting cache                                                                                                 0.4s
 => => preparing build cache for export                                                                                   0.0s
 => => writing config sha256:3421f7614aa60ab573777aec27d7d447103f640aac396e029e33d37af5055aaa                             0.4s
------
 > exporting cache:
------
failed to solve: rpc error: code = Unknown desc = error writing config blob: failed to open writer: unexpected response: 400 Bad Request

For what it's worth, I've also tried exporting cache (max) to Google Cloud Artifact Registry (beta) with no success.

[crazy-max]

Looks related to docker/build-push-action#171. Can you use the following command to create a new builder and try again?:

docker buildx create --name builder --driver docker-container --driver-opt image=moby/buildkit:v0.8-beta --use

[jbielick]

gcr.io: not working, probably won't

us.gcr.io and docker buildx create --use

...
------
 > exporting cache:
------
failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:8ae820ecd9b90df06294e862b8d38c5fcfd7ebd22d2fe1ced90ddf07868c0d81": unexpected status: 400 Bad Request

us.gcr.io and docker buildx create --name builder --driver docker-container --driver-opt image=moby/buildkit:v0.8-beta --use

...
------
 > exporting cache:
------
failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:8ae820ecd9b90df06294e862b8d38c5fcfd7ebd22d2fe1ced90ddf07868c0d81": unexpected status: 400 Bad Request

---

pkg.dev: working

ℹ️ Requires format: `{projectid}/{docker_repository}/{image_repository}

us-east1-docker.pkg.dev/proj/repo/cache and docker buildx create --use

[+] Building 2.2s (5/5) FINISHED   

us-east1-docker.pkg.dev/proj/repo/cache and docker buildx create --name builder --driver docker-container --driver-opt image=moby/buildkit:v0.8-beta --use

[+] Building 3.1s (5/5) FINISHED    

@crazy-max similar to this comment about the missing third path segment, that fixed it for me as well on pkg.dev. Interesting! Thanks for the help 🙂 Overall I think this means pkg.dev is what one should use if expecting remote caching to work with GCP, right? Caveat: it is in beta and all.

[crazy-max]

@jbielick

similar to this comment about the missing third path segment, that fixed it for me as well on pkg.dev. Interesting! Thanks for the help 🙂 Overall I think this means pkg.dev is what one should use if expecting remote caching to work with GCP, right? Caveat: it is in beta and all.

AFAIK Google suggests to switch to GAR:

Google Artifact Registry is the evolution of Google Container Registry. As a fully-managed service with support for both container images and non-container artifacts. If you currently use Google Container Registry, use the information on this page to learn about transitioning to Google Artifact Registry.

cc. @jonjohnsonjr

[tonistiigi]

If you get unexpected status: 400 Bad Request you can create buildx builder with --buildkitd-flags '--debug' and then in the container logs you should have a better error. The message in the failed response body does not propagate to the error atm docker/build-push-action#171 (comment) .

[jonjohnsonjr]

For reproducing the 400 when exporting the manifest

Thanks!

For what it's worth, I've also tried exporting cache (max) to Google Cloud Artifact Registry (beta) with no success.

This is really frustrating, because we return a structured error describing exactly the issue, but containerd swallows it: docker/build-push-action#171 (comment)

gcr.io: not working, probably won't

Hang tight.

Caveat: it is in beta and all.

This changed recently! https://cloud.google.com/blog/products/devops-sre/artifact-registry-is-ga

[mjgallag]

GitHub Container Registry doesn't work either, see #1865.

[gajus]

Am I reading this correct that cache export does not work with GAR?

[mjgallag]

@gajus I can confirm cache export does work with Google Artifact Registry.

[gajus]

Perhaps I am misunderstanding something, but even though I have it configured as such:

buildctl-daemonless.sh build \
  --frontend=dockerfile.v0 \
  --local context=. \
  --local dockerfile=$CI_PROJECT_DIR \
  --opt build-arg:CWA_API_URL="$CWA_API_URL" \
  --opt build-arg:SENTRY_AUTH_TOKEN="$SENTRY_AUTH_TOKEN" \
  --opt build-arg:SENTRY_ORG="$SENTRY_ORG" \
  --opt build-arg:SENTRY_PROJECT="$SENTRY_PROJECT" \
  --opt build-arg:SENTRY_RELEASE="$CI_COMMIT_SHORT_SHA" \
  --export-cache type=registry,ref=$GAR_REGISTRY_IMAGE_PRODUCTION:buildcache,mode=max \
  --import-cache type=registry,ref=$GAR_REGISTRY_IMAGE_PRODUCTION:buildcache \
  --output type=image,name=$GAR_REGISTRY_IMAGE_PRODUCTION:$CI_COMMIT_SHORT_SHA,push=true

It seems like cache is still not being used:

#4 importing cache manifest from us-west2-docker.pkg.dev/contrawork/contra-web-app/production:buildcache
#4 sha256:3a3935c18cb6152d5f3f554704e152251ca50941488e4aff53f2335dfa2be2b3
#4 ERROR: us-west2-docker.pkg.dev/contrawork/contra-web-app/production:buildcache: not found

Looking at the build logs, cache export completes suspiciously fast:

#14 exporting to image
#14 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#14 exporting layers done
#14 exporting manifest sha256:87773f06788bc5bbc0dedf51fd443e9d7b65be1a5fd4a1b052e5a27f38bb88e1 done
#14 exporting config sha256:85e43cc7e13d315828be8007eee197eeecb84a0659051dd1c2918fbbda1f17ec
#14 exporting config sha256:85e43cc7e13d315828be8007eee197eeecb84a0659051dd1c2918fbbda1f17ec done
#14 pushing layers
#14 ...
#15 [auth] contrawork/contra-web-app/production:pull,push token for us-west2-docker.pkg.dev
#15 sha256:f5df065c25e33c9eb605236c85bee05010c5d82c3c7f90393aa599006c0c27c0
#15 DONE 0.0s
#14 exporting to image
#14 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#14 pushing layers 7.8s done
#14 pushing manifest for us-west2-docker.pkg.dev/contrawork/contra-web-app/production:3e0c41df
#14 pushing manifest for us-west2-docker.pkg.dev/contrawork/contra-web-app/production:3e0c41df 0.2s done
#14 DONE 8.1s
#16 exporting cache
#16 sha256:2700d4ef94dee473593c5c614b55b2dedcca7893909811a8f2b48291a1f581e4
#16 preparing build cache for export 0.0s done
#16 DONE 0.0s
------
 > importing cache manifest from us-west2-docker.pkg.dev/contrawork/contra-web-app/production:buildcache:

[gajus]

After a few more trials, it seems to be completely random when it works and when it does not.

When it works, it completes in a few seconds. When it doesn't... Is this a known issue?

[max-sixty]

@gajus is that a Google Artifact Repository issue or a Docker issue?

[gajus]

@gajus is that a Google Artifact Repository issue or a Docker issue?

I had the same issue with GitLab's registry too, so I doubt it is GAR issue.

[frapeshka]

I have same issue with Artifactory Docker Registry.

2021-07-01T12:01:05.363Z �[1;32m[jfrt ]�[0;39m �[1;31m[ERROR]�[0;39m [5e45cfc06a15a746] [h.DockerManifestPutHandler:282] [http-nio-8081-exec-8] - Error uploading manifest: 'N/A (through reference chain: org.jfrog.repomd.docker.manifest.ManifestListJson["manifests"]->java.util.ArrayList[0]->org.jfrog.repomd.docker.manifest.ManifestListJson$ManifestJson["annotations"])'

[pvlltvk]

Hi everyone!

Can someone clarify the difference between a common docker image manifest and the one that buildkit creates for cache?
I see that some registries still haven't supported cache images that buildkit exports and I think that the main reason is the same.

[tonistiigi]

#1143 (comment)

[JCMais]

So the fix here is basically to not use GCR and use GAR instead?

Edit: I can confirm that after switching to GAR the error is gone.

[thclark]

I know this is closed, but leaving it here for people experiencing similar errors when porting GCR to GAR:

It's worth noting that there is now an additional element in the repository path... You can experience this 400 BAD_REQUEST error when you don't add that in. For example:

docker buildx build -f ./Dockerfile --push -t gcr.io/myproject/myapp:latest .

Works fine, but creating a GAR repository called myapp and simply switching the base address...

docker buildx build -f ./Dockerfile --push -t europe-west1-docker.pkg.dev/myproject/myapp:latest .

Will give a BAD_REQUEST that looks a lot like this error. You need to specify the container name within the repository:

docker buildx build -f ./Dockerfile --push -t europe-west1-docker.pkg.dev/myproject/myapp/mycontainer:latest .

Happy artifacting 😎

[georgeseifada]

I'm running into the same issue. Was anyone able to get it to work for caching with ECR?

[fw-jcarrillo]

Do not close this issue, it still happening

[flimzy]

GCR is deprecated, and will be shut down in 2024.

[Hronom]

Same issue for ECR.

I tried provenance=false, oci-mediatypes=true,image-manifest=true, not works.

[OJFord]

Should really be called out in the readme that 'here's how to push cache to registry theoretically, but by the way it basically doesn't work with any of the popular registries' imo.

It does work with ECR, it needs image-manifest=true.