Docker push fails with "failed to copy: io: read/write on closed pipe"

[teor2345]

Motivation

Docker image builds are regularly failing due to a server error. This impacts all the Docker builds.

It is currently happening multiple times per day.

Here is an example log:

#27 exporting to image
#27 pushing layers 2.0s done
#27 ERROR: failed to push us-docker.pkg.dev/zealous-zebra/zebra/zebrad-test:pr-5952: failed to copy: io: read/write on closed pipe
------
 > importing cache manifest from us-docker.pkg.dev/zealous-zebra/zebra/zebrad-test:5952-merge-cache:
------
------
 > exporting to image:
------
ERROR: failed to solve: failed to push us-docker.pkg.dev/zealous-zebra/zebra/zebrad-test:pr-5952: failed to copy: io: read/write on closed pipe
Error: buildx failed with: ERROR: failed to solve: failed to push us-docker.pkg.dev/zealous-zebra/zebra/zebrad-test:pr-5952: failed to copy: io: read/write on closed pipe

https://github.com/ZcashFoundation/zebra/actions/runs/3912882750/jobs/6688092494#step:9:2059

This could either be a network error, a bug in our workflows or actions, or a configuration problem on the Docker server or our Docker account.

Testing

We'll need to run any fix a few times in CI to be sure it fixes the problem.

[teor2345]

Also failed in the merge queue #5962:
https://github.com/ZcashFoundation/zebra/actions/runs/3926053727/jobs/6711423682#step:9:951

[teor2345]

Also failed in #5952:
https://github.com/ZcashFoundation/zebra/actions/runs/3912882750/jobs/6715351895

[mpguerra]

@gustavovalverde have we changed anything with our Docker builds/configuration lately?

[gustavovalverde]

Sadly, we haven't changed anything related to builds, just deleting extra images from the registry, but this is failing even we the cache image is found, so it doesn't seem to be related. This started happening after a GCP outage in the network, but the outage was solved and this is still happening.

I've been looking at this while working on ci: add a test to validate Zebra's config file and path
#5968

Once I'm done with that I'll deep dive into this one.

[mpguerra]

Sadly, we haven't changed anything related to builds, just deleting extra images from the registry, but this is failing even we the cache image is found, so it doesn't seem to be related. This started happening after a GCP outage in the network, but the outage was solved and this is still happening.

Let's contact GCP support then

[teor2345]

This seems to be failing in PR #5992, both on the PR and in the merge queue:

Error: buildx failed with: ERROR: failed to solve: failed to push us-docker.pkg.dev/zealous-zebra/zebra/zebrad-test:pr-5997: failed to copy: io: read/write on closed pipe

https://github.com/ZcashFoundation/zebra/actions/runs/3945475841/jobs/6752400182#step:9:2108
https://github.com/ZcashFoundation/zebra/actions/runs/3953457309/jobs/6769753452#step:9:2118

There is also a new warning:

Warning: Unexpected input(s) 'driver-opts', valid inputs are ['add-hosts', 'allow', 'attests', 'build-args', 'build-contexts', 'builder', 'cache-from', 'cache-to', 'cgroup-parent', 'context', 'file', 'labels', 'load', 'network', 'no-cache', 'no-cache-filters', 'outputs', 'platforms', 'provenance', 'pull', 'push', 'sbom', 'secrets', 'secret-files', 'shm-size', 'ssh', 'tags', 'target', 'ulimit', 'github-token']

It seems like we need a different way of setting driver-opts, or a version of docker/build-push-action that supports the driver-opts key.