Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

400 BAD REQUEST (quay.io) #3222

Closed
ghost opened this issue Oct 25, 2022 · 3 comments
Closed

400 BAD REQUEST (quay.io) #3222

ghost opened this issue Oct 25, 2022 · 3 comments
Labels
kind/external

Comments

@ghost
Copy link

ghost commented Oct 25, 2022 •
edited by ghost
Loading

Using the command as shown as rootless buildkit.

buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=quay.io/<org>/<servicename>:<tag>,push=true --export-cache type=registry,ref=quay.io/<org>/<servicename>,mode=max --import-cache type=registry,ref=quay.io/<org>/<servicename>

gives an error as shown:

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 253B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/golang:1.13
#3 DONE 1.3s
#4 [internal] load build context
#4 DONE 0.0s
#5 [1/6] FROM docker.io/library/golang:1.13@sha256:8ebb6d5a48deef738381b56b1d4cd33d99a5d608e0d03c5fe8dfa3f68d41a1f8
#5 resolve docker.io/library/golang:1.13@sha256:8ebb6d5a48deef738381b56b1d4cd33d99a5d608e0d03c5fe8dfa3f68d41a1f8 done
#5 DONE 0.0s
#6 importing cache manifest from quay.io/<org>/autodevops-test-app
#6 ...
#7 [auth] <org>/autodevops-test-app:pull token for quay.io
#7 DONE 0.0s
#6 importing cache manifest from quay.io/<org>/autodevops-test-app
#6 DONE 1.2s
#4 [internal] load build context
#4 transferring context: 111.94kB 0.0s done
#4 DONE 0.0s
#5 [1/6] FROM docker.io/library/golang:1.13@sha256:8ebb6d5a48deef738381b56b1d4cd33d99a5d608e0d03c5fe8dfa3f68d41a1f8
#5 sha256:16b5038bccc853e96f534bc85f4f737109ef37ad92d877b54f080a3c86b3cb3a 126B / 126B 0.1s done
#5 sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e 7.34MB / 120.17MB 0.2s
#5 sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551 0B / 68.67MB 0.2s
#5 sha256:80931cf6881673fd161a3fd73e8971fe4a569fd7fbb44e956d261ca58d97dfab 0B / 51.83MB 0.2s
#5 sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e 36.70MB / 120.17MB 0.3s
#5 sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551 9.44MB / 68.67MB 0.3s
#5 sha256:80931cf6881673fd161a3fd73e8971fe4a569fd7fbb44e956d261ca58d97dfab 23.07MB / 51.83MB 0.3s
#5 sha256:edaf0a6b092f5673ec05b40edb606ce58881b2f40494251117d31805225ef064 0B / 10.00MB 0.2s
#5 sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e 65.01MB / 120.17MB 0.5s
#5 sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551 22.02MB / 68.67MB 0.5s
#5 sha256:80931cf6881673fd161a3fd73e8971fe4a569fd7fbb44e956d261ca58d97dfab 51.83MB / 51.83MB 0.5s
#5 sha256:edaf0a6b092f5673ec05b40edb606ce58881b2f40494251117d31805225ef064 10.00MB / 10.00MB 0.4s done
#5 sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e 96.25MB / 120.17MB 0.6s
#5 sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551 34.60MB / 68.67MB 0.6s
#5 sha256:80931cf6881673fd161a3fd73e8971fe4a569fd7fbb44e956d261ca58d97dfab 51.83MB / 51.83MB 0.6s done
#5 sha256:c958d65b3090aefea91284d018b2a86530a3c8174b72616c4e76993c696a5797 7.81MB / 7.81MB 0.2s done
#5 sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e 120.17MB / 120.17MB 0.8s
#5 sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551 51.38MB / 68.67MB 0.8s
#5 sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7 9.44MB / 50.40MB 0.2s
#5 sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551 66.06MB / 68.67MB 0.9s
#5 sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7 22.02MB / 50.40MB 0.3s
#5 sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e 120.17MB / 120.17MB 1.1s done
#5 sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7 50.40MB / 50.40MB 0.5s
#5 sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551 68.67MB / 68.67MB 1.3s done
#5 sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7 50.40MB / 50.40MB 0.8s done
#5 extracting sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7
#5 extracting sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7 2.0s done
#5 extracting sha256:c958d65b3090aefea91284d018b2a86530a3c8174b72616c4e76993c696a5797
#5 extracting sha256:c958d65b3090aefea91284d018b2a86530a3c8174b72616c4e76993c696a5797 0.8s done
#5 extracting sha256:edaf0a6b092f5673ec05b40edb606ce58881b2f40494251117d31805225ef064
#5 extracting sha256:edaf0a6b092f5673ec05b40edb606ce58881b2f40494251117d31805225ef064 0.3s done
#5 extracting sha256:80931cf6881673fd161a3fd73e8971fe4a569fd7fbb44e956d261ca58d97dfab
#5 extracting sha256:80931cf6881673fd161a3fd73e8971fe4a569fd7fbb44e956d261ca58d97dfab 2.9s done
#5 DONE 7.4s
#5 [1/6] FROM docker.io/library/golang:1.13@sha256:8ebb6d5a48deef738381b56b1d4cd33d99a5d608e0d03c5fe8dfa3f68d41a1f8
#5 extracting sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551
#5 extracting sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551 2.6s done
#5 extracting sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e
#5 extracting sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e 5.8s done
#5 extracting sha256:16b5038bccc853e96f534bc85f4f737109ef37ad92d877b54f080a3c86b3cb3a 0.0s done
#5 DONE 15.9s
#8 [2/6] WORKDIR /app/autodevops-test-app
#8 DONE 0.0s
#9 [3/6] COPY go.mod .
#9 DONE 0.0s
#10 [4/6] RUN go mod download
#10 0.528 go: finding github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
#10 0.531 go: finding github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4
#10 0.534 go: finding github.com/beorn7/perks v1.0.1
#10 0.538 go: finding github.com/cespare/xxhash/v2 v2.1.1
#10 0.541 go: finding github.com/davecgh/go-spew v1.1.1
#10 0.544 go: finding github.com/go-kit/kit v0.9.0
#10 0.548 go: finding github.com/go-logfmt/logfmt v0.4.0
#10 0.551 go: finding github.com/go-stack/stack v1.8.0
#10 0.554 go: finding github.com/gogo/protobuf v1.1.1
#10 0.557 go: finding github.com/golang/protobuf v1.4.2
#10 0.561 go: finding github.com/google/go-cmp v0.4.0
#10 0.564 go: finding github.com/google/gofuzz v1.0.0
#10 0.567 go: finding github.com/json-iterator/go v1.1.10
#10 0.571 go: finding github.com/julienschmidt/httprouter v1.2.0
#10 0.574 go: finding github.com/konsorten/go-windows-terminal-sequences v1.0.1
#10 0.578 go: finding github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
#10 0.582 go: finding github.com/kr/pretty v0.1.0
#10 0.586 go: finding github.com/kr/pty v1.1.1
#10 0.589 go: finding github.com/kr/text v0.1.0
#10 0.593 go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
#10 0.598 go: finding github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
#10 0.605 go: finding github.com/modern-go/reflect2 v1.0.1
#10 0.609 go: finding github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223
#10 0.612 go: finding github.com/pkg/errors v0.8.1
#10 0.615 go: finding github.com/pmezard/go-difflib v1.0.0
#10 0.618 go: finding github.com/prometheus/client_golang v1.7.1
#10 0.622 go: finding github.com/prometheus/client_model v0.2.0
#10 0.625 go: finding github.com/prometheus/common v0.10.0
#10 0.628 go: finding github.com/prometheus/procfs v0.1.3
#10 0.632 go: finding github.com/sirupsen/logrus v1.4.2
#10 0.635 go: finding github.com/stretchr/objx v0.1.1
#10 0.638 go: finding github.com/stretchr/testify v1.4.0
#10 0.642 go: finding golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
#10 0.646 go: finding golang.org/x/net v0.0.0-20190613194153-d28f0bde5980
#10 0.652 go: finding golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e
#10 0.656 go: finding golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1
#10 0.659 go: finding golang.org/x/text v0.3.0
#10 0.662 go: finding golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
#10 0.667 go: finding google.golang.org/protobuf v1.23.0
#10 0.671 go: finding gopkg.in/alecthomas/kingpin.v2 v2.2.6
#10 0.674 go: finding gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15
#10 0.678 go: finding gopkg.in/yaml.v2 v2.2.5
#10 DONE 1.9s
#11 [5/6] COPY . .
#11 DONE 0.1s
#12 [6/6] RUN go build
#12 DONE 4.0s
#13 exporting to image
#13 exporting layers
#13 exporting layers 4.3s done
#13 exporting manifest sha256:6941c3aff45c9c783a507b73b0477d2192821c2f50aa2d50723b00c2d9ad4397 done
#13 exporting config sha256:c85e8cdc109f7314f3d3edb8b9c5c78308a6c91e747ed1e9eab99b7d105bfe3d done
#13 pushing layers
#13 ...
#14 [auth] <org>/autodevops-test-app:pull,push token for quay.io
#14 DONE 0.0s
#13 exporting to image
#13 pushing layers 3.9s done
#13 pushing manifest for quay.io/<org>/autodevops-test-app:ab7d91c1@sha256:6941c3aff45c9c783a507b73b0477d2192821c2f50aa2d50723b00c2d9ad4397
#13 pushing manifest for quay.io/<org>/autodevops-test-app:ab7d91c1@sha256:6941c3aff45c9c783a507b73b0477d2192821c2f50aa2d50723b00c2d9ad4397 0.3s done
#13 DONE 8.5s
#15 exporting cache
#15 preparing build cache for export 0.0s done
#15 writing layer sha256:16b5038bccc853e96f534bc85f4f737109ef37ad92d877b54f080a3c86b3cb3a 0.1s done
#15 writing layer sha256:2fee09a6509a3fc0fbe61ef0a83d8ec64204af669fb92438be5e75b39b8ba241
#15 writing layer sha256:2fee09a6509a3fc0fbe61ef0a83d8ec64204af669fb92438be5e75b39b8ba241 0.1s done
#15 writing layer sha256:4df19829c2995a950484f9babe4bbd9970655483e3742044298a5afee8814e6b 0.1s done
#15 writing layer sha256:518836aa7fab3db6c5d58ba56843874083d6187a632c48a5a967afa1687328d6
#15 writing layer sha256:518836aa7fab3db6c5d58ba56843874083d6187a632c48a5a967afa1687328d6 0.1s done
#15 writing layer sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e
#15 writing layer sha256:799f41bb59c9731aba2de07a7b3d49d5bc5e3a57ac053779fc0e405d3aed0b9e 0.1s done
#15 writing layer sha256:80931cf6881673fd161a3fd73e8971fe4a569fd7fbb44e956d261ca58d97dfab 0.1s done
#15 writing layer sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551
#15 writing layer sha256:813643441356759e9202aeebde31d45192b5e5e6218cd8d2ad216304bf415551 0.1s done
#15 writing layer sha256:c6838f3ee0e9a34da77229c51b6561368d923fbfe83c962cfa4e5b487941ca30
#15 writing layer sha256:c6838f3ee0e9a34da77229c51b6561368d923fbfe83c962cfa4e5b487941ca30 0.1s done
#15 writing layer sha256:c958d65b3090aefea91284d018b2a86530a3c8174b72616c4e76993c696a5797 0.1s done
#15 writing layer sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7
#15 writing layer sha256:d6ff36c9ec4822c9ff8953560f7ba41653b348a9c1136755e653575f58fbded7 0.1s done
#15 writing layer sha256:e80d2666b5d9af9eee27319b9e332adda7cc9a6f8885da92fcec7bdd9f140520
#15 writing layer sha256:e80d2666b5d9af9eee27319b9e332adda7cc9a6f8885da92fcec7bdd9f140520 0.1s done
#15 writing layer sha256:edaf0a6b092f5673ec05b40edb606ce58881b2f40494251117d31805225ef064 0.1s done
#15 writing config sha256:0b3935cdd42db2d0a31d5fd847d40cbc9cdebec143ef60ac00038642c2911de0
#15 writing config sha256:0b3935cdd42db2d0a31d5fd847d40cbc9cdebec143ef60ac00038642c2911de0 1.2s done
#15 writing manifest sha256:3f2dc23319f9dd980226b814ff0875bacea096723895cb4882758c80b51c7c47
#15 writing manifest sha256:3f2dc23319f9dd980226b814ff0875bacea096723895cb4882758c80b51c7c47 0.3s done
#15 ERROR: error writing manifest blob: failed commit on ref "sha256:3f2dc23319f9dd980226b814ff0875bacea096723895cb4882758c80b51c7c47": unexpected status: 400 BAD REQUEST
------
 > exporting cache:
------
error: failed to solve: error writing manifest blob: failed commit on ref "sha256:3f2dc23319f9dd980226b814ff0875bacea096723895cb4882758c80b51c7c47": unexpected status: 400 BAD REQUEST

While manifest pushes to quay.io are successful, every single layer seems to show up in quay.io as part of my tag instead of a single image. I also find that there is always one layer added to the layers from the previous tag.
Any help you can provide would be greatly appreciated.
@jedevc
Copy link
Member

jedevc commented Nov 3, 2022 •
edited
Loading

It looks like there's something in the cache output that quay isn't liking.

This looks similar to #1440 - but just to confirm:

  1. Could you share your BuildKit version? buildctl --version and buildctl debug info
  2. Could you try enabling oci-mediatypes in your --export-cache flag? i.e. --export-cache type=registry,ref=quay.io/<org>/<servicename>,oci-mediatypes=true,mode=max.
    I'd either expect this to "just work", or to return a 415, like in the other issue.
  3. Could you try exporting the cache to a different registry just to confirm if the issue is in quay?

@ghost
Copy link
Author

ghost commented Nov 3, 2022 •
edited by ghost
Loading

@jedevc
For 1. On VM buildkitd --version buildkitd github.com/moby/buildkit v0.10.5 bc26045116045516ff2427201abd299043eaf8f71
On k8s I am using this image: moby/buildkit:master-rootless

For 3. I did export the cache to dockerhub and it works just fine.

I will work on 2 and get back to you

@AkihiroSuda AkihiroSuda changed the title 400 BAD REQUEST 400 BAD REQUEST (quay.io) Nov 4, 2022
@jedevc
Copy link
Member

jedevc commented Nov 4, 2022

Thanks @flyingsaucerer - we've managed to reproduce internally, let's track this in #1440.

@jedevc jedevc closed this as completed Nov 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/external
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jedevc @AkihiroSuda