failed to push image after GCB build

[cpanato]

We have a presubmit prow job to test if the images are built correctly

we have an issue that is the job does not complete successfully because it cannot push image we got this error

denied: Token exchange failed for project 'k8s-staging-build-image'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control

failed job example: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/release/1890/pull-release-image-kube-cross/1359158451615305728

[cpanato]

/assign @cpanato @spiffxp

[spiffxp]

Ohhhhhh. Ok so here's what's happening.

gcb-builder-releng-test is just used to trigger cloud builds, it is not what those builds run as

AFAIK those always run as a special service account created by GCB in the project that is running those builds

(Dropping k8s-staging since typing on a device) So, I think, the prowjob is running in k8s-infra-prow-build. But its triggering a build in releng-test. That build is running as a different SA in releng-test (https://cloud.google.com/build/docs/cloud-build-service-account#default_permissions_of_service_account). Where it's failing, I think, is trying to push images to some completely different project build-image.

Help me understand what you're trying to do again?

I suspect you do not want presubmits (possibly untrusted code) pushing to build-image, but instead verifying that images can build and push somewhere

[spiffxp]

Piecemeal configuring certain projects being able to push to certain repos sounds like a recipe for complication. What's the general pattern we're trying to accomplish?

I'm interested in being able to push to buckets/images within a project, whether from prow or from gcb. Moving away from one special SA being able to push to all projects at once.

[spiffxp]

tl;dr have you tried pushing to gcr.io/k8s-staging-releng-test instead?

[justaugustus]

Another thought...

Ultimately, I'm not sure that we care the entire workflow, just the fact that the image was built successfully.
What about enabling the presubmit/RelEng tools to have a --local mode, which would use cloud-build-local instead: https://cloud.google.com/build/docs/build-debug-locally

[justaugustus]

tl;dr have you tried pushing to gcr.io/k8s-staging-releng-test instead?

But agreed w/ @spiffxp's assessment, the most recent (non-go1.16.1) job is running in k8s-staging-releng-test, but attempting to push to k8s-staging-build-image, which is going to fail unless the GCB SA has access to write into GCR for the other project.

If we decide we definitely want to test the push step, then the job should be reconfigured to build and push to the same project.

[spiffxp]

You might be able to change your cloudbuild to not push, depending on presence of an env var? And/or if image-builder doesn't support "no-push" that seems like a worthwhile addition

FWIW it may not be a problem for your use case, but bumping into this fun with buildx is why I like the idea of exercising a push

kubernetes/test-infra#20884 (comment)

[cpanato]

I will check how to set a different registry and the no push option as well

[cpanato]

thanks for all the inputs and feedback

[cpanato]

after some changes now it works! 🎉

one GCB build for example: https://console.cloud.google.com/cloud-build/builds;region=global/9e1057f7-eb76-4839-a95f-b34c61815278?project=k8s-staging-releng-test
prow job for that: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/release/1890/pull-release-image-kube-cross/1370028453318889472

we can close this issue

thanks Aaron and Stephen!
/woof
/close

[k8s-ci-robot]

@cpanato:

In response to this:

after some changes now it works! 🎉

one GCB build for example: https://console.cloud.google.com/cloud-build/builds;region=global/9e1057f7-eb76-4839-a95f-b34c61815278?project=k8s-staging-releng-test
prow job for that: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/release/1890/pull-release-image-kube-cross/1370028453318889472

we can close this issue

thanks Aaron and Stephen!
/woof
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@cpanato: Closing this issue.

In response to this:

after some changes now it works! 🎉

one GCB build for example: https://console.cloud.google.com/cloud-build/builds;region=global/9e1057f7-eb76-4839-a95f-b34c61815278?project=k8s-staging-releng-test
prow job for that: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/release/1890/pull-release-image-kube-cross/1370028453318889472

we can close this issue

thanks Aaron and Stephen!
/woof
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.