broken and out of date buildroots

[darkmuggle]

The fcos-buildroot and the cosa-buildroot are currently non-functional:

• the cosa-buildroot was out of date (last build of March 25th)
• the fcos-buildroot was last built on Jun 11

After updating cosa-buildroot via openshift/release@3ee5f3f, Prow is unable to execute:

Storing signatures
STEP 2: ENV "BUILD_LOGLEVEL"="0" "CLONEREFS_OPTIONS"="{\"src_root\":\"/go\",\"log\":\"/dev/null\",\"git_user_name\":\"ci-robot\",\"git_user_email\":\"[email protected]\",\"refs\":[{\"org\":\"coreos\",\"repo\":\"coreos-assembler\",\"repo_link\":\"https://github.com/coreos/coreos-assembler\",\"base_ref\":\"main\",\"base_sha\":\"513a1592d54ffddccbe8a6963512c438c1d1b083\",\"base_link\":\"https://github.com/coreos/coreos-assembler/commit/513a1592d54ffddccbe8a6963512c438c1d1b083\",\"pulls\":[{\"number\":2263,\"author\":\"darkmuggle\",\"sha\":\"62cdeac9a551da0a0a9a627b22cd86b99c6bd100\",\"link\":\"https://github.com/coreos/coreos-assembler/pull/2263\",\"commit_link\":\"https://github.com/coreos/coreos-assembler/pull/2263/commits/62cdeac9a551da0a0a9a627b22cd86b99c6bd100\",\"author_link\":\"https://github.com/darkmuggle\"}]}],\"fail\":true}"
STEP 3: ADD ./clonerefs /clonerefs
STEP 4: RUN umask 0002 && /clonerefs && find /go/src -type d -not -perm -0775 | xargs --max-procs 10 --max-args 100 --no-run-if-empty chmod g+xw
{"component":"clonerefs","file":"prow/pod-utils/clone/clone.go:49","func":"k8s.io/test-infra/prow/pod-utils/clone.Run","level":"info","msg":"Cloning refs","refs":{"org":"coreos","repo":"coreos-assembler","repo_link":"https://github.com/coreos/coreos-assembler","base_ref":"main","base_sha":"513a1592d54ffddccbe8a6963512c438c1d1b083","base_link":"https://github.com/coreos/coreos-assembler/commit/513a1592d54ffddccbe8a6963512c438c1d1b083","pulls":[{"number":2263,"author":"darkmuggle","sha":"62cdeac9a551da0a0a9a627b22cd86b99c6bd100","link":"https://github.com/coreos/coreos-assembler/pull/2263","commit_link":"https://github.com/coreos/coreos-assembler/pull/2263/commits/62cdeac9a551da0a0a9a627b22cd86b99c6bd100","author_link":"https://github.com/darkmuggle"}]},"severity":"info","time":"2021-07-06T14:49:05Z"}
{"command":"mkdir -p /go/src/github.com/coreos/coreos-assembler","component":"clonerefs","error":"exit status 1","file":"prow/pod-utils/clone/clone.go:61","func":"k8s.io/test-infra/prow/pod-utils/clone.Run.func2","level":"info","msg":"Ran command","output":"mkdir: cannot create directory '/go': Permission denied\n","severity":"info","time":"2021-07-06T14:49:05Z"}
{"component":"clonerefs","error":"1 clone records failed","file":"prow/cmd/clonerefs/main.go:40","func":"main.main","level":"fatal","msg":"Failed to clone refs","severity":"fatal","time":"2021-07-06T14:49:05Z"}
error: build error: error building at STEP "RUN umask 0002 && /clonerefs && find /go/src -type d -not -perm -0775 | xargs --max-procs 10 --max-args 100 --no-run-if-empty chmod g+xw": error while running runtime: exit status 1
�[36mINFO�[0m[2021-07-06T14:49:11Z] Ran for 2m46s                                
�[31mERRO�[0m[2021-07-06T14:49:11Z] Some steps failed:                           
�[31mERRO�[0m[2021-07-06T14:49:11Z] 
  * could not run steps: step src failed: the build src failed after 2m44s with reason DockerBuildFailed: Dockerfile build strategy has failed.

[darkmuggle]

This might be related to #787

[jlebon]

Ughh, this is Quay.io caching hitting us. The last build for fcos-buildroot according to https://quay.io/repository/coreos-assembler/fcos-buildroot?tab=tags was 17 hours ago. But yeah, indeed looks like the last full build was June 11th. So short-term fix is to just bump the hacky comment that invalidates cache. Mid/long-term... we move it to openshift/release and just have it push to Quay.io instead?

[jlebon]

OK, so unwinding a few things here. I think instead of openshift/release@3ee5f3f, we should basically just nuke the cosa-buildroot image completely. The idea now is that building OS components should use fcos-buildroot. (That was the intention of coreos/fedora-coreos-config#740 and coreos/coreos-assembler#2098).

For the cosa Prow test, instead of having it run make && make check in a buildroot container, let's just have it build the cosa container proper using the regular Dockerfile?

[cgwalters]

The core issue here was fixed. We clearly don't want to need to continue to do the periodic "invalidate quay.io cache" PRs, but I think what we have now is at least not totally broken.