Image builds should be tested in presubmit

[justaugustus]

What would you like to be added:

Image builds should be tested in presubmit.

Rough expectations:

• new GCP project (k8s-releng-test): PR -> releng: create new gcp project k8s-infra-staging-releng-test to test images build k8s.io#1592
• new presubmits (in the k/release job config) that mirror the postsubmits in:
  • https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/releng/k8s-staging-build-image.yaml
  • https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/releng/k8s-staging-releng.yaml

Why is this needed:

As we build critical images for core Kubernetes releases, failure to build those images should be blocking.

@kubernetes/release-engineering -- this is a good one to pick up.
/help
ref: #1849

[sladyn98]

Any way I can contribute to this ?

[cpanato]

@sladyn98 I started over the weekend (forgot to remove the help wanted), but I think you can work on that if you want.

Need to check how to create the new GCP project

[sladyn98]

@cpanato So If I understand correctly this would involve creating the new gcp project and then mirroring the post submit steps in the release.yml right ?

[cpanato]

@sladyn98 the new jobs should go here https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/release/release-config.yaml

[sladyn98]

Yeah :) however what about creating the new gcp project, is that something I need to take up or the team would do that for me

[cpanato]

@sladyn98 this part i dont know either, we need to check in #test-infra or #wg-k8s-infra to see if there is a process or how to request that

[cpanato]

/assign @sladyn98
/remove-help

[ameukam]

@sladyn98 @cpanato AFAIK the GCP projects related to the release process are managed here : https://github.com/kubernetes/k8s.io/blob/master/infra/gcp/ensure-release-projects.sh. However their access is limited to specific groups (https://github.com/kubernetes/k8s.io/blob/master/infra/gcp/ensure-release-projects.sh#L46-L48). The exiting project k8s-release-test-prod can also be used.

[cpanato]

@ameukam maybe @justaugustus can comment on that

[justaugustus]

Another example: #1853 / https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/post-release-push-image-kube-cross/1350255401592950784 / https://console.cloud.google.com/cloud-build/builds/2f761f3d-d532-43c0-903b-e385c6069b4c?project=960211007710

[sladyn98]

@ameukam So if i understand correctly I can add the steps without creating a new project.
CC cpanato

[cpanato]

Checking the project creation over slack: https://kubernetes.slack.com/archives/CCK68P2Q2/p1611581520009100

[saschagrunert]

I think we could achieve the test cases without adding an additional project. We could just add a set of tests which run make all-build within the image/* directories. This way we do not have to push anything out and could succeed if the builds work. Any thoughts on that?

This way we can not test the logic from cloudbuild.yaml, but we could at least test if the images are building correctly.

[cpanato]

If that is enough for now we can move on this direction

[saschagrunert]

Thought about the test approach and I think adding a new project might be the most clean way to test everything including the cloudbuild process.

[justaugustus]

Some of the failures we've seen have been distinct to running in GCB, so while I think a "local" test build could be good, we should also test in GCB.

[justaugustus]

@sladyn98 -- Unfortunately, this is going to require Release Manager access to configure and debug, so I need to unassign you.

/unassign @sladyn98

@kubernetes/release-engineering -- part of the intent for opening this issue was so that someone on the team (that wasn't me) could:

• get up-to-speed with the project creation process
• get more experience with the base image building
• start working on fixing any gaps we have in documentation

Who would like to take this on?

[sladyn98]

@justaugustus No worries, I will pick up something else that I can work on. Thanks

[cpanato]

@justaugustus I started the discussion I think I can take it over

[justaugustus]

@justaugustus No worries, I will pick up something else that I can work on. Thanks

Thanks for being understanding! 💕

@justaugustus I started the discussion I think I can take it over

SGTM!
/assign @cpanato

[cpanato]

I think we have all jobs in place 😄
will close this issue and if we need to do any followup we can open a fresh one!

/close

[k8s-ci-robot]

@cpanato: Closing this issue.

In response to this:

I think we have all jobs in place 😄
will close this issue and if we need to do any followup we can open a fresh one!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.