Create a plugin that tests job changes

[alvaroaleman]

What would you like to be added:

A way to test job changes. Openshift does this via a CLI that checks out the config repo, finds changed jobs and creates a so-called "rehearse" job": https://github.com/openshift/ci-tools/tree/master/cmd/pj-rehearse. What openshift does is highly specific because many jobs there rely on configmaps that then also need to be deployed. This shouldn't be the case for most jobs of other prow instances.

So basically a plugin that on config-repo (test-infra in kube casse) PRs:

• Finds added/changed jobs
• Creates an instance of them and reports that back to the job change pr
• Potentially this needs opt out logic and/or a dedicated command to trigger this
• Some railguards i.E. create at most N rehearse jobs

Why is this needed:

Because currently job config changes are untestable if access to the prow.k8s.io build cluster is not given.

/area prow
/cc @BenTheElder @fejta

[BenTheElder]

We usually just ask that users create a manually triggered "canary" copy of the job where risky changes are first trialed for things that depend on CI resources. For things that don't there's the local running options (mkpod, pj-on-kind, phaino ...)

…

On Thu, Jul 16, 2020, 06:54 Alvaro Aleman ***@***.***> wrote: *What would you like to be added*: A way to test job changes. Openshift does this via a CLI that checks out the config repo, finds changed jobs and creates a so-called "rehearse" job": https://github.com/openshift/ci-tools/tree/master/cmd/pj-rehearse. What openshift does is highly specific because many jobs there rely on configmaps that then also need to be deployed. This shouldn't be the case for most jobs of other prow instances. So basically a plugin that on config-repo (test-infra in kube casse) PRs: - Finds added/changed jobs - Creates an instance of them and reports that back to the job change pr - Potentially this needs opt out logic and/or a dedicated command to trigger this *Why is this needed*: Because currently job config changes are untestable if access to the prow.k8s.io build cluster is not given. /area prow /cc @BenTheElder <https://github.com/BenTheElder> @fejta <https://github.com/fejta> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18338>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADKYKS5HUNUURZUHMWG3R34BBFANCNFSM4O4JMQMQ> .

[alvaroaleman]

Yeah in my experience jobs that do not actually run in the real cluster are just different. mkpod requires to set up a different decoration config or know other intimate details. Many jobs generally assume the presence of certain secrets.

I just ran into the issue with my "increase CPU requests oh damn the cluster doesn't have nodes this size" change and since I am used to getting feedback on job changes, I wanted to leave the idea here.

I am not going to build this, it is just intended as an idea for the ppl that have to regularly deal with the fun of this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.