1.14 Testgrid test jobs should match master, not 1.13

[jberkus]

@mariantalla @spiffxp @amwat etc.

The new 1.14 Testgrid Boards were created with a set of test jobs matching the 1.13 jobs. This is wrong; the list of jobs for 1.14 should match master, as discussed at prior sig-release meetings and one Release Team meeting. This means, among other things, Blocking and Informing, not Blocking and All.

I'd just swap out the test list, but since I don't know why the boards got created with the wrong list of jobs to begin with, I wanted to make sure first that it was an accident.

/area testgrid
/milestone v1.14
/kind cleanup

[k8s-ci-robot]

@jberkus: You must be a member of the kubernetes/kubernetes-milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

@mariantalla @spiffxp @amwat etc.

The new 1.14 Testgrid Boards were created with a set of test jobs matching the 1.13 jobs. This is wrong; the list of jobs for 1.14 should match master, as discussed at prior sig-release meetings and one Release Team meeting. This means, among other things, Blocking and Informing, not Blocking and All.

I'd just swap out the test list, but since I don't know why the boards got created with the wrong list of jobs to begin with, I wanted to make sure first that it was an accident.

/area testgrid
/milestone v1.14
/kind cleanup

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

/milestone v1.14
/area release-team
/sig release

[krzyzacy]

oh dohhhh that's still not unified...

@jberkus the current workflow is we have a script to promote the version for all jobs, like:
1.10 -> 1.11
1.11 -> 1.12
1.12 -> 1.13
1.13 -> 1.14

Looks like that's not valid anymore and we should update the workflow in test-infra doc

[krzyzacy]

well - @jberkus I think the issue is still we don't have all equivalent jobs for master and release branches, aka packages-pushed-master, and we are still pending to remove kubeadm jobs from release-blocking dashboards.

[spiffxp]

I honestly forget if my blocking/informing proposal was meant to propogate to all boards or just master. I will look. I feel like linting this could help us catch it, if/when we are at a point where the job list is consistent.

[jberkus]

@krzyzacy so sounds like I need to do specific surgery on the 1.14 boards then?

[krzyzacy]

@jberkus there's already kubernetes/sig-release#518 - and afaik the packages-pushed-master doesn't have a branch version for a reason? cc @neolit123

[jberkus]

@krzyzacy Take a look at the master boards, there's a LOT more changed than that.

[krzyzacy]

hummm sorry I was staring at the blocking dashboard - feel free to shuffle all the -all dashboard to become -informing I guess? (all - blocking = informing I assume)

[neolit123]

@jberkus

The new 1.14 Testgrid Boards were created with a set of test jobs matching the 1.13 jobs. This is wrong; the list of jobs for 1.14 should match master

given we already have a 1.14 k/k branch shouldn't we just match all 1.14 jobs to 1.14? here is a 1.14 job that runs against release-1.14 fine, or rather was resurrected yesterday.

@jberkus there's already kubernetes/sig-release#518 - and afaik the packages-pushed-master doesn't have a branch version for a reason? cc @neolit123

the packages-* jobs from sig-cluster lifecycle are jobs that run over the whole support skew...
these tests are so small that running them in separate jobs/pods is a complete waste.

just have them in a generic sig-release dashboard somewhere.
also the way they work is they check already existing artifacts, so it's not like they can "block" a release that is already out (packages were pushed).

[neolit123]

actually, packages-install-deb can catch a problem from CI packages too. 🤔

ok, i'm thinking about moving the packages* jobs:
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all#periodic-packages-pushed
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all#periodic-packages-install-deb

to here:
https://k8s-testgrid.appspot.com/sig-release-misc

i also explained this here:
kubernetes/sig-release#518 (comment)

is misc a better home? any objections?

[neolit123]

#11560

[jberkus]

@spiffxp we agreed in sig-release meeting that it would apply to the 1.14 boards, but that we'd decide later whether to push it down to the older boards.

This was also a follow-up to the 1.13 cycle, to clean up the boards, not just master but versioned.

[krzyzacy]

@jberkus so currently all branch dashboards are sorta unified - if we want to do sweeping changes I'd vote we still keep them consistent across old supported releases or else it will be painful when we bump for next release.

[jberkus]

Sen, I'm up for that, but I can't decide it unilaterally.

[jberkus]

@mariantalla @spiffxp @tpepper how can we move ahead on this? I really don't like going into code freeze with the same mess of noncompliant tests we had in 1.13.

[mariantalla]

Hey, picking this up again (with the intention to make it for 1.14, but failing that let's try to have a way forward for 1.15)

I wonder if we can decouple the two; first, bring 1.14 to match master and then create a workflow for future releases.

I'm a bit worried about:

currently all branch dashboards are sorta unified

@krzyzacy , if I got that right, does that mean that changing the 1.14 dashboards might affect 1.13 dashboards? If yes, in what way?

#11370 seems to have created the 1.14 jobs - is the script that made the testgrid config changes somewhere in test-infra? 🤔bump_e2e_image.sh seems to create the prow jobs but (as far as I can see) not the testgrid ones 👀

[krzyzacy]

changing the 1.14 dashboards might affect 1.13 dashboards? If yes, in what way?

Not right now. But when we create 1.15 branch jobs, we are not doing that by deleting 1.11 and creating 1.15, we remap the dashboard from 1.11 -> 1.12, 1.12 -> 1.13, 1.13 -> 1.14, 1.14 -> 1.15

Feel free to do that for 1.14 for now, we can bikeshed older dashboards before 1.15 happens.

bump_e2e_image.sh seems to create the prow jobs but (as far as I can see) not the testgrid ones eyes

that script bumps the image used in the job, which has nothing to do with testgrid :-)

[mariantalla]

I've just started (started being the keyword) work in #11849 for 1.14 in specific. I think it will be a good exercise to do once manually and then consider automating for v1.15 onwards.

Other things on my list:

• Open issue for future non-master dashboards to match master dashboards (depending on how [WIP] Refactor 1.14 testgrid dashboards to match master #11849 goes too). I feel this is a bigger discussion to have with sig-testing and sig-release though.

[spiffxp]

Not just testgrid configs, also job configs, we just bumped into an error with 1.13->1.14 missing a change in master (ref: #11850)

[imkin]

Audited the PR #10795 and found that the other job configs were fine except for the one reported in #11850

Verfied the same in the master. Automation needs to change based on this but will do that async to this issue.

[spiffxp]

/milestone v1.15
ref: #11977

[spiffxp]

/milestone v1.16

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.