Create kubeadm presubmit e2e job

[pipejakob]

kubeadm should have end-to-end tests that run against open pull requests and report success or failure before changes are merged. There's been pushback against creating new PR-blocking jobs given the current state of the world, but we should be able to at least auto-run against pull requests that change anything in cmd/kubeadm/*, and report the results so that honest contributors can catch and fix their own regressions early.

This is a subtask of #190, but I wanted to break it out for better granularity, especially in light of the 1.6.0 postmortem.

[pipejakob]

Status: this job was created a while ago for experimentation pull-kubernetes-e2e-kubeadm-gce.sh, and there's a pending PR to fix a critical flaw in its prow pipeline: kubernetes/test-infra#2509.

[dmmcquay]

@pipejakob let me know how I can help.

[coeki]

@pipejakob Not sure if this is the right place to say something, but most ppl follow the install doc, using apt-get or rpm's. This was a huge problem with the release (and still is, 1.6.2 is released, but no packages in the repo's, although one older release is, it should be all), the old, known working packages were not there. So ppl could not go back to what they knew was working. So testing e2e should be based on, patched releases (eg 1.5.6), and newer releases (1.6.x) to avoid running into problems.

I'm refactoring my dev environment (vagrant/ansible, multiple vm's) to catch errors on different distro's (centos, ubuntu, debian and fedora), so bascially a manual kind of testing. And just to make sure the main elements works, so kubeadm, kubernetes, network cni plugins, multiple nodes can join, and for selinux.

I'm saying this because, on a binary level everything might work, automated tests work. But on a packaging side, see kubernetes/release#306, things may go haywire.

Not sure how we can test that e2e.

[pipejakob]

@coeki I think I follow. So, right now, our end-to-end tests build a fresh set of .debs as part of the CI process, and do dpkg -i *.deb to install them. It sounds like part of what you're suggesting is to at least have end-to-end tests that instead do apt-get install kubeadm to test the latest stable/unstable offerings, like users actually do while installing (and likely the equivalent yum commands for .rpms).

The other part is that you'd like to see fixed-version tests (which I think is achievable by doing something like apt-get install kubeadm=1.5.6) to make sure older versions are still accessible in the repos and working as intended. Does that sound right?

I think these are great requests, but definitely outside the scope of this particular issue. Can you open new issues to track them? Also, the 1.6.2 .debs/.rpms are now mostly released, although I think we're still waiting on an internal request to be able to release .rpms for certain architectures.

[jamiehannaford]

@coeki @pipejakob I created those issues (#263, #264)

[coeki]

@pipejakob That was what I meant, yes.

@jamiehannaford Thanks for creating those issues.

[timothysc]

@kubernetes/sig-testing-feature-requests I'd really like to get this into the merge blocking test suite. I mean if kops can, this should. /cc @spiffxp @fejta

[k8s-ci-robot]

@timothysc: These labels do not exist in this repository: sig/testing.

In response to this:

@kubernetes/sig-testing-feature-requests I'd really like to get this into the merge blocking test suite. I mean if kops can, this should. /cc @spiffxp @fejta

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[fejta]

First step is to ensure it runs on every PR with reporting enabled. I think reporting is disabled?

After that then ensure it flakes less than others: http://storage.googleapis.com/k8s-metrics/flakes-latest.json

[0xmichalis]

Consider enabling the kubeadm self-hosted mode in the job.

[timothysc]

'self-hosted' imo is not ready for prime time. full-stop.

[luxas]

@fejta A consistency of 98.9% is good IMO. I saw some blocking tests are around 91% so...

@pipejakob @mikedanese @timothysc Will you do the switch in the test-infra repo or should I?

[fejta]

Yeah, flakiness right now is good (~99% consistent) however it only runs on ~200 commits vs 1200 for others: http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json

[pipejakob]

@luxas I'm going to try to tackle #278 first, but then I should be able to get to this. If anyone else has the time, feel free (and assign me to the PR)!

[luxas]

@pipejakob Perfect, thank you!

[0xmichalis]

'self-hosted' imo is not ready for prime time. full-stop.

Are you guys expecting to start testing it after you deem it 'ready for prime-time'? I would expect it to be the other way around. A non-blocking job that proves it works is better than nothing.

[pipejakob]

I forgot to link it here, but I have an open PR for whenever our master CI goes back to green: kubernetes/test-infra#2976

Hopefully, this should happen with the release of 1.8.0-alpha.1 on Monday.

[luxas]

ping @pipejakob any movement on this issue lately?

[luxas]

@pipejakob I'm moving this to the v1.9 milestone, this shouldn't block v1.8 although it had been great to have...

[luxas]

After kubernetes/test-infra#4480, kubernetes/test-infra#4705, kubernetes/test-infra#4784 and some other various test-infra changes, our presubmit job finally works and is enabled 🎉

Of course there are some flakes, but those are due to flaky e2e tests: http://prow.k8s.io/?job=pull-kubernetes-e2e-kubeadm-gce

I consider this done, and will open smaller, more targeted issues if something yet needs to be improved.