[CI] Use Kubernetes GC to clean kubevirt VMs (packet-* jobs) #11530
Conversation
k8s-ci-robot
added
release-note-none
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: VannTen The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
approved
|
/ok-to-test |
k8s-ci-robot
added
the
ok-to-test
|
/cc @ant31 |
|
/retest |
|
/retest
|
VannTen
force-pushed
the
ci/cleanup_with_k8s_gc
branch
2 times, most recently
from
fd43216
to
d70f8e2
Compare
|
/label ci-full (To test it works correctly for everything) |
|
@VannTen: The label(s) In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
VannTen
force-pushed
the
ci/cleanup_with_k8s_gc
branch
from
d70f8e2
to
bbfd93a
Compare
|
@ant31
What's the process to add the ci-{extended,full} label ? Can't find them with a quick grepping
|
VannTen
force-pushed
the
ci/cleanup_with_k8s_gc
branch
2 times, most recently
from
b67b0b2
to
25bb5d0
Compare
|
I think the PR to add them via /label is still in review For now you can add them manually |
|
I'll do that once the initial set of tests pass then 👍 |
VannTen
force-pushed
the
ci/cleanup_with_k8s_gc
branch
from
25bb5d0
to
32a0dfc
Compare
VannTen
force-pushed
the
ci/cleanup_with_k8s_gc
branch
from
32a0dfc
to
2c501df
Compare
This leverage the Kubernetes GC to delete kubevirt VMs, by using ownerReferences, with the CI pod running the playbook as the owner. This concretely means that the control plane in our CI cluster will delete the kubevirt VMs associated with a particular ci job as soon as that pod job is deleted, which usually happens when the job terminates, (barring errors, which will be addressed in the cluster directly) Upgrade to kubevirt.io/v1 for the VirtualMachine manifests, since the alpha version is deprecated.
Kubevirt VMs deletion will be handled by the Kubernetes GC (see previous commit), remove all the codes handling that.
VannTen
force-pushed
the
ci/cleanup_with_k8s_gc
branch
from
2c501df
to
81ac78a
Compare
Also, use the Ready condition of VirtualMachine instead of custom checks
VannTen
force-pushed
the
ci/cleanup_with_k8s_gc
branch
2 times, most recently
from
d1ca52f
to
1b5fa4b
Compare
VannTen
force-pushed
the
ci/cleanup_with_k8s_gc
branch
2 times, most recently
from
b1bdc8f
to
077dcad
Compare
Not constraining the inventory to .ini allows us to use dynamic inventory, which is needed for simplifying kubevirt jobs inventory. Also reduces the scope of the ANSIBLE_INVENTORY variable.
This allows a single source of truth for the virtual machines in a kubevirt ci-run. `etcd_member_name` should be correctly handled in kubespray-defaults for testing the recover cases.
VMI in Kubevirt are the abstraction below VirtualMachine. - We don't really need the extra abstraction of VirtualMachine objects - Fix the provisioning playbook not waiting correctly on the VMs until they have an IP address to use for the dynamic inventory
VannTen
force-pushed
the
ci/cleanup_with_k8s_gc
branch
from
077dcad
to
19a4aab
Compare
|
I still need to figure out how to avoid breaking the upgrade testing (since the inventory file is not there) but otherwise this should be ready soonish :) |
k8s-ci-robot
added
the
needs-rebase
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What type of PR is this?
/kind feature
What this PR does / why we need it:
We regularly have CI flakes where the job failed to delete k8s namespace in the CI cluster.
It's not much, but it's a little hiccup in the PR process which I'd like to eliminate.
I'm not sure what the exact reason is, probably some race between the jobs and the time between fetching the list of namespace and the deletion.
Regardless, a simpler way to delete the VMs is to let them be dependants (in the kubernetes sense) of the job pod. This way, once the job pod is deleted, kubernetes garbage collection in the CI cluster will take care of removing the associated VMs
Special notes for your reviewer:
PR on the ci infra kubespray/kspray-infra#1 (private repo, maintainers have access)
Does this PR introduce a user-facing change?:
/label tide/merge-method-merge