Migrate CRI-O jobs away from kubernetes_e2e.py

[saschagrunert]

The kubernetes_e2e.py script is deprecated and we should use kubetest2 instead.

All affected tests are listed in https://testgrid.k8s.io/sig-node-cri-o

cc @kubernetes/sig-node-cri-o-test-maintainers

Ref: https://github.com/kubernetes/test-infra/tree/master/scenarios, #20760

[haircommander]

/sig node

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[saschagrunert]

/remove-lifecycle stale

[kannon92]

/triage accepted
/priority important-longterm

[elieser1101]

Does this still need help? can i start looking at it?

[saschagrunert]

@elieser1101 I'd appreciate your eyes on that. 🙏

[elieser1101]

/assign

[bart0sh]

@kannon92 @elieser1101 @haircommander
Looking at failing splitfs and imagefs pr jobs, I noticed random test failures inside a container with this error message:

sh: error while loading shared libraries: /lib/libc.so.6: cannot apply additional memory protection after relocation: Permission denied"

This seems to be a core issue causing jobs to fail.

Unfortunately I can't reproduce it in my environment. Here is how I run kubetest2 for splitfs tests:

$ GCE_SSH_PUBLIC_KEY_FILE=/home/ed/.ssh/google_compute_engine.pub KUBE_SSH_USER=core IGNITION_INJECT_GCE_SSH_PUBLIC_KEY_FILE=1 JENKINS_GCE_SSH_PRIVATE_KEY_FILE=/home/ed/.ssh/google_compute_engine kubetest2-gce --test=node --down=false -- --parallelism=8 --gcp-zone=us-west1-b --gcp-project=service-mesh-296815 --repo-root=. --image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgroupv2-splitfs.yaml --delete-instances=false --test-args='--container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"' --skip-regex='\[Flaky\]|\[Slow\]|\[Serial\]' --focus-regex='\[NodeConformance\]|\[NodeFeature:.+\]|\[NodeFeature\]' 2>&1 | tee /tmp/log

I suspect that this could be caused by the host/vm kernel and container resource restrictions, but I don't know how to specify upper level instance image (gcr.io/k8s-staging-test-infra/kubekins-e2e:v20241128-8df65c072f-master) and container resources (cpu 4 and memory 6Gi) when running kubetest2 locally.

Any ideas how to proceed further?

[haircommander]

do you have access to the nodes you've provisioned @bart0sh ? can I poke around? basically, we want to be able to run ausearch -m AVC -ts recent after the failure to see what was being blocked, then we can update the selinux policy we create in ignition to include the new option

[bart0sh]

@haircommander yes, I have access to the nodes, but I can't reproduce the error there :(

[bart0sh]

btw, decreasing parallelism seem to help a bit: https://testgrid.k8s.io/sig-node-presubmits#pr-crio-cgrpv2-splitfs-e2e-kubetest2&width=90

[elieser1101]

Trigered the job couple of times and seem to be improved(still failed at some point), also takes longer to complete. We could test the imagefs one with the same approach @bart0sh but i guess the right fix includes the selinux cahnge?

[bart0sh]

I'm not sure about it. selinux configuration is the same for kubetest2 and old jobs, but only kubetest2 jobs fail.

[bart0sh]

@elieser1101

We could test the imagefs one with the same approach

decreasing parallelism improved imagefs test. Previously I didn't see any successful job runs. With the change I can see at least one so far.

[bart0sh]

I'm still wondering why I can't repro error while loading shared libraries: /lib/libc.so.6: cannot apply additional memory protection after relocation: Permission denied in my environment. I'm using the same kubetest2 command line parameters and the same image configs, .ign file, instance type, gcp zone etc. Even using --processes=100 command line option doesn't help to trigger the error.

[bart0sh]

Unfortunately using more powerful instance didn't change much for imagefs job. I can still see the same error in the logs.

[bart0sh]

@elieser1101 I can see a lot of green kubetest2 jobs in the test grid. Is there anything that prevents replacing kubernetes_e2e.py jobs with them? I did it for splitfs and imagefs jobs as I was involved in fixing them. I can do it for the rest of jobs if needed.

[elieser1101]

@bart0sh thank you very much for the splitfs/imagefs that was a great finding

---

What would come next is to validate that the kubetest2 are actually working. Meaning, I noticed that some of the jobs are completing but are skipping all the specs. We would like to ensure we are running the jobs properly before replacing the kubernetes_e2e.py jobs.

At the moment im loking at the DRA ones wich were missing some kubetest2 features and this