fix tests in sig-node related to CRIv1 update #28071

akhilerm · 2022-11-21T08:43:32Z

Fixes tests under

sig-nod-containerd
sig-node-release-blocking
sig-node-presubmits

The following tests are fixed:

ci-kubernetes-node-kubelet-containerd-eviction
ci-kubernetes-node-kubelet-containerd-resource-managers
ci-kubernetes-node-kubelet-serial-containerd
pull-kubernetes-node-kubelet-serial-containerd
pull-kubernetes-node-kubelet-serial-containerd-kubetest2
pull-kubernetes-node-kubelet-serial-pod-disruption-conditions

Signed-off-by: Akhil Mohan <[email protected]>

Updated cos images to cos-stable Updated ubuntu to ubuntu-2204-lts image_family This update will fix the following jobs: - ci-kubernetes-node-kubelet-containerd-eviction - ci-kubernetes-node-kubelet-serial-containerd - pull-kubernetes-node-kubelet-serial-containerd - pull-kubernetes-node-kubelet-serial-containerd-kubetest2 - pull-kubernetes-node-kubelet-serial-pod-disruption-conditions Signed-off-by: Akhil Mohan <[email protected]>

k8s-ci-robot · 2022-11-21T08:43:40Z

Hi @akhilerm. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

akhilerm · 2022-11-21T08:47:23Z

jobs/e2e_node/containerd/image-config-serial.yaml

    machine: n1-standard-2 # These tests need a lot of memory
    metadata: "user-data</workspace/test-infra/jobs/e2e_node/containerd/init.yaml,cni-template</workspace/test-infra/jobs/e2e_node/containerd/cni.template,containerd-config</workspace/test-infra/jobs/e2e_node/containerd/config.toml"
  cos-stable2:
-    image_family: cos-93-lts # deprecated after October 2023 (https://cloud.google.com/container-optimized-os/docs/release-notes)
+    image_family: cos-stable


@SergeyKanzhelev Why were we maintaining 2 separate cos versions for the tests in image-config-serial ?

akhilerm · 2022-11-21T08:48:59Z

/cc @SergeyKanzhelev @bobbypage @adisky

Created this as a separate PR from #27943 so as not to make the review process of that more complex.

adisky · 2022-11-21T10:00:24Z

/ok-to-test

bobbypage · 2022-11-27T00:17:30Z

jobs/e2e_node/containerd/image-config-serial.yaml

-    image_family: pipeline-1-24
-    project: ubuntu-os-gke-cloud
+    image_family: ubuntu-2204-lts
+    project: ubuntu-os-cloud
    machine: n1-standard-2 # These tests need a lot of memory
    metadata: "user-data</workspace/test-infra/jobs/e2e_node/containerd/init.yaml,cni-template</workspace/test-infra/jobs/e2e_node/containerd/cni.template,containerd-config</workspace/test-infra/jobs/e2e_node/containerd/config.toml"


These need to point to containerd config file that will set CONTAINERD_SYSTEMD_CGROUP so that systemd cgroup driver will be configured on containerd side as well. I don't see that being the case currently?

It looks like these jobs just use this helper script (https://github.com/kubernetes/test-infra/blob/master/jobs/e2e_node/containerd/init.yaml) which installs this containerd config file - https://github.com/kubernetes/test-infra/blob/master/jobs/e2e_node/containerd/config.toml

Do we need to maybe make a copy of that config.toml for systemd cgroup driver and point the test to use that config file?

The other option is to configure this test to also use containerd from main like the some of the other jobs, so we can use CONTAINERD_SYSTEMD_CGROUP setting.

The other option is to configure this test to also use containerd from main like the some of the other jobs, so we can use CONTAINERD_SYSTEMD_CGROUP setting.

What was the reason for these tests using separate config.toml file? If we change to how its done in other tests, like you suggested, is there something else that need to be taken care of?

I have created a new config-systemd.toml that will be used instead of config.toml to use systemd as cgroup driver.

use the config file with systemd enabled in tests which use cgroupv2 Signed-off-by: Akhil Mohan <[email protected]>

akhilerm · 2022-11-30T12:34:22Z

@bobbypage I have made the changes. Can you take a look at it.

ameukam · 2022-11-30T12:59:42Z

jobs/e2e_node/containerd/config-systemd.toml

+# Enable registry.k8s.io as the primary mirror for k8s.gcr.io
+# See: https://github.com/kubernetes/k8s.io/issues/3411
+[plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
+  endpoint = ["https://registry.k8s.io", "https://k8s.gcr.io",]
+


Now registry.k8s.io is GA (and introduced in the master branch of k/k) there is no specific value to do this.
we should remove this as a follow-up.

Will create a separate cleanup PR for that, so that the original config.toml can also be fixed.

@ameukam Found this comment in the referring issue for the migration, So should we wait for some more time before switching?

ameukam · 2022-11-30T13:00:07Z

jobs/e2e_node/containerd/image-config-serial-resource-managers.yaml

@@ -3,14 +3,14 @@
 # `gcloud compute --project <to-project> images create <image-name> --source-disk=<image-name>`
 images:
  ubuntu:
-    image_family: pipeline-1-24
-    project: ubuntu-os-gke-cloud
+    image_family: ubuntu-2204-lts


Out of curiosity, why stop use the GKE images ?

@ameukam So that it will pick up the latest public ubuntu 22.04 image which is cgroupv2 enabled. Ref

Ok. I had a wrong assumption of pipeline-1-24. I thought since this family is ubuntu-based:

$gcloud compute images describe-from-family pipeline-1-24 --project ubuntu-os-gke-cloud --format='value(name,selfLink)' ubuntu-gke-2204-1-24-v20221128 https://www.googleapis.com/compute/v1/projects/ubuntu-os-gke-cloud/global/images/ubuntu-gke-2204-1-24-v20221128

the node(s) will have cgroupsv2 enabled.

akhilerm · 2022-12-01T05:35:40Z

/cc @bobbypage

jobs/e2e_node/containerd/config-systemd.toml

bobbypage · 2022-12-06T01:58:23Z

One small change needed, other than that LGTM, thank you for updating!

Signed-off-by: Akhil Mohan <[email protected]>

bobbypage · 2022-12-06T02:02:32Z

/lgtm

bobbypage · 2022-12-06T02:02:51Z

/assign @dims

dims · 2022-12-06T02:16:32Z

/approve

k8s-ci-robot · 2022-12-06T02:16:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: akhilerm, dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/jobs/kubernetes/sig-node/OWNERS~~ [dims]
~~jobs/e2e_node/OWNERS~~ [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2022-12-06T02:18:43Z

@akhilerm: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

key containerd.yaml using file config/jobs/kubernetes/sig-node/containerd.yaml
key node-kubelet.yaml using file config/jobs/kubernetes/sig-node/node-kubelet.yaml
key sig-node-presubmit.yaml using file config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml

In response to this:

Fixes tests under

sig-nod-containerd

sig-node-release-blocking

sig-node-presubmits

The following tests are fixed:

ci-kubernetes-node-kubelet-containerd-eviction

ci-kubernetes-node-kubelet-containerd-resource-managers

ci-kubernetes-node-kubelet-serial-containerd

pull-kubernetes-node-kubelet-serial-containerd

pull-kubernetes-node-kubelet-serial-containerd-kubetest2

pull-kubernetes-node-kubelet-serial-pod-disruption-conditions

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

liggitt · 2022-12-06T17:25:58Z

it looks like https://testgrid.k8s.io/sig-node-containerd#pull-node-e2e went red right around when these two PRs merged:

akhilerm added 2 commits November 21, 2022 11:13

fix node-kubelet-containerd-resource-managers job

4cbbb58

Signed-off-by: Akhil Mohan <[email protected]>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 21, 2022

k8s-ci-robot requested review from endocrimes and MHBauer November 21, 2022 08:43

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 21, 2022

akhilerm commented Nov 21, 2022

View reviewed changes

k8s-ci-robot requested review from adisky, bobbypage and SergeyKanzhelev November 21, 2022 08:49

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 21, 2022

bobbypage reviewed Nov 27, 2022

View reviewed changes

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 29, 2022

akhilerm force-pushed the fix-sig-node-containerd-tests branch from 540ed38 to 7ae16b9 Compare November 30, 2022 12:27

use separate containerd config file for systemd cgroup driver

ff69de5

use the config file with systemd enabled in tests which use cgroupv2 Signed-off-by: Akhil Mohan <[email protected]>

akhilerm force-pushed the fix-sig-node-containerd-tests branch from 7ae16b9 to ff69de5 Compare November 30, 2022 12:32

ameukam reviewed Nov 30, 2022

View reviewed changes

k8s-ci-robot requested a review from bobbypage December 1, 2022 05:35

bobbypage reviewed Dec 6, 2022

View reviewed changes

jobs/e2e_node/containerd/config-systemd.toml Outdated Show resolved Hide resolved

remove quoting TOML boolean

5f6e009

Signed-off-by: Akhil Mohan <[email protected]>

k8s-ci-robot assigned bobbypage Dec 6, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 6, 2022

k8s-ci-robot assigned dims Dec 6, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 6, 2022

k8s-ci-robot merged commit 4c443dd into kubernetes:master Dec 6, 2022

akhilerm deleted the fix-sig-node-containerd-tests branch December 6, 2022 03:26

liggitt mentioned this pull request Dec 6, 2022

add systemd as cgroup driver #27943

Merged

bobbypage mentioned this pull request Dec 6, 2022

use ubuntu from gke cloud in node e2e #28190

Merged

akhilerm mentioned this pull request Dec 9, 2022

REQUEST: New membership for akhilerm kubernetes/org#3889

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix tests in sig-node related to CRIv1 update #28071

fix tests in sig-node related to CRIv1 update #28071

akhilerm commented Nov 21, 2022

k8s-ci-robot commented Nov 21, 2022

akhilerm Nov 21, 2022

akhilerm commented Nov 21, 2022

adisky commented Nov 21, 2022

bobbypage Nov 27, 2022 •

edited

Loading

bobbypage Nov 27, 2022 •

edited

Loading

akhilerm Nov 29, 2022

akhilerm Nov 29, 2022

akhilerm commented Nov 30, 2022

ameukam Nov 30, 2022

akhilerm Nov 30, 2022 •

edited

Loading

akhilerm Nov 30, 2022

ameukam Nov 30, 2022

akhilerm Nov 30, 2022

ameukam Nov 30, 2022

akhilerm commented Dec 1, 2022

bobbypage commented Dec 6, 2022 •

edited

Loading

bobbypage commented Dec 6, 2022

bobbypage commented Dec 6, 2022

dims commented Dec 6, 2022

k8s-ci-robot commented Dec 6, 2022

k8s-ci-robot commented Dec 6, 2022

liggitt commented Dec 6, 2022

fix tests in sig-node related to CRIv1 update #28071

fix tests in sig-node related to CRIv1 update #28071

Conversation

akhilerm commented Nov 21, 2022

k8s-ci-robot commented Nov 21, 2022

Choose a reason for hiding this comment

akhilerm commented Nov 21, 2022

adisky commented Nov 21, 2022

bobbypage Nov 27, 2022 • edited Loading

Choose a reason for hiding this comment

bobbypage Nov 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akhilerm commented Nov 30, 2022

Choose a reason for hiding this comment

akhilerm Nov 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akhilerm commented Dec 1, 2022

bobbypage commented Dec 6, 2022 • edited Loading

bobbypage commented Dec 6, 2022

bobbypage commented Dec 6, 2022

dims commented Dec 6, 2022

k8s-ci-robot commented Dec 6, 2022

k8s-ci-robot commented Dec 6, 2022

liggitt commented Dec 6, 2022

bobbypage Nov 27, 2022 •

edited

Loading

bobbypage Nov 27, 2022 •

edited

Loading

akhilerm Nov 30, 2022 •

edited

Loading

bobbypage commented Dec 6, 2022 •

edited

Loading