Add ARM64 node_e2e test #29192

ike-ma · 2023-03-30T18:43:32Z

Run node_e2e serial tests on GCP T2A (arm64) machine

Test locally with the following command

KUBE_BUILD_FOR_ARM64=true ARTIFACTS="${TMPDIR}" JENKINS_GCE_SSH_PRIVATE_KEY_FILE="${SSH_KEY}" \
  kubetest   --up   --test   --provider=gce   \
  --deployment=node --gcp-project="${PROJECT}"   --gcp-zone="${ZONE}" \
  "--node-args=--image-config-file="${IMAGE_CONFIG_OUT}   \
  '--node-test-args="" --kubelet-flags=""'   --node-tests=true   \
  '--test_args=--nodes=1 --focus="\[Serial\]" \
  --skip="\[Flaky\]|\[Slow\]|\[Benchmark\]|\[NodeSpecialFeature:.+\]|\[NodeSpecialFeature\]|\[NodeAlphaFeature:.+\]|\[NodeAlphaFeature\]|\[NodeFeature:Eviction\]|\[NodeFeature:NodeProblemDetector\]|\[NodeFeature:OOMScoreAdj\]"'   \
  '--timeout=300m' 2>&1 | tee -i "${TMPDIR}/build-log-$(date +%Y%m%d.%M.%S.%3N).txt"

Test Result: PASS

Ran 34 of 390 Specs in 1619.273 seconds
SUCCESS! -- 34 Passed | 0 Failed | 0 Pending | 356 Skipped
PASS

Cross-ref: kubernetes/kubernetes#117017 (Setup node_e2e to support ARM64)

k8s-ci-robot · 2023-03-30T18:43:41Z

Hi @ike-ma. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ike-ma · 2023-03-30T18:44:08Z

/cc @bobbypage

dims · 2023-03-30T19:09:16Z

/ok-to-test

dims · 2023-03-30T19:11:49Z

config/jobs/kubernetes/sig-node/node-kubelet.yaml

+          - --scenario=kubernetes_e2e
+          - --
+          - --deployment=node
+          - --env=KUBE_BUILD_FOR_ARM64=true


@ike-ma where is this used? KUBE_BUILD_FOR_ARM64

I was using it to control whether to build for ARM64 or AMD64: https://github.com/kubernetes/kubernetes/pull/117017/files/c7c8c7fe388079adf3a24e0d465dd35afea15357#diff-4e140fc3b0b901a6cdad3f45948687ebd709753a4c6b48a52e876a1fec856247

Is this still relevant? It seems to be removed from kubernetes/kubernetes#117017

+1, this should no longer be needed?

This has been removed

ike-ma · 2023-04-04T16:32:43Z

/assign @bobbypage

SergeyKanzhelev · 2023-04-26T17:36:38Z

/assign @mmiranda96

mmiranda96

Added a couple of comments.

mmiranda96 · 2023-05-01T17:44:51Z

jobs/e2e_node/arm/image-config-serial.yaml

+# `gcloud compute --project <to-project> disks create <image name> --image=https://www.googleapis.com/compute/v1/projects/<from-project>/global/images/<image-name>`
+# `gcloud compute --project <to-project> images create <image-name> --source-disk=<image-name>`
+images:
+  ubuntu:


Do you plan to use COS as well? We can add both configs here.

Currently only Ubuntu is in scope, will add COS after more testing in a separate PR.

mmiranda96 · 2023-05-01T17:45:08Z

jobs/e2e_node/arm/image-config-serial.yaml

+# `gcloud compute --project <to-project> images create <image-name> --source-disk=<image-name>`
+images:
+  ubuntu:
+    image: ubuntu-gke-2204-1-24-arm64-v20230217


I suggest that we use image-family instead, otherwise we'd need to update this job constantly.

Done. Updated.

mmiranda96 · 2023-05-01T17:46:18Z

jobs/e2e_node/arm/init.yaml

+runcmd:
+  - echo "Test run from /tmp folder, remounting it"
+  - mount /tmp /tmp -o remount,exec,suid
+


Not sure if having empty lines across a YAML list is supported.

The empty lines will be ignored by the parser. I have tested it without any issue.

mmiranda96 · 2023-05-01T17:51:53Z

config/jobs/kubernetes/sig-node/node-kubelet.yaml

+          - --scenario=kubernetes_e2e
+          - --
+          - --deployment=node
+          - --env=KUBE_BUILD_FOR_ARM64=true


Is this still relevant? It seems to be removed from kubernetes/kubernetes#117017

mmiranda96 · 2023-05-01T17:53:08Z

/retest

mmiranda96 · 2023-05-01T18:05:06Z

/assign @ndixita

SergeyKanzhelev · 2023-05-17T16:39:02Z

@ike-ma can you please review the @mmiranda96 's comments. I'd like to merge this so we can run this test for the changes in k/k repository

tzneal · 2023-05-17T17:11:03Z

/cc

bobbypage · 2023-05-18T17:57:45Z

config/jobs/kubernetes/sig-node/node-kubelet.yaml

+    description: "Run serial node e2e tests on ARM64 environment on Ubuntu"
+  labels:
+    preset-service-account: "true"
+    preset-k8s-ssh: "true"


This will likely need

preset-dind-enabled: "true"

because ARM build needs to use dockerized build so we need to enable docker-in-docker.

Also for docker-in-docker it will also need to be marked as

securityContext: privileged: true

Thanks for the pointer, added.

bobbypage · 2023-05-18T17:59:28Z

config/jobs/kubernetes/sig-node/node-kubelet.yaml

+          - --deployment=node
+          - --env=KUBE_BUILD_FOR_ARM64=true
+          - --gcp-zone=us-central1-a
+          - --node-args=--image-config-file=/workspace/test-infra/jobs/e2e_node/arm/image-config.yaml


This will likely need the new flags introduced in https://github.dev/kubernetes/kubernetes/pull/117017, i.e.

--use-dockerized-build=true --target-build-arch=linux/arm64

Done, updated with the latest command.

seems like there is a typo in the config file,

--image-config-file=/workspace/test-infra/jobs/e2e_node/arm/image-config.yaml

while it should be

--image-config-file=/workspace/test-infra/jobs/e2e_node/arm/image-config-serial.yaml

Can you confirm?

/hold

Good catch. Named as image-config.yaml in my own test. Updated to match.

ike-ma · 2023-05-24T18:29:36Z

Addressed all review comments, and updated the test config based on #28888.

Tested with

go run /usr/local/google/home/ikema/go-k8s-oss/src/k8s.io/kubernetes/test/e2e_node/runner/remote/run_remote.go --cleanup -vmodule=*=4 --ssh-env=gce --results-dir=/tmp/e2e-node-results/ason --project=ikema-gke-dev-2 --use-dockerized-build=true --target-build-arch=linux/arm64 --zone=us-central1-a --ssh-user=ikema --ssh-key=/usr/local/google/home/ikema/.ssh/google_compute_engine --ginkgo-flags='--nodes=1 --focus="\[Serial\]" --skip="\[Flaky\]|\[Slow\]|\[Benchmark\]|\[NodeSpecialFeature:.+\]|\[NodeSpecialFeature\]|\[NodeAlphaFeature:.+\]|\[NodeAlphaFeature\]|\[NodeFeature:Eviction\]|\[NodeFeature:NodeProblemDetector\]|\[NodeFeature:OOMScoreAdj\]|\[NodeFeature:DevicePluginProbe\]|\[NodeConformance\]" '  --test-timeout=5h0m0s --image-config-file=/tmp/e2e-node-results/ason/image-config.yaml

BenTheElder · 2023-05-24T18:49:52Z

config/jobs/kubernetes/sig-node/node-kubelet.yaml

+    containers:
+      - image: gcr.io/k8s-staging-test-infra/kubekins-e2e:v20230513-7e1db2f1bb-master
+        args:
+          - --repo=k8s.io/kubernetes=master


we should stop writing new jobs with bootstrap.py instead of decorate: true

these jobs have been logging a warning to migrate for years

this doesn't have to block adding the job, but please seriously consider migrating.

https://gist.github.com/dims/c1296f8ed42238baea0a5fcae45f4cf4

Acknowledge, thanks for the gist pointer, will follow up in a separate PR.

BenTheElder

/approve
with a request to please switch to prow native cloning etc. (decorate: true) and stop using the bootstrap.py entrypoint script.

k8s-ci-robot · 2023-05-24T18:51:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, ike-ma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/jobs/kubernetes/sig-node/OWNERS~~ [BenTheElder]
~~jobs/e2e_node/OWNERS~~ [BenTheElder]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Run node_e2e serial tests on GCP T2A (arm64) machine

ike-ma · 2023-05-25T23:31:09Z

Addressed review comments, and rebased to upstream/master

chendave · 2023-05-27T02:29:56Z

/hold cancel
/lgtm
thanks!

chendave · 2023-05-27T02:37:16Z

/test pull-test-infra-unit-test

k8s-ci-robot · 2023-05-27T02:45:43Z

@ike-ma: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

key node-kubelet.yaml using file config/jobs/kubernetes/sig-node/node-kubelet.yaml

In response to this:

Run node_e2e serial tests on GCP T2A (arm64) machine

Test locally with the following command

KUBE_BUILD_FOR_ARM64=true ARTIFACTS="${TMPDIR}" JENKINS_GCE_SSH_PRIVATE_KEY_FILE="${SSH_KEY}" \
 kubetest   --up   --test   --provider=gce   \
 --deployment=node --gcp-project="${PROJECT}"   --gcp-zone="${ZONE}" \
 "--node-args=--image-config-file="${IMAGE_CONFIG_OUT}   \
 '--node-test-args="" --kubelet-flags=""'   --node-tests=true   \
 '--test_args=--nodes=1 --focus="\[Serial\]" \
 --skip="\[Flaky\]|\[Slow\]|\[Benchmark\]|\[NodeSpecialFeature:.+\]|\[NodeSpecialFeature\]|\[NodeAlphaFeature:.+\]|\[NodeAlphaFeature\]|\[NodeFeature:Eviction\]|\[NodeFeature:NodeProblemDetector\]|\[NodeFeature:OOMScoreAdj\]"'   \
 '--timeout=300m' 2>&1 | tee -i "${TMPDIR}/build-log-$(date +%Y%m%d.%M.%S.%3N).txt"

Test Result: PASS

Ran 34 of 390 Specs in 1619.273 seconds
SUCCESS! -- 34 Passed | 0 Failed | 0 Pending | 356 Skipped
PASS

Cross-ref: kubernetes/kubernetes#117017 (Setup node_e2e to support ARM64)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

SergeyKanzhelev · 2023-05-27T05:53:09Z

Cool! Thank you for merging. It is failing: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-node-arm64-ubuntu-serial/1662289187593785344

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 30, 2023

k8s-ci-robot added area/config Issues or PRs related to code in /config area/jobs sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Mar 30, 2023

k8s-ci-robot requested review from SergeyKanzhelev and tallclair March 30, 2023 18:43

k8s-ci-robot requested a review from bobbypage March 30, 2023 18:44

ike-ma mentioned this pull request Mar 30, 2023

Setup e2e_node to support testing on ARM64 kubernetes/kubernetes#117017

Merged

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 30, 2023

dims reviewed Mar 30, 2023

View reviewed changes

ike-ma mentioned this pull request Mar 30, 2023

Add GNU C compiler for arm64 architecture #29197

Closed

k8s-ci-robot assigned bobbypage Apr 4, 2023

k8s-ci-robot assigned mmiranda96 Apr 26, 2023

mmiranda96 reviewed May 1, 2023

View reviewed changes

k8s-ci-robot assigned ndixita May 1, 2023

k8s-ci-robot requested a review from tzneal May 17, 2023 17:11

bobbypage reviewed May 18, 2023

View reviewed changes

SergeyKanzhelev mentioned this pull request May 23, 2023

Support running e2e-node tests on ARM64 machine type #28888

Closed

ike-ma force-pushed the arm-ci branch from 09319ca to 873ea4c Compare May 24, 2023 18:28

BenTheElder reviewed May 24, 2023

View reviewed changes

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels May 24, 2023

Add ARM64 node_e2e test

b747cfd

Run node_e2e serial tests on GCP T2A (arm64) machine

ike-ma force-pushed the arm-ci branch from 873ea4c to b747cfd Compare May 25, 2023 23:30

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 27, 2023

k8s-ci-robot assigned chendave May 27, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 27, 2023

k8s-ci-robot merged commit 553d4c1 into kubernetes:master May 27, 2023

chendave mentioned this pull request May 29, 2023

Fix the ARM64 node_e2e test #29617

Merged

ike-ma mentioned this pull request May 31, 2023

REQUEST: New membership for ike-ma@ kubernetes/org#4257

Closed

9 tasks

pacoxu mentioned this pull request Jun 6, 2023

[umbrella issue] arm ci for node e2e testing #29693

Closed

rayowang mentioned this pull request Jun 17, 2023

REQUEST: New membership for rayowang kubernetes/org#4302

Closed

9 tasks

Add ARM64 node_e2e test #29192

Add ARM64 node_e2e test #29192

Conversation

ike-ma commented Mar 30, 2023 • edited Loading

k8s-ci-robot commented Mar 30, 2023

ike-ma commented Mar 30, 2023

dims commented Mar 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ike-ma commented Apr 4, 2023

SergeyKanzhelev commented Apr 26, 2023

mmiranda96 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ike-ma May 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmiranda96 commented May 1, 2023

mmiranda96 commented May 1, 2023

SergeyKanzhelev commented May 17, 2023

tzneal commented May 17, 2023

bobbypage May 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chendave May 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ike-ma commented May 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 24, 2023

ike-ma commented May 25, 2023

chendave commented May 27, 2023

chendave commented May 27, 2023

k8s-ci-robot commented May 27, 2023

SergeyKanzhelev commented May 27, 2023

ike-ma commented Mar 30, 2023 •

edited

Loading

ike-ma May 24, 2023 •

edited

Loading

bobbypage May 18, 2023 •

edited

Loading

chendave May 25, 2023 •

edited

Loading