Running node e2e tests on AWS

[upodroid]

Kubelet and containerd e2e tests run on google.com GCP projects today and we would like to start running them on AWS as we have a lot of credits there and more importantly, eliminate GCE specific behaviors. Ideally, the only requirement is that the node is bootstrapped by cloud-init. The k/k repo will have cloud-provider specific code to create instances in GCP or AWS, or X, but it will bootstrap nodes from a standard cloud-init file per distribution.

All the kubelet/containerd tests are single node tests and much easier to refactor than kubernetes/kubernetes#78995

Reading/background/relevant files:

• https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/e2e-node-tests.md
• https://github.com/kubernetes/test-infra/tree/master/jobs/e2e_node/containerd
• https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-node/containerd.yaml

Initial testing + Bug reports
kubernetes/kubernetes#116114 AWS public AMI
kubernetes/kubernetes#116112 GCE public Ubuntu Image

Next steps:

• Create an AWS account in k/k8s.io for kubelet/containerd test
• Merge test: support remote e2e node test runs on AWS kubernetes#116236
• Merge Add a cloud agnostic configure.sh to bootstrap Ubuntu test nodes containerd/containerd#8170
• resolve the issues in k/k
• Merge Use a public ubuntu image for kubelet node e2e tests #28856
• Add new CI prow jobs
• Promote tests to pre-submits
• Promote tests to release blocking

/sig node
/sig k8s-infra

[upodroid]

/priority important-longterm

[SergeyKanzhelev]

/remove-kind bug
/kind feature

[upodroid]

There has been many changes since this issue was opened.

@dims @tzneal

New plan:

• The AWS e2e test logic lives in https://github.com/kubernetes-sigs/provider-aws-test-infra
• We merged the first "canary" job Create Initial EC2 sig-node e2e tests #29776
• Create a dedicated AWS account for node e2e tests(currently using the kops e2e account)
• Configure cross-account IRSA https://aws.amazon.com/blogs/containers/cross-account-iam-roles-for-kubernetes-service-accounts/
• These new jobs will run on the EKS clusters and can leverage IRSA to access this account.(boskos is not required as tests have max_concurrency set and thousands of instances can run in a single region with the correct limit updates)
• Using this k/k PR to test the jobs: Switch to e2 machine types kubernetes#118626

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[dims]

we have a bunch now

https://testgrid.k8s.io/amazon-ec2-node#Summary&width=20

/close

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

we have a bunch now

https://testgrid.k8s.io/amazon-ec2-node#Summary&width=20

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.