scale-test: Use single nodes instance group for AWS #16204

hakman · 2024-01-03T04:59:35Z

hakman · 2024-01-03T05:01:49Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-03T05:17:22Z

  AutoscalingGroup/nodes.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io
  	Granularity         	1Minute
  	InstanceProtection  	false
  	LaunchTemplate      	name:nodes.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io
  	LoadBalancers       	[]
  	MaxInstanceLifetime 	0
  	MaxSize             	5000
  	Metrics             	[GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
  	MinSize             	5000
  	MixedInstanceOverrides	[t3a.medium, t3.medium, t2.medium, t3a.large, c5a.large, t3.large, c5.large, m5a.large, m6a.large, m5.large, c4.large, c7a.large, r5a.large, r6a.large, m7a.large]
  	MixedSpotMaxPrice   	
  	Subnets             	[name:us-east-2a-1.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2a-4.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2a-7.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2a-10.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2b-2.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2b-5.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2b-8.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2b-11.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2c-3.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2c-6.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2c-9.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, name:us-east-2c-12.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io]
  	SuspendProcesses    	[]
  	Tags                	{kubernetes.io/cluster/e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io: owned, aws-node-termination-handler/managed: , k8s.io/cluster-autoscaler/node-template/label/node-role.kubernetes.io/node: , k8s.io/role/node: 1, kops.k8s.io/instancegroup: nodes, Name: nodes.e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io, KubernetesCluster: e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io}
  	TargetGroups        	[]

INSTANCE GROUPS
NAME				ROLE		MACHINETYPE	MIN	MAX	SUBNETS
control-plane-us-east-2a	ControlPlane	c5.18xlarge	1	1	us-east-2a
control-plane-us-east-2b	ControlPlane	c5.18xlarge	1	1	us-east-2b
control-plane-us-east-2c	ControlPlane	c5.18xlarge	1	1	us-east-2c
nodes				Node		t3a.medium	5000	5000	us-east-2a-1,us-east-2a-4,us-east-2a-7,us-east-2a-10,us-east-2b-2,us-east-2b-5,us-east-2b-8,us-east-2b-11,us-east-2c-3,us-east-2c-6,us-east-2c-9,us-east-2c-12

hakman · 2024-01-03T08:45:45Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-03T09:43:14Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-03T11:21:22Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-03T15:05:37Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakuna-matatah · 2024-01-03T15:47:33Z

Thanks @hakman for making this change, I was looking in the account to see how the IP allocation was happening across subnets but it looks like it's deleting the nodes already, scale test failed w following error.

+ kubetest2 kops -v=2 --cloud-provider=aws --cluster-name=e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io --kops-binary-path=/home/prow/go/src/k8s.io/bin/kops --admin-access=0.0.0.0/0 --env=KOPS_FEATURE_FLAGS=AWSSingleNodesInstanceGroup, --validation-wait=55m --up --kubernetes-version=v1.29.0 '--create-args=--network-cidr=10.0.0.0/16,10.1.0.0/16,10.2.0.0/16,10.3.0.0/16,10.4.0.0/16,10.5.0.0/16,10.6.0.0/16,10.7.0.0/16,10.8.0.0/16,10.9.0.0/16,10.10.0.0/16,10.11.0.0/16,10.12.0.0/16 --networking=amazonvpc --set spec.networking.amazonVPC.env=ENABLE_PREFIX_DELEGATION=true --image=ssm:/aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id --set spec.etcdClusters[0].manager.listenMetricsURLs=http://localhost:2382 --set spec.etcdClusters[0].manager.env=ETCD_QUOTA_BACKEND_BYTES=8589934592 --set spec.etcdClusters[1].manager.env=ETCD_QUOTA_BACKEND_BYTES=8589934592 --set spec.kubelet.maxPods=96 --set spec.kubeScheduler.authorizationAlwaysAllowPaths=/healthz --set spec.kubeScheduler.authorizationAlwaysAllowPaths=/metrics --set spec.kubeScheduler.kubeAPIQPS=500 --set spec.kubeScheduler.kubeAPIBurst=500 --set spec.kubeScheduler.enableProfiling=true --set spec.kubeScheduler.enableContentionProfiling=true --set spec.kubeControllerManager.endpointUpdatesBatchPeriod=500ms --set spec.kubeControllerManager.endpointSliceUpdatesBatchPeriod=500ms --set spec.kubeControllerManager.kubeAPIQPS=100 --set spec.kubeControllerManager.kubeAPIBurst=100 --set spec.kubeControllerManager.enableProfiling=true --set spec.kubeControllerManager.enableContentionProfiling=true --set spec.kubeAPIServer.maxRequestsInflight=800 --set spec.kubeAPIServer.maxMutatingRequestsInflight=400 --set spec.kubeAPIServer.enableProfiling=true --set spec.kubeAPIServer.enableContentionProfiling=true --set spec.kubeAPIServer.anonymousAuth=true --set spec.kubeProxy.metricsBindAddress=0.0.0.0:10249 --node-count=5000 --dns none --node-size=t3a.medium,t3.medium,t2.medium,t3a.large,c5a.large,t3.large,c5.large,m5a.large,m6a.large,m5.large,c4.large,c7a.large,r5a.large,r6a.large,m7a.large --node-volume-size=20 --control-plane-count=3 --master-size=c5.18xlarge --zones=us-east-2a,us-east-2b,us-east-2c --set spec.cloudProvider.aws.ebsCSIDriver.kubeAPIQPS=100 --set spec.cloudProvider.aws.ebsCSIDriver.kubeAPIBurst=100'
I0103 15:10:08.070486   14416 app.go:61] The files in RunDir shall not be part of Artifacts
I0103 15:10:08.070516   14416 app.go:62] pass rundir-in-artifacts flag True for RunDir to be part of Artifacts
I0103 15:10:08.070538   14416 app.go:64] RunDir for this run: "/home/prow/go/src/k8s.io/kops/_rundir/82bd5d80-4c72-4594-99c5-972a62b27c51"
I0103 15:10:08.114585   14416 app.go:130] ID for this run: "82bd5d80-4c72-4594-99c5-972a62b27c51"
I0103 15:10:08.121633   14416 up.go:58] Cleaning up any leaked resources from previous cluster
I0103 15:10:08.333799   14416 dumplogs.go:47] /home/prow/go/src/k8s.io/bin/kops toolbox dump --name e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io --dir /logs/artifacts --private-key /tmp/kops/e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io/id_ed25519 --ssh-user ubuntu
I0103 15:10:08.342055   14452 featureflag.go:168] FeatureFlag "AWSSingleNodesInstanceGroup"=true
W0103 15:10:37.878181   14452 toolbox_dump.go:172] cannot load kubeconfig settings for "e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io": context "e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io" does not exist

hakman · 2024-01-03T16:23:53Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-03T16:32:59Z

@hakuna-matatah Give it another try

hakman · 2024-01-03T18:08:42Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-03T19:15:24Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-04T05:48:25Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-04T06:26:50Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-04T08:17:50Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-04T17:35:09Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-05T09:49:10Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-05T19:43:55Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-06T07:32:33Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-06T07:38:19Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-06T13:31:37Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-06T18:41:55Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-07T09:03:30Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-07T10:49:34Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

upodroid · 2024-01-07T13:55:49Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

pod got killed :(

hakman · 2024-01-08T08:58:37Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

hakman · 2024-01-08T11:08:09Z

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

upodroid

LGTM

hakman · 2024-01-08T12:01:39Z

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2

hakman · 2024-01-08T13:36:42Z

/approve

k8s-ci-robot · 2024-01-08T13:37:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman, upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hakuna-matatah · 2024-01-10T16:19:14Z

This PR is relevant to this issue

k8s-ci-robot requested review from dims, hakuna-matatah and upodroid January 3, 2024 04:59

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 3, 2024

hakman force-pushed the single-nodes-ig branch from 6527c8c to df2e66a Compare January 3, 2024 17:59

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 4, 2024

hakman force-pushed the single-nodes-ig branch from 85e9e6e to 09aa060 Compare January 6, 2024 07:37

hakman added 2 commits January 8, 2024 10:55

test: Wait 60 min for cluster to be deleted

6f04995

scale-test: Fix cluster deletion twice

f3c4d96

hakman force-pushed the single-nodes-ig branch from 5c7587b to 553e77d Compare January 8, 2024 08:58

hakman force-pushed the single-nodes-ig branch 2 times, most recently from 59504fb to 6ed0d60 Compare January 8, 2024 11:35

hakman added 2 commits January 8, 2024 13:35

scale-test: Use single nodes instance group for AWS

6492bed

scale-test: Increase CCM ConcurrentNodeSyncs

09c1703

hakman force-pushed the single-nodes-ig branch from 6ed0d60 to 09c1703 Compare January 8, 2024 11:36

upodroid approved these changes Jan 8, 2024

View reviewed changes

k8s-ci-robot assigned upodroid Jan 8, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 8, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 8, 2024

k8s-ci-robot merged commit f3c9c05 into kubernetes:master Jan 8, 2024
23 checks passed

k8s-ci-robot added this to the v1.29 milestone Jan 8, 2024

hakuna-matatah mentioned this pull request Jan 10, 2024

KOPS worker nodes distribution with ASG's with N subnets. #16180

Closed

BrewTestBot mentioned this pull request May 16, 2024

kops 1.29.0 Homebrew/homebrew-core#171874

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scale-test: Use single nodes instance group for AWS #16204

scale-test: Use single nodes instance group for AWS #16204

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024 •

edited

Loading

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakuna-matatah commented Jan 3, 2024 •

edited

Loading

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 4, 2024

hakman commented Jan 4, 2024

hakman commented Jan 4, 2024

hakman commented Jan 4, 2024

hakman commented Jan 5, 2024

hakman commented Jan 5, 2024

hakman commented Jan 6, 2024

hakman commented Jan 6, 2024

hakman commented Jan 6, 2024

hakman commented Jan 6, 2024

hakman commented Jan 7, 2024

hakman commented Jan 7, 2024

upodroid commented Jan 7, 2024

hakman commented Jan 8, 2024

hakman commented Jan 8, 2024

upodroid left a comment

hakman commented Jan 8, 2024

hakman commented Jan 8, 2024

k8s-ci-robot commented Jan 8, 2024

hakuna-matatah commented Jan 10, 2024

scale-test: Use single nodes instance group for AWS #16204

scale-test: Use single nodes instance group for AWS #16204

Conversation

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024 • edited Loading

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakuna-matatah commented Jan 3, 2024 • edited Loading

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 3, 2024

hakman commented Jan 4, 2024

hakman commented Jan 4, 2024

hakman commented Jan 4, 2024

hakman commented Jan 4, 2024

hakman commented Jan 5, 2024

hakman commented Jan 5, 2024

hakman commented Jan 6, 2024

hakman commented Jan 6, 2024

hakman commented Jan 6, 2024

hakman commented Jan 6, 2024

hakman commented Jan 7, 2024

hakman commented Jan 7, 2024

upodroid commented Jan 7, 2024

hakman commented Jan 8, 2024

hakman commented Jan 8, 2024

upodroid left a comment

Choose a reason for hiding this comment

hakman commented Jan 8, 2024

hakman commented Jan 8, 2024

k8s-ci-robot commented Jan 8, 2024

hakuna-matatah commented Jan 10, 2024

hakman commented Jan 3, 2024 •

edited

Loading

hakuna-matatah commented Jan 3, 2024 •

edited

Loading