aws: Create subnets for additional network CIDRs #15805

hakman · 2023-08-20T16:40:37Z

This will be needed for scale tests, to be able to use clusters with more than 500 nodes.

hakman · 2023-08-22T13:53:29Z

dims · 2023-08-22T19:02:04Z

prateekgogia · 2023-08-23T03:28:58Z

cmd/kops/create_cluster.go

@@ -300,7 +299,7 @@ func NewCmdCreateCluster(f *util.Factory, out io.Writer) *cobra.Command {
 	cmd.RegisterFlagCompletionFunc("subnets", completeSubnetID(options))
 	cmd.Flags().StringSliceVar(&options.UtilitySubnetIDs, "utility-subnets", options.UtilitySubnetIDs, "Shared utility subnets to use")
 	cmd.RegisterFlagCompletionFunc("utility-subnets", completeSubnetID(options))
-	cmd.Flags().StringVar(&options.NetworkCIDR, "network-cidr", options.NetworkCIDR, "Network CIDR to use")
+	cmd.Flags().StringSliceVar(&options.NetworkCIDRs, "network-cidr", options.NetworkCIDRs, "Network CIDR(s) to use")


Do we need to make this change here as well - https://github.com/kubernetes/kops/blob/master/tests/e2e/kubetest2-kops/deployer/up.go#L112?

I don't see any mention of network-cidr there.

IIUC, kubetest2 uses that interface which I linked earlier and we would need to provide this flag there as well.

I think we can pass them via the "passthrough" CreateArgs:

kops/tests/e2e/kubetest2-kops/deployer/up.go

Lines 126 to 135 in 9e0db3d

if d.CreateArgs != "" {

if strings.Contains(d.CreateArgs, "arm64") {

isArm = true

}

createArgs, err := shlex.Split(d.CreateArgs)

if err != nil {

return err

}

args = append(args, createArgs...)

}

prateekgogia · 2023-08-23T03:34:02Z

cmd/kops/create_cluster.go

@@ -300,7 +299,7 @@ func NewCmdCreateCluster(f *util.Factory, out io.Writer) *cobra.Command {
 	cmd.RegisterFlagCompletionFunc("subnets", completeSubnetID(options))
 	cmd.Flags().StringSliceVar(&options.UtilitySubnetIDs, "utility-subnets", options.UtilitySubnetIDs, "Shared utility subnets to use")
 	cmd.RegisterFlagCompletionFunc("utility-subnets", completeSubnetID(options))
-	cmd.Flags().StringVar(&options.NetworkCIDR, "network-cidr", options.NetworkCIDR, "Network CIDR to use")
+	cmd.Flags().StringSliceVar(&options.NetworkCIDRs, "network-cidr", options.NetworkCIDRs, "Network CIDR(s) to use")


My suggestion would be to add a new flag called additional-network-cidrs to be backward compatible.

It is already backwards compatible. Nothing changes from user point of view.

Sorry, might be missing something here but won't going from StringVar to StringSliceVar require users to make a change to pass a slice of strings now?

You can pass a slice as multiple --network-cidr flags or comma separated.
For example, with this change, one can just do:

--network-cidr=10.0.0.0/16 --network-cidr=10.1.0.0/16 --network-cidr=10.2.0.0/16 --network-cidr=10.3.0.0/16 --network-cidr=10.4.0.0/16 --network-cidr=10.5.0.0/16

Right, it's a nice feature of the flags library!

justinsb · 2023-08-25T15:20:31Z

upup/pkg/fi/cloudup/new_cluster.go

+	if len(opt.NetworkCIDRs) > 1 {
+		for i, cidr := range opt.NetworkCIDRs[1:] {
+			zoneName := opt.Zones[i%len(opt.Zones)]
+			subnetName := fmt.Sprintf("%s-%d", zoneName, i+1)


I think this is pretty specific to AWS. Also the way we round-robin around the zones is convenient, but might be surprising (along with the numbering).

Should we feature-flag this feature (and/or restrict to AWS only at first)?

Additional network CIDRs are already restricted to AWS in validation, so it should not be a problem.
Round-robin may be surprising, but could not think of any other behaviour when adding multiple network cirds and multiple zones. In any case, I doubt many people will create clusters with multiple network cidrs. It's quite an edge case. 😄

One possibility would be to put the -%d suffix on the first subnet as well when len(opt.NetworkCIDRs) > 1 but on AWS the primary CIDR of a VPC does have slightly different semantics.

I agree about the slightly different semantics. This is why I preferred to have different names for additional network CIDRs.

johngmyers · 2023-08-26T03:59:05Z

tests/integration/create_cluster/complex/expected-v1alpha2.yaml

+  - cidr: 10.1.0.0/16
+    name: us-test-1a-1
+    type: Public
+    zone: us-test-1a
+  - cidr: 10.2.0.0/16
+    name: us-test-1a-2
+    type: Public
+    zone: us-test-1a
+  - cidr: 10.3.0.0/16
+    name: us-test-1a-3
+    type: Public
+    zone: us-test-1a


I'm thinking this test should have a second zone. And we should have a private_complex test.

Agreed, done.

johngmyers · 2023-08-26T04:05:28Z

/lgtm

Holding off on approval to give opportunity to add those additional test cases. I would be okay with this going in as-is.

justinsb · 2023-08-26T11:27:12Z

Thanks @hakman

We discussed in office hours, this is good to go I think, my concern was whether we should have a feature flag. @johngmyers agreed with you that we didn't, so...

/approve
/lgtm

k8s-ci-robot · 2023-08-26T11:27:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 20, 2023

k8s-ci-robot requested a review from johngmyers August 20, 2023 16:40

k8s-ci-robot added the area/documentation label Aug 20, 2023

k8s-ci-robot requested a review from zetaab August 20, 2023 16:40

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 21, 2023

hakman force-pushed the aws_additional_network_cidrs branch from 5b8e669 to f63bdaa Compare August 21, 2023 06:20

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 21, 2023

hakman added the kind/office-hours label Aug 22, 2023

k8s-ci-robot requested a review from justinsb August 22, 2023 13:53

k8s-ci-robot assigned justinsb Aug 22, 2023

prateekgogia reviewed Aug 23, 2023

View reviewed changes

justinsb reviewed Aug 25, 2023

View reviewed changes

johngmyers reviewed Aug 26, 2023

View reviewed changes

k8s-ci-robot assigned johngmyers Aug 26, 2023

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Aug 26, 2023

aws: Create subnets for additional network CIDRs

4e0e28a

hakman force-pushed the aws_additional_network_cidrs branch from 60c656d to f41e93f Compare August 26, 2023 07:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 26, 2023

Apply suggestions from code review

35e7bba

hakman force-pushed the aws_additional_network_cidrs branch from f41e93f to 35e7bba Compare August 26, 2023 07:38

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 26, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 26, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 26, 2023

k8s-ci-robot merged commit 1b31a46 into kubernetes:master Aug 26, 2023
8 checks passed

k8s-ci-robot added this to the v1.28 milestone Aug 26, 2023

hakman mentioned this pull request Sep 1, 2023

Allow more than one instance type for workers when creating a cluster #15849

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws: Create subnets for additional network CIDRs #15805

aws: Create subnets for additional network CIDRs #15805

hakman commented Aug 20, 2023 •

edited

Loading

hakman commented Aug 22, 2023

dims commented Aug 22, 2023

prateekgogia Aug 23, 2023

hakman Aug 23, 2023

prateekgogia Aug 23, 2023

justinsb Aug 25, 2023

prateekgogia Aug 23, 2023

hakman Aug 23, 2023

prateekgogia Aug 23, 2023

hakman Aug 23, 2023 •

edited

Loading

justinsb Aug 25, 2023

justinsb Aug 25, 2023

hakman Aug 25, 2023

johngmyers Aug 26, 2023

hakman Aug 26, 2023

johngmyers Aug 26, 2023

hakman Aug 26, 2023

johngmyers commented Aug 26, 2023

justinsb commented Aug 26, 2023

k8s-ci-robot commented Aug 26, 2023

	if d.CreateArgs != "" {
	if strings.Contains(d.CreateArgs, "arm64") {
	isArm = true
	}
	createArgs, err := shlex.Split(d.CreateArgs)
	if err != nil {
	return err
	}
	args = append(args, createArgs...)
	}

aws: Create subnets for additional network CIDRs #15805

aws: Create subnets for additional network CIDRs #15805

Conversation

hakman commented Aug 20, 2023 • edited Loading

hakman commented Aug 22, 2023

dims commented Aug 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hakman Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johngmyers commented Aug 26, 2023

justinsb commented Aug 26, 2023

k8s-ci-robot commented Aug 26, 2023

hakman commented Aug 20, 2023 •

edited

Loading

hakman Aug 23, 2023 •

edited

Loading