Separate subnets for machine pools

[alex-dabija]

User Story

- As a cluster admin, I want the machine pool nodes to be on separate AWS subnets (one per availability zone) in order to have clear network boundaries.

Details, Background

Cluster API for AWS (CAPA) adds the nodes of a new machine pool to the same AWS subnets that are used by the cluster's control plane. In contrast, Giant Swarm clusters create separate subnets (one per availability zone) for each machine pool.

The gap between Cluster API and Giant Swarm clusters can be reduced by changing the default Cluster API behavior to create subnets (one per availability zone) for each machine pool.

Blocked by / depends on

None

[fiunchinho]

Apart from trying to change the CAPA default, would it make sense to supply this configuration as part of our future GS release concept? That way, we would not care about the CAPA default. The templates in our release would contain the recommended configuration.

[calvix]

The main problem is that this is not configurable or achievable with CAPA, CAPA either uses CP subnets or you need to provide subnet-id for the machine pools. No default will save us or any other template.

we cannot even build a simple operator that would create the subnets for us, as the subnets must exist at the time you are trying to create a machine pool, so the operator would have to know it beforehand and create them and then inject their ID during machine pool CR creation via webhook which I have no idea how to achieve.

[calvix]

blocked by upstream PR kubernetes-sigs/cluster-api-provider-aws#2634

nobody responded and its waiting for merge

[calvix]

upstream PR is merged, we need to wait for a new release and test this with the new subnet operator

[AverageMarcus]

@alex-dabija What is the desired "user interface" for this?

• Should all machine pools always get their own subnets?
• If not, how do we want to expose that to customers? Complete (messy) control via config? This will be easy to get wrong and have clusters not come up right.
• How should the control plane subnets look? Should it be 1 machine per AZ? Should the control plane be made up of 3 very tiny subnets?
• Should there instead be a flag on machine pools to indicate "use own subnets" ?

[alex-dabija]

• Should all machine pools always get their own subnets?

No, I think that cluster-aws should allow the cluster admin to choose which subnets are used by tagging them. The machine pool should have a tag filter to choose the correct subnets. I don't know if this is possible with CAPA though.

• If not, how do we want to expose that to customers? Complete (messy) control via config? This will be easy to get wrong and have clusters not come up right.

It's more important now to allow for separate subnets to be configured for each machine pool and availability zone. We already have this on AWS vintage. Yes, it true that on vintage we are very inflexible: CIDRs are chosen automatically from a larger one, etc.

• How should the control plane subnets look? Should it be 1 machine per AZ? Should the control plane be made up of 3 very tiny subnets?

Subnets are per availability zone, which means we need 3 for a highly-available cluster.

• Should there instead be a flag on machine pools to indicate "use own subnets" ?

What do you mean by "use own subnets"? Is it CAPA creating the subnets?

I was thinking that in values.yaml we could have a list of subnets, each subnet would have a CIDR and a list of tags. On the control plane configuration and machine pool configuration tag filters could be used to choose which subnets should be used. This would also align with all the other components which need their own separate subnets: API LB, ingress LB, VPC endpoints, bastion hosts & transit gateway.

Yes, overall this would not be the best UX and the user could do lots of mistakes but it's going to allow us to be very responsive to future customer configurations. The UX limitations could be mitigated by:

• the recent --cluster-type support that was added to kubectl-gs which allows us to have different cluster shapes. These could have their own pre-computed network layout;
• an admission control to validate the network configuration and to provide feedback to the user if mistakes are present.

[AverageMarcus]

Work in progress implementation: giantswarm/cluster-aws#196

Current state:
Default values work out-of-the-box when creating public workload clusters but currently having trouble launching private clusters (which appropriately changed values). The subnets seem to be created correctly (and tagged) but the control plane fails to bring up the first machine.

Some things still need to be handled:

• There's a lot of wasted IPs in the CIDR range when using the default values. e.b. the bastion subnets should use something like a /29
• This change in default values is a breaking change and we'll need to think about how we handle cluster upgrades (they can upgrade cleanly is the values are overwritten)

[AverageMarcus]

Current state of things:

The changes work if only creating one grouping of subnets (the same number of subnets as availabilityZoneUsageLimit).

When creating multiple groupings of subnets the control plane nodes fail to come up correctly.

Example configuration snippet:

    controlPlane:
      replicas: 3
      subnetTags:
        - subnet.giantswarm.io/role: control-plane
    bastion:
      subnetTags:
        - subnet.giantswarm.io/role: bastion
    machinePools:
    - instanceType: m5.xlarge
      maxSize: 10
      minSize: 3
      name: machine-pool0
      rootVolumeSizeGB: 300
      subnetTags:
        - subnet.giantswarm.io/role: worker
    network:
      topologyMode: GiantSwarmManaged
      vpcMode: private
      apiMode: private
      dnsMode: private
      vpcCIDR: 10.228.0.0/16
      availabilityZoneUsageLimit: 3
      subnets:
      - cidrBlocks:
        - cidr: 10.228.48.0/20
          availabilityZone: a
        - cidr: 10.228.64.0/20
          availabilityZone: b
        - cidr: 10.228.80.0/20
          availabilityZone: c
        isPublic: false
        tags:
          subnet.giantswarm.io/role: control-plane
      - cidrBlocks:
        - cidr: 10.228.0.0/20
          availabilityZone: a
        - cidr: 10.228.16.0/20
          availabilityZone: b
        - cidr: 10.228.32.0/20
          availabilityZone: c
        isPublic: false
        tags:
          subnet.giantswarm.io/role: worker
      - cidrBlocks:
        - cidr: 10.228.96.0/20
          availabilityZone: a
        - cidr: 10.228.112.0/20
          availabilityZone: b
        - cidr: 10.228.128.0/20
          availabilityZone: c
        isPublic: false
        tags:
          subnet.giantswarm.io/role: bastion

This created 3 grouping of subnets each with a specific subnet.giantswarm.io/role AWS tag added to them.

The bastion instance correctly launches into one of the 3 subnets tagged with subnet.giantswarm.io/role: bastion and is accessible using opsctl ssh.

The first control plane node is correctly launched into one of the 3 subnets tagged with subnet.giantswarm.io/role: control-plane but it fails to get to a working state, thus preventing the remaining 2 control plane nodes being created or any of the worker nodes.
The initial control plane node seems to abruptly stop during the cloud-init process with the final line of the syslog (according to the AWS web console) being the following:

         Starting �[0;1;39mInitial cloud-ini� (metadata service crawler)�[0m...

There is no entries in the syslog related to the setup or starting of kubernetes.

Some observations:

• Multiple subnets end up with the same name in AWS as this is generated by CAPA in the following format: ${CLUSTER_NAME}-subnet-[public|private]-${AVAILABILITY_ZONE}
• The apiserver ELB ends up using the first subnets specified in the AWSCluster CR. To ensure the same subnets as the control plane are used these need to be specified first in the list.
• Security groups all look correct
• The transit gateway is associated with all subnets.
• The issue occurs as soon as there are multiple subnets in the same AZ (with the same public/private status) regardless of if those subnets are associated with anything. This was tested by setting all subnetTags to target the control-plane tags.

[AverageMarcus]

Update

With the release of aws-vpc-operator v0.2.2 this PR now allows for the successful creation of private workload clusters with multiple subnet groupings.

TODO

• Update the default to match the default subnet layout of previous version
• Add docs with example subnet layouts (with better IP utilization)
• Include this PR in our fork - AWSMachinePool subnet filters ignored if no matches found kubernetes-sigs/cluster-api-provider-aws#3977
• Test upgrade path

[AverageMarcus]

Fixed in giantswarm/cluster-aws#205 (v0.21.0)

kubectl-gs change to support the new layout when using k-gs template cluster - giantswarm/kubectl-gs#1004