Skip to content

Latest commit

 

History

History
 
 

3-gke-dev

README.md

GKE Multitenant

This stage allows creation and management of a fleet of GKE multitenant clusters for a single environment, optionally leveraging GKE Hub to configure additional features.

The following diagram illustrates the high-level design of created resources, which can be adapted to specific requirements via variables:

GKE multitenant

Design overview and choices

The general idea behind this stage is to deploy a single project hosting multiple clusters leveraging several useful GKE features like Config Sync, which lend themselves well to a multitenant approach to GKE.

Some high level choices applied here:

How to run this stage

This stage is meant to be executed after the FAST "foundational" stages: bootstrap, resource management, security and networking stages.

It's of course possible to run this stage in isolation, refer to the Running in isolation section below for details.

Before running this stage, you need to make sure you have the correct credentials and permissions, and localize variables by assigning values that match your configuration.

Provider and Terraform variables

As all other FAST stages, the mechanism used to pass variable values and pre-built provider files from one stage to the next is also leveraged here.

The commands to link or copy the provider and terraform variable files can be easily derived from the fast-links.sh script in the FAST root folder, passing it a single argument with the local output files folder (if configured) or the GCS output bucket in the automation project (derived from stage 0 outputs). The following examples demonstrate both cases, and the resulting commands that then need to be copy/pasted and run.

../fast-links.sh ~/fast-config

# File linking commands for GKE (dev) stage

# provider file
ln -s ~/fast-config/providers/3-gke-dev-providers.tf ./

# input files from other stages
ln -s ~/fast-config/tfvars/0-globals.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/0-bootstrap.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/1-resman.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/2-networking.auto.tfvars.json ./

# conventional place for stage tfvars (manually created)
ln -s ~/fast-config/3-gke-dev.auto.tfvars ./
../fast-links.sh gs://xxx-prod-iac-core-outputs-0

# File linking commands for GKE (dev) stage

# provider file
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/providers/3-gke-dev-providers.tf ./

# input files from other stages
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-globals.auto.tfvars.json ./
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/0-bootstrap.auto.tfvars.json ./
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/1-resman.auto.tfvars.json ./
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/tfvars/2-networking.auto.tfvars.json ./

# conventional place for stage tfvars (manually created)
gcloud storage cp gs://xxx-prod-iac-core-outputs-0/3-gke-dev.auto.tfvars ./

Impersonating the automation service account

The preconfigured provider file uses impersonation to run with this stage's automation service account's credentials. The gcp-devops and organization-admins groups have the necessary IAM bindings in place to do that, so make sure the current user is a member of one of those groups.

Variable configuration

Variables in this stage -- like most other FAST stages -- are broadly divided into three separate sets:

  • variables which refer to global values for the whole organization (org id, billing account id, prefix, etc.), which are pre-populated via the 0-globals.auto.tfvars.json file linked or copied above
  • variables which refer to resources managed by previous stage, which are prepopulated here via the *.auto.tfvars.json files linked or copied above
  • and finally variables that optionally control this stage's behaviour and customizations, and can to be set in a custom terraform.tfvars file

The latter set is explained in the Customization sections below, and the full list can be found in the Variables table at the bottom of this document.

Running the stage

Once provider and variable values are in place and the correct user is configured, the stage can be run:

terraform init
terraform apply

Customizations

This stage is designed with multi-tenancy in mind, and the expectation is that GKE clusters will mostly share a common set of defaults. Variables allow management of clusters, nodepools, and fleet registration and configurations.

Clusters and node pools

This is an example of declaring a private cluster with one nodepool via tfvars file:

clusters = {
  test-00 = {
    description = "Cluster test 0"
    location    = "europe-west8"
    private_cluster_config = {
      enable_private_endpoint = true
      master_global_access    = true
    }
    vpc_config = {
      subnetwork             = "projects/ldj-dev-net-spoke-0/regions/europe-west8/subnetworks/gke"
      master_ipv4_cidr_block = "172.16.20.0/28"
      master_authorized_ranges = {
        private = "10.0.0.0/8"
      }
    }
  }
}
nodepools = {
  test-00 = {
    00 = {
      node_count = { initial = 1 }
    }
  }
}
# tftest skip

If clusters share similar configurations, those can be centralized via locals blocks in this stage's main.tf file, and merged in with clusters via a simple for_each loop.

Fleet management

Fleet management is entirely optional, and uses two separate variables:

  • fleet_config: specifies the GKE fleet features to activate
  • fleet_configmanagement_templates: defines configuration templates for specific sets of features (Config Management currently)

Clusters can then be configured for fleet registration and one of the config management templates attached via the cluster-level fleet_config attribute.

Files

name description modules
gke-clusters.tf GKE clusters. gke-cluster-standard · gke-nodepool
gke-hub.tf GKE hub configuration. gke-hub
main.tf Project and usage dataset. bigquery-dataset · iam-service-account · project
outputs.tf Module outputs.
variables-fast.tf None
variables-fleet.tf GKE fleet configurations.
variables.tf Module variables.

Variables

name description type required default producer
billing_account Billing account id. If billing account is not part of the same org set is_org_level to false. object({…}) ✓ 0-bootstrap
environments Long environment names. object({…}) ✓ 1-resman
prefix Prefix used for resources that need unique names. Use a maximum of 9 chars for organizations, and 11 chars for tenants. string ✓ 0-bootstrap
clusters Clusters configuration. Refer to the gke-cluster module for type details. map(object({…})) {}
deletion_protection Prevent Terraform from destroying data resources. bool false
fleet_config Fleet configuration. object({…}) null
fleet_configmanagement_templates Sets of fleet configurations that can be applied to member clusters, in config name => {options} format. map(object({…})) {}
folder_ids Folder name => id mappings. map(string) {} 1-resman
host_project_ids Shared VPC host project name => id mappings. map(string) {} 2-networking
iam Project-level authoritative IAM bindings for users and service accounts in {ROLE => [MEMBERS]} format. map(list(string)) {}
iam_by_principals Authoritative IAM binding in {PRINCIPAL => [ROLES]} format. Principals need to be statically defined to avoid cycle errors. Merged internally with the iam variable. map(list(string)) {}
nodepools Nodepools configuration. Refer to the gke-nodepool module for type details. map(map(object({…}))) {}
stage_config FAST stage configuration used to find resource ids. Must match name defined for the stage in resource management. object({…}) {…}
subnet_self_links Subnet VPC name => { name => self link } mappings. map(map(string)) {} 2-networking
vpc_config VPC-level configuration for project and clusters. object({…}) {…}
vpc_self_links Shared VPC name => self link mappings. map(string) {} 2-networking

Outputs

name description sensitive consumers
cluster_ids Cluster ids.
clusters Cluster resources. ✓
project_id GKE project id.