Terraform Setup to Test Cilium on Talos

A Terraform module to manage a Talos-based Kubernetes on AWS (EC2 instances). Is uses the official Talos Terraform provider in the background. We mainly followed the provided siderolabs/contrib example.

Supported Features

• Install Talos Linux OS EC2 VMs
  • For now, it's only supported to deploy the VMs in public subnets with public IPs assigned
• Support for single- and multi-node cluster architectures
• Bootstrap Talos Kubernetes cluster with some infrastructure components:
  • Talos' KubePrism to get an internal endpoint for the KAPI (used for Cilium Kube-Proxy replacement)
  • kubernetes-sigs/metrics-server
  • alex1989hu/kubelet-serving-cert-approver inspired by Talos' Deploying Metrics Server guide.
• Cilium features:
  • Kube-Proxy replacement
  • IPAM modes: kubernetes, cluster-pool

Example Usage

// Create a Talos Linux cluster
module "talos" {
  source = "git::https://github.com/isovalent/terraform-aws-talos?ref=<RELEASE_TAG>"

  // Supported Talos versions (and therefore K8s versions) can be found here: https://github.com/siderolabs/talos/releases
  talos_version      = "v1.5.3"
  kubernetes_version = "1.27.3"
  cluster_name       = "talos-cute"
  region             = "eu-west-1"
  tags               = local.tags
  // VPC needs to be created in advance via https://github.com/isovalent/terraform-aws-vpc
  vpc_id             = module.vpc.id
  pod_cidr           = "100.64.0.0/14"
  service_cidr       = "100.68.0.0/16"
}

Terraform Module Doc

Requirements

Name	Version
terraform	>= 1.4.0
aws	~> 5.0
random	~> 3.5
talos	0.6.1

Providers

Name	Version
aws	~> 5.0
local	n/a
null	n/a
random	~> 3.5
talos	0.6.1

Modules

Name	Source	Version
cluster_sg	terraform-aws-modules/security-group/aws	~> 5.1
elb_k8s_elb	terraform-aws-modules/elb/aws	~> 4.0
kubernetes_api_sg	terraform-aws-modules/security-group/aws//modules/https-443	~> 5.1
talos_control_plane_nodes	terraform-aws-modules/ec2-instance/aws	~> 5.5
talos_worker_group	terraform-aws-modules/ec2-instance/aws	~> 5.5

Resources

Name	Type
local_file.kubeconfig	resource
local_file.talosconfig	resource
null_resource.wait_for_public_subnets	resource
random_string.workspace_id	resource
talos_cluster_kubeconfig.this	resource
talos_machine_bootstrap.this	resource
talos_machine_configuration_apply.controlplane	resource
talos_machine_configuration_apply.worker_group	resource
talos_machine_secrets.this	resource
aws_ami.talos	data source
aws_region.current	data source
aws_subnets.public	data source
aws_vpc.vpc	data source
talos_client_configuration.this	data source
talos_machine_configuration.controlplane	data source
talos_machine_configuration.worker_group	data source

Inputs

Name	Description	Type	Default	Required
allocate_node_cidrs	Whether to assign PodCIDRs to Node resources or not. Only needed in case Cilium runs in 'kubernetes' IPAM mode.	bool	true	no
allow_workload_on_cp_nodes	Allow workloads on CP nodes or not. Allowing it means Talos Linux default taints are removed from CP nodes. More details here: https://www.talos.dev/v1.5/talos-guides/howto/workers-on-controlplane/	bool	false	no
cluster_architecture	Cluster architecture. Choose 'arm64' or 'amd64'. If you choose 'arm64', ensure to also override the control_plane.instance_type and worker_groups.instance_type with an ARM64-based instance type like 'm7g.large'.	string	"amd64"	no
cluster_id	The ID of the cluster.	number	"1"	no
cluster_name	Name of cluster	string	n/a	yes
config_patch_files	Path to talos config path files that applies to all nodes	list(string)	[]	no
control_plane	Info for control plane that will be created	object({ instance_type = optional(string, "m5.large") config_patch_files = optional(list(string), []) tags = optional(map(string), {}) })	{}	no
controlplane_count	Defines how many controlplane nodes are deployed in the cluster.	number	3	no
disable_kube_proxy	Whether to deploy Kube-Proxy or not. By default, KP shouldn't be deployed.	bool	true	no
kubernetes_api_allowed_cidr	The CIDR from which to allow to access the Kubernetes API	string	"0.0.0.0/0"	no
kubernetes_version	Kubernetes version to use for the Talos cluster, if not set, the K8s version shipped with the selected Talos version will be used. Check https://www.talos.dev/latest/introduction/support-matrix/. For example '1.29.3'.	string	""	no
pod_cidr	The CIDR to use for Pods. Only required in case allocate_node_cidrs is set to 'true'. Otherwise, simply configure it inside Cilium's Helm values.	string	"100.64.0.0/14"	no
region	The region in which to create the Talos Linux cluster.	string	n/a	yes
service_cidr	The CIDR to use for services.	string	"100.68.0.0/16"	no
tags	The set of tags to place on the cluster.	map(string)	n/a	yes
talos_api_allowed_cidr	The CIDR from which to allow to access the Talos API	string	"0.0.0.0/0"	no
talos_version	Talos version to use for the cluster, if not set, the newest Talos version. Check https://github.com/siderolabs/talos/releases for available releases.	string	"v1.8.0"	no
vpc_cidr	The IPv4 CIDR block for the VPC.	string	"10.0.0.0/16"	no
vpc_id	ID of the VPC where to place the VMs.	string	n/a	yes
worker_groups	List of node worker node groups to create	list(object({ name = string instance_type = optional(string, "m5.large") config_patch_files = optional(list(string), []) tags = optional(map(string), {}) }))	[ { "name": "default" } ]	no
workers_count	Defines how many worker nodes are deployed in the cluster.	number	2	no

Outputs

Name	Description
cluster_name	Name of cluster
elb_dns_name	Public ELB DNS name.
kubeconfig	Kubeconfig content
path_to_kubeconfig_file	The generated kubeconfig.
path_to_talosconfig_file	The generated talosconfig.