Skip to content

Commit

Permalink
October 2024 revamp
Browse files Browse the repository at this point in the history
- Replaced Bird by native MetalLB
- Setup Ingress IP properly
- Deploy an extra Ingress to handle the Global IP (instead of abusing the regular Ingress that makes unable to expose cluster-wide apps and only global ones)
- Added Rancher deployment (Rancher flavor and version can be provided)
- K3s or RKE2 (any version can be specified)
- Added a helper script to perform the Rancher bootstrap process automatically as well as importing all the clusters to it
- Added `shfmt` to the pre-commit action
- Switched to Terraform >= 1.9.0 (as required to include a new validation)
- Fixed pre-commit action -> pre-commit/action#210

Closes equinix-labs#53, closes equinix-labs#81, closes equinix-labs#67, closes equinix-labs#66
  • Loading branch information
e-minguez committed Oct 3, 2024
1 parent 0be4cdf commit 3be137a
Show file tree
Hide file tree
Showing 20 changed files with 1,315 additions and 579 deletions.
7 changes: 6 additions & 1 deletion .github/workflows/pre-commit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest]
tf: [1.3.0]
tf: [1.9.0]
tflint: [v0.44.1]
permissions:
pull-requests: write
Expand All @@ -31,6 +31,8 @@ jobs:

- name: Install Python3
uses: actions/setup-python@v5
with:
python-version: '3.x'

- name: Install tflint
uses: terraform-linters/setup-tflint@v4
Expand Down Expand Up @@ -69,4 +71,7 @@ jobs:
platform: linux
arch: amd64

- name: Install shfmt
uses: mfinelli/setup-shfmt@v3

- uses: pre-commit/[email protected]
356 changes: 259 additions & 97 deletions README.md

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions examples/demo_cluster/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# SiDemo Cluster Example
# Demo Cluster Examples

This example demonstrates usage of the Equinix Metal K3s module. A Demo application is installed.
This example demonstrates usage of the Equinix Metal K3s/RKE2 module. A Demo application is installed.

## Usage

Expand Down Expand Up @@ -36,15 +36,15 @@ No resources.

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_metal_auth_token"></a> [metal\_auth\_token](#input\_metal\_auth\_token) | Your Equinix Metal API key | `string` | n/a | yes |
| <a name="input_metal_project_id"></a> [metal\_project\_id](#input\_metal\_project\_id) | Your Equinix Metal Project ID | `string` | n/a | yes |
| <a name="input_clusters"></a> [clusters](#input\_clusters) | K3s cluster definition | <pre>list(object({<br> name = optional(string, "K3s demo cluster")<br> metro = optional(string, "FR")<br> plan_control_plane = optional(string, "c3.small.x86")<br> plan_node = optional(string, "c3.small.x86")<br> node_count = optional(number, 0)<br> k3s_ha = optional(bool, false)<br> os = optional(string, "debian_11")<br> control_plane_hostnames = optional(string, "k3s-cp")<br> node_hostnames = optional(string, "k3s-node")<br> custom_k3s_token = optional(string, "")<br> ip_pool_count = optional(number, 0)<br> k3s_version = optional(string, "")<br> metallb_version = optional(string, "")<br> }))</pre> | <pre>[<br> {}<br>]</pre> | no |
| <a name="input_clusters"></a> [clusters](#input\_clusters) | Cluster definition | <pre>list(object({<br/> name = optional(string, "Demo cluster")<br/> metro = optional(string, "FR")<br/> plan_control_plane = optional(string, "c3.small.x86")<br/> plan_node = optional(string, "c3.small.x86")<br/> node_count = optional(number, 0)<br/> ha = optional(bool, false)<br/> os = optional(string, "debian_11")<br/> control_plane_hostnames = optional(string, "cp")<br/> node_hostnames = optional(string, "node")<br/> custom_token = optional(string, "")<br/> ip_pool_count = optional(number, 0)<br/> kube_version = optional(string, "")<br/> metallb_version = optional(string, "")<br/> rancher_version = optional(string, "")<br/> rancher_flavor = optional(string, "")<br/> custom_rancher_password = optional(string, "")<br/> }))</pre> | <pre>[<br/> {}<br/>]</pre> | no |
| <a name="input_deploy_demo"></a> [deploy\_demo](#input\_deploy\_demo) | Deploys a simple demo using a global IP as ingress and a hello-kubernetes pods | `bool` | `false` | no |
| <a name="input_global_ip"></a> [global\_ip](#input\_global\_ip) | Enables a global anycast IPv4 that will be shared for all clusters in all metros | `bool` | `false` | no |
| <a name="input_metal_auth_token"></a> [metal\_auth\_token](#input\_metal\_auth\_token) | Your Equinix Metal API key | `string` | n/a | yes |
| <a name="input_metal_project_id"></a> [metal\_project\_id](#input\_metal\_project\_id) | Your Equinix Metal Project ID | `string` | n/a | yes |

### Outputs

| Name | Description |
|------|-------------|
| <a name="output_demo_cluster"></a> [demo\_cluster](#output\_demo\_cluster) | Passthrough of the root module output |
| <a name="output_clusters_output"></a> [clusters\_output](#output\_clusters\_output) | Passthrough of the root module output |
<!-- END_TF_DOCS -->
118 changes: 118 additions & 0 deletions examples/demo_cluster/clusters-to-rancher.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
#!/usr/bin/env bash
set -euo pipefail

usage() {
echo "Usage: $0 -p <password>"
exit 1
}

die() {
echo ${1} 1>&2
exit ${2}
}

prechecks() {
command -v kubectl >/dev/null 2>&1 || die "Error: kubectl not found" 1
command -v curl >/dev/null 2>&1 || die "Error: curl not found" 1
command -v jq >/dev/null 2>&1 || die "Error: jq not found" 1
command -v scp >/dev/null 2>&1 || die "Error: scp not found" 1
}

wait_for_rancher() {
while ! curl -k "${RANCHERURL}/ping" >/dev/null 2>&1; do sleep 1; done
}

bootstrap_rancher() {
# Get token
TOKEN=$(curl -sk -X POST ${RANCHERURL}/v3-public/localProviders/local?action=login -H 'content-type: application/json' -d "{\"username\":\"admin\",\"password\":\"${RANCHERPASS}\"}" | jq -r .token)

# Set password
curl -q -sk ${RANCHERURL}/v3/users?action=changepassword -H 'content-type: application/json' -H "Authorization: Bearer ${TOKEN}" -d "{\"currentPassword\":\"${RANCHERPASS}\",\"newPassword\":\"${PASSWORD}\"}"

# Create a temporary API token (ttl=60 minutes)
APITOKEN=$(curl -sk ${RANCHERURL}/v3/token -H 'content-type: application/json' -H "Authorization: Bearer ${TOKEN}" -d '{"type":"token","description":"automation","ttl":3600000}' | jq -r .token)

# Set the Rancher URL
curl -q -sk ${RANCHERURL}/v3/settings/server-url -H 'content-type: application/json' -H "Authorization: Bearer ${APITOKEN}" -X PUT -d "{\"name\":\"server-url\",\"value\":\"${RANCHERURL}\"}"
}

get_cluster_kubeconfig() {
cluster="${1}"
FIRSTHOST=$(echo ${OUTPUT} | jq -r "first(.clusters_output.value.cluster_details[\"${cluster}\"].nodes[].node_public_ipv4)")
API=$(echo ${OUTPUT} | jq -r ".clusters_output.value.cluster_details[\"${cluster}\"].api")
KUBECONFIG="$(mktemp)"
export KUBECONFIG
scp -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@${FIRSTHOST}:/root/.kube/config ${KUBECONFIG}
# Linux
[ "$(uname -o)" == "GNU/Linux" ] && sed -i "s/127.0.0.1/${API}/g" ${KUBECONFIG}
# OSX
[ "$(uname -o)" == "Darwin" ] && sed -i "" "s/127.0.0.1/${API}/g" ${KUBECONFIG}
chmod 600 ${KUBECONFIG}
echo ${KUBECONFIG}
}

clusters_to_rancher() {
RANCHERKUBE=$(get_cluster_kubeconfig "${RANCHERCLUSTER}")

IFS=$'\n'
for clustername in ${OTHERCLUSTERS}; do
export KUBECONFIG=${RANCHERKUBE}
normalizedname=$(echo ${clustername} | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9 ]/-/g' | sed 's/ /-/g' | sed 's/^-*\|-*$/''/g')
cat <<-EOF | kubectl apply -f - >/dev/null 2>&1
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: ${normalizedname}
namespace: fleet-default
spec: {}
EOF
MANIFEST="$(kubectl get clusterregistrationtokens.management.cattle.io -n "$(kubectl get clusters.provisioning.cattle.io -n fleet-default "${normalizedname}" -o jsonpath='{.status.clusterName}')" default-token -o jsonpath='{.status.manifestUrl}')"
DESTKUBECONFIG=$(get_cluster_kubeconfig "${clustername}")
curl --insecure -sfL ${MANIFEST} | kubectl --kubeconfig ${DESTKUBECONFIG} apply -f - >/dev/null 2>&1
rm -f "${DESTKUBECONFIG}"
done

rm -f "${RANCHERKUBE}"
}

PASSWORD=""
while getopts ":p:" opt; do
case $opt in
p)
PASSWORD=$OPTARG
;;
\?)
echo "Invalid option: -$OPTARG" >&2
usage
;;
:)
echo "Option -$OPTARG requires an argument." >&2
usage
;;
esac
done

if [ -z "$PASSWORD" ]; then
echo "Error: Password is required." 1>&2
usage
fi

if [ ${#PASSWORD} -lt 12 ]; then
die "Error: Password must be at least 12 characters long." 1
fi

[ ! -f "./terraform.tfstate" ] && die "Error: ./terraform.tfstate does not exist." 1

OUTPUT=$(terraform output -json)

[ "${OUTPUT}" == "{}" ] && die "Error. terraform output is '{}'." 1

RANCHERCLUSTER=$(echo ${OUTPUT} | jq -r 'first(.clusters_output.value.rancher_urls | keys[])')
RANCHERURL=$(echo ${OUTPUT} | jq -r ".clusters_output.value.rancher_urls[\"${RANCHERCLUSTER}\"].rancher_url")
RANCHERPASS=$(echo ${OUTPUT} | jq -r ".clusters_output.value.rancher_urls[\"${RANCHERCLUSTER}\"].rancher_initial_password_base64" | base64 -d)
OTHERCLUSTERS=$(echo ${OUTPUT} | jq -r ".clusters_output.value.cluster_details | keys[] | select(. != \"${RANCHERCLUSTER}\")")

prechecks
wait_for_rancher
bootstrap_rancher
clusters_to_rancher
2 changes: 1 addition & 1 deletion examples/demo_cluster/outputs.tf
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
output "demo_cluster" {
output "clusters_output" {
description = "Passthrough of the root module output"
value = module.demo
}
23 changes: 19 additions & 4 deletions examples/demo_cluster/terraform.tfvars.example
Original file line number Diff line number Diff line change
@@ -1,11 +1,26 @@
metal_auth_token="your_token_here" #This must be a user API token
metal_project_id="your_project_id"
clusters = [
clusters = [
{
name = "Your cluster name"
name = "FR DEV Cluster"
rancher_flavor = "stable"
ip_pool_count = 1
kube_version = "v1.29.9+k3s1"
},
{
name = "Your cluster name"
metro = "SV"
name = "SV DEV Cluster"
metro = "SV"
node_count = 1
kube_version = "v1.30.3+rke2r1"
},
{
name = "SV Production"
ip_pool_count = 4
ha = true
metro = "SV"
node_count = 3
}
]

global_ip = true
deploy_demo = true
17 changes: 10 additions & 7 deletions examples/demo_cluster/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -22,21 +22,24 @@ variable "deploy_demo" {
}

variable "clusters" {
description = "K3s cluster definition"
description = "Cluster definition"
type = list(object({
name = optional(string, "K3s demo cluster")
name = optional(string, "Demo cluster")
metro = optional(string, "FR")
plan_control_plane = optional(string, "c3.small.x86")
plan_node = optional(string, "c3.small.x86")
node_count = optional(number, 0)
k3s_ha = optional(bool, false)
ha = optional(bool, false)
os = optional(string, "debian_11")
control_plane_hostnames = optional(string, "k3s-cp")
node_hostnames = optional(string, "k3s-node")
custom_k3s_token = optional(string, "")
control_plane_hostnames = optional(string, "cp")
node_hostnames = optional(string, "node")
custom_token = optional(string, "")
ip_pool_count = optional(number, 0)
k3s_version = optional(string, "")
kube_version = optional(string, "")
metallb_version = optional(string, "")
rancher_version = optional(string, "")
rancher_flavor = optional(string, "")
custom_rancher_password = optional(string, "")
}))
default = [{}]
}
17 changes: 9 additions & 8 deletions main.tf
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
locals {
global_ip_cidr = var.global_ip ? equinix_metal_reserved_ip_block.global_ip[0].cidr_notation : ""
# tflint-ignore: terraform_unused_declarations
validate_demo = (var.deploy_demo == true && var.global_ip == false) ? tobool("Demo is only deployed if global_ip = true.") : true
}

################################################################################
# K3S Cluster In-line Module
# K8s Cluster In-line Module
################################################################################

module "k3s_cluster" {
source = "./modules/k3s_cluster"
module "kube_cluster" {
source = "./modules/kube_cluster"

for_each = { for cluster in var.clusters : cluster.name => cluster }

Expand All @@ -18,14 +16,17 @@ module "k3s_cluster" {
plan_control_plane = each.value.plan_control_plane
plan_node = each.value.plan_node
node_count = each.value.node_count
k3s_ha = each.value.k3s_ha
ha = each.value.ha
os = each.value.os
control_plane_hostnames = each.value.control_plane_hostnames
node_hostnames = each.value.node_hostnames
custom_k3s_token = each.value.custom_k3s_token
k3s_version = each.value.k3s_version
custom_token = each.value.custom_token
kube_version = each.value.kube_version
metallb_version = each.value.metallb_version
ip_pool_count = each.value.ip_pool_count
rancher_flavor = each.value.rancher_flavor
rancher_version = each.value.rancher_version
custom_rancher_password = each.value.custom_rancher_password
metal_project_id = var.metal_project_id
deploy_demo = var.deploy_demo
global_ip_cidr = local.global_ip_cidr
Expand Down
4 changes: 0 additions & 4 deletions modules/k3s_cluster/outputs.tf

This file was deleted.

Loading

0 comments on commit 3be137a

Please sign in to comment.