Skip to content

Commit

Permalink
all: force sc wc cluster names
Browse files Browse the repository at this point in the history
  • Loading branch information
anders-elastisys committed Nov 1, 2023
1 parent 150d225 commit adf88fe
Show file tree
Hide file tree
Showing 11 changed files with 39 additions and 37 deletions.
29 changes: 14 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,14 @@ ansible-playbook -e 'ansible_python_interpreter=/usr/bin/python3' --ask-become-p

```bash
export CK8S_CONFIG_PATH=~/.ck8s/my-environment
./bin/ck8s-kubespray init <prefix> <flavor> [<SOPS fingerprint>]
./bin/ck8s-kubespray init <wc|sc> <flavor> [<SOPS fingerprint>]
```

Arguments:
* `prefix` will be used to differentiate this cluster from others in the same CK8S_CONFIG_PATH.
For now you need to set this to `wc` or `sc` if you want to install compliantkubernetes apps on top afterwards, this restriction will be removed later.
* `flavor` will determine some default values for a variety of config options.
- The init command accepts `wc` (*workload cluster*) or `sc` (*service cluster*) as first argument as to create separate folders for each cluster's configuration files.
- `flavor` will determine some default values for a variety of config options.
Supported options are `default`, `gcp`, `aws`, `vsphere`, and `openstack`.
* `SOPS fingerprint` is the gpg fingerprint that will be used for SOPS encryption.
- `SOPS fingerprint` is the gpg fingerprint that will be used for SOPS encryption.
You need to set this or the environment variable `CK8S_PGP_FP` the first time SOPS is used in your specified config path.
1. Edit the `inventory.ini` (found in your config path) to match the VMs (IP addresses and other settings that might be needed for your setup) that should be part of the cluster.
Expand All @@ -55,24 +54,24 @@ ansible-playbook -e 'ansible_python_interpreter=/usr/bin/python3' --ask-become-p
1. Run kubespray to set up the kubernetes cluster:
```bash
./bin/ck8s-kubespray apply <prefix> [<options>]
./bin/ck8s-kubespray apply <wc|sc> [<options>]
```
Any `options` added will be forwarded to ansible.
1. Done.
You should now have a working kubernetes cluster.
You should also have an encrypted kubeconfig at `<CK8S_CONFIG_PATH>/.state/kube_config_<prefix>.yaml` that you can use to access the cluster.
You should also have an encrypted kubeconfig at `<CK8S_CONFIG_PATH>/.state/kube_config_<wc|sc>.yaml` that you can use to access the cluster.
## Changing authorized SSH keys for a cluster
Authorized SSH keys can be changed for a cluster using:
```bash
./bin/ck8s-kubespray apply-ssh <prefix> [<options>]
./bin/ck8s-kubespray apply-ssh <wc|sc> [<options>]
```
It will set the public SSH key(s) found in`<CK8S_CONFIG_PATH>/<prefix>-config/group_vars/all/ck8s-ssh-keys.yaml` as authorized keys in your cluster (just add the keys you want to be authorized as elements in `ck8s_ssh_pub_keys_list`).
It will set the public SSH key(s) found in`<CK8S_CONFIG_PATH>/<wc|sc>-config/group_vars/all/ck8s-ssh-keys.yaml` as authorized keys in your cluster (just add the keys you want to be authorized as elements in `ck8s_ssh_pub_keys_list`).
Note that the authorized SSH keys for the cluster will be set to these keys _exclusively_, removing any keys that may already be authorized, so make sure the list includes **every SSH key** that should be authorized.
When running this command, the SSH keys are applied to each node in the cluster sequentially, in reverse inventory order (first the workers and then the masters).
Expand All @@ -84,7 +83,7 @@ If the connection test fails, you may have lost your SSH access to the node; to
You can reboot all nodes that wants to restart (usually to finish installing new packages) by running:
```bash
./bin/ck8s-kubespray reboot-nodes <prefix> [--extra-vars manual_prompt=true] [<options>]
./bin/ck8s-kubespray reboot-nodes <wc|sc> [--extra-vars manual_prompt=true] [<options>]
```
If you set `--extra-vars manual_prompt=true` then you get a manual prompt before each reboot so you can stop the playbook if you want.
Expand All @@ -96,7 +95,7 @@ Note that this playbook requires you to use ansible version >= 2.10.
You can remove a node from a ck8s cluster by running:
```bash
./bin/ck8s-kubespray remove-node <prefix> <node-name>[,<node-name-2>,...] [<options>]
./bin/ck8s-kubespray remove-node <wc|sc> <node-name>[,<node-name-2>,...] [<options>]
```
### Known issues
Expand All @@ -109,15 +108,15 @@ You can remove a node from a ck8s cluster by running:
With the following command you can run any ansible playbook available in kubespray:
```bash
./bin/ck8s-kubespray run-playbook <prefix> <playbook> [<options>]
./bin/ck8s-kubespray run-playbook <wc|sc> <playbook> [<options>]
```
Where `playbook` is the filename of the playbook that you want to run, e.g. `cluster.yml` if you want to create a cluster (making the command functionally the same as our `ck8s-kubespray apply` command) or `scale.yml` if you want to just add more nodes. Remember to check the kubespray documentation before running a playbook.
This will use the inventory, group-vars, and ssh key in your config path and therefore requires that you first run the init command. Any `options` added will be forwarded to ansible.
## Kubeconfig
We recommend that you use OIDC kubeconfigs instead of regular cluster-admin kubeconfigs. The default settings will create OIDC kubeconfigs for you when you run `./bin/ck8s-kubespray apply <prefix>`, but there are some variables you need to set. See the variables in: `<prefix>-config/group_vars/k8s_cluster/ck8s-k8s-cluster.yaml` in your config path.
We recommend that you use OIDC kubeconfigs instead of regular cluster-admin kubeconfigs. The default settings will create OIDC kubeconfigs for you when you run `./bin/ck8s-kubespray apply <wc|sc>`, but there are some variables you need to set. See the variables in: `<wc|sc>-config/group_vars/k8s_cluster/ck8s-k8s-cluster.yaml` in your config path.
But if you need to use a regular cluster-admin kubeconfig in a break-glass situation, then you can ssh to one of the controleplane nodes and use the kubeconfig at `/etc/kubernetes/admin.conf`. We recommend that you do not copy that kubeconfig to your local host, when dealing with production clusters.
Expand All @@ -126,6 +125,6 @@ For development you can skip OIDC and instead get a regular cluster admin kubeco
The kubeconfig and OIDC cluster admin RBAC are managed with the playbooks `playbooks/kubeconfig.yml` and `playbooks/cluster_admin_rbac.yml`. You can run them manually with:
```bash
./bin/ck8s-kubespray run-playbook <prefix> ../../playbooks/kubeconfig.yml -b
./bin/ck8s-kubespray run-playbook <prefix> ../../playbooks/cluster_admin_rbac.yml -b
./bin/ck8s-kubespray run-playbook <wc|sc> ../playbooks/kubeconfig.yml -b
./bin/ck8s-kubespray run-playbook <wc|sc> ../playbooks/cluster_admin_rbac.yml -b
```
2 changes: 1 addition & 1 deletion bin/apply-ssh.bash
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash

# This script will run an ansible playbook.
# It's not to be executed on its own but rather via `ck8s-kubespray apply-ssh <prefix>`.
# It's not to be executed on its own but rather via `ck8s-kubespray apply-ssh <wc|sc>`.

set -eu -o pipefail

Expand Down
2 changes: 1 addition & 1 deletion bin/apply.bash
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# This script will create a kubernetes cluster using kubespray.
# It will also install some python dependencies for kubespray in a virtual environment
# It's not to be executed on its own but rather via `ck8s-kubespray apply <prefix>`.
# It's not to be executed on its own but rather via `ck8s-kubespray apply <wc|sc>`.

set -eu -o pipefail
shopt -s globstar nullglob dotglob
Expand Down
16 changes: 8 additions & 8 deletions bin/ck8s-kubespray
Original file line number Diff line number Diff line change
Expand Up @@ -10,25 +10,25 @@ here="$(dirname "$(readlink -f "$0")")"
usage() {
echo "COMMANDS:" 1>&2
echo " init initialize the config path" 1>&2
echo " args: <prefix> <flavor> [<SOPS fingerprint>]" 1>&2
echo " args: <wc|sc> <flavor> [<SOPS fingerprint>]" 1>&2
echo " apply runs kubespray to create the cluster" 1>&2
echo " args: <prefix> [<options>]" 1>&2
echo " args: <wc|sc> [<options>]" 1>&2
echo " remove-node removes specified node from cluster" 1>&2
echo " args: <prefix> <node_name> [<options>]" 1>&2
echo " args: <wc|sc> <node_name> [<options>]" 1>&2
echo " run-playbook runs any ansible playbook in kubespray" 1>&2
echo " args: <prefix> <playbook> [<options>]" 1>&2
echo " args: <wc|sc> <playbook> [<options>]" 1>&2
echo " apply-ssh applies SSH keys from a file to a cluster" 1>&2
echo " args: <prefix> [<options>]" 1>&2
echo " args: <wc|sc> [<options>]" 1>&2
echo " reboot-nodes reboots all nodes in a cluster if needed" 1>&2
echo " args: <prefix> [--extra-vars manual_prompt=true] [<options>]" 1>&2
echo " args: <wc|sc> [--extra-vars manual_prompt=true] [<options>]" 1>&2
echo " prune-nerdctl removes unused container resources on all nodes" 1>&2
echo " args: <prefix> [<options>]" 1>&2
echo " args: <wc|sc> [<options>]" 1>&2
echo " upgrade prepares config for upgrade" 1>&2
echo " args: <wc|sc|both> <version> prepare" 1>&2
exit 1
}

if [ $# -lt 2 ]; then
if [ $# -lt 2 ] || [[ ! "${1}" = upgrade ]] && [[ ! "${2}" =~ ^(wc|sc)$ ]] || [[ ! "${2}" =~ ^(wc|sc|both)$ ]]; then
usage
else
export prefix="${2}"
Expand Down
5 changes: 5 additions & 0 deletions bin/common.bash
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@
: "${CK8S_CONFIG_PATH:?Missing CK8S_CONFIG_PATH}"
: "${prefix:?Missing prefix}"

if [[ ! "${prefix}" =~ ^(wc|sc|both)$ ]]; then
echo "ERROR: invalid value set for \"prefix\", valid values are <wc|sc|both>" 1>&2
exit 1
fi

# Check for this mistake https://github.com/koalaman/shellcheck/wiki/SC2088
# shellcheck disable=SC2088
if [[ "${CK8S_CONFIG_PATH:0:2}" == "~/" ]]; then
Expand Down
2 changes: 1 addition & 1 deletion bin/init.bash
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# This script takes care of initializing a CK8S configuration path for kubespray.
# It writes the default configuration files to the config path and generates
# some defaults where applicable.
# It's not to be executed on its own but rather via `ck8s-kubespray init ...`.
# It's not to be executed on its own but rather via `ck8s-kubespray init <wc|sc>``.

set -eu -o pipefail
shopt -s globstar nullglob dotglob
Expand Down
2 changes: 1 addition & 1 deletion bin/prune-nerdctl.bash
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash

# This script will run an ansible playbook.
# It's not to be executed on its own but rather via `ck8s-kubespray prune-nerdctl <prefix>`.
# It's not to be executed on its own but rather via `ck8s-kubespray prune-nerdctl <wc|sc>`.

set -eu -o pipefail

Expand Down
2 changes: 1 addition & 1 deletion bin/reboot-nodes.bash
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash

# This script will run an ansible playbook.
# It's not to be executed on its own but rather via `ck8s-kubespray reboot-nodes <prefix>`.
# It's not to be executed on its own but rather via `ck8s-kubespray reboot-nodes <wc|sc>`.
# Add the variable "manual_prompt = true" for a manual prompt.
# The script may fail and in such situations rerun the script.

Expand Down
2 changes: 1 addition & 1 deletion bin/remove-node.bash
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# This script will run the remove-node.yml playbook.
# It will also install some python dependencies for kubespray in a virtual environment
# It's not to be executed on its own but rather via `ck8s-kubespray remove-node <prefix> <node_name>` [<options>].
# It's not to be executed on its own but rather via `ck8s-kubespray remove-node <wc|sc> <node_name>` [<options>].

set -eu -o pipefail
shopt -s globstar nullglob dotglob
Expand Down
2 changes: 1 addition & 1 deletion bin/run-playbook.bash
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# This script will run an ansible playbook available in kubespray.
# It will also install some python dependencies for kubespray in a virtual environment
# It's not to be executed on its own but rather via `ck8s-kubespray run-playbook <prefix> <playbook>`.
# It's not to be executed on its own but rather via `ck8s-kubespray run-playbook <wc|sc> <playbook>`.

set -eu -o pipefail
shopt -s globstar nullglob dotglob
Expand Down
12 changes: 5 additions & 7 deletions docs/citycloud-snippets.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ To set up a cluster on citycloud these snippets can be used
Start by setting up some environments for this setup

```bash
SERVICE_CLUSTER="my-new-sc-cluster"
WORKLOAD_CLUSTERS=( "my-new-wc-cluster" )
PUB_SSH_KEY_FILE="${HOME}/.ssh/id_rsa.pub"
```

Expand Down Expand Up @@ -41,7 +39,7 @@ Set up the clusters into respective folders

```bash
pushd kubespray
for CLUSTER in "${SERVICE_CLUSTER}" "${WORKLOAD_CLUSTERS[@]}"; do
for CLUSTER in sc wc; do
mkdir -p "${CK8S_CONFIG_PATH}/${CLUSTER}-config/group_vars"
ln -s "${CK8S_CONFIG_PATH}/${CLUSTER}-config/" "inventory/$CLUSTER"
# shellcheck disable=SC2016
Expand Down Expand Up @@ -73,7 +71,7 @@ k8s_allowed_remote_ips = ["1.2.3.4/32"]
Now you're ready to deploy the infrastructure

```bash
for CLUSTER in "${SERVICE_CLUSTER}" "${WORKLOAD_CLUSTERS[@]}"; do
for CLUSTER in sc wc; do
pushd "kubespray/inventory/${CLUSTER}"
terraform init ../../contrib/terraform/openstack
terraform apply \
Expand All @@ -94,7 +92,7 @@ To set up kubernetes with compliantkubernetes-kubespray you can follow these ste
Initialize the configuration with.

```bash
for CLUSTER in "${SERVICE_CLUSTER}" "${WORKLOAD_CLUSTERS[@]}"; do
for CLUSTER in sc wc; do
./bin/ck8s-kubespray init "${CLUSTER}" openstack ~/.ssh/id_rsa
ln -s "$(pwd)/kubespray/inventory/${CLUSTER}/tfstate-${CLUSTER}.tfstate" "${CK8S_CONFIG_PATH}/${CLUSTER}-config/" || true
cp "kubespray/contrib/terraform/openstack/hosts" "${CK8S_CONFIG_PATH}/${CLUSTER}-config/inventory.ini"
Expand All @@ -107,7 +105,7 @@ Check the variables in the `group_vars` folder for each cluster and make sure th
Apply the configuration and set up kubernetes.

```bash
for CLUSTER in "${SERVICE_CLUSTER}" "${WORKLOAD_CLUSTERS[@]}"; do
for CLUSTER in sc wc; do
bin/ck8s-kubespray apply "${CLUSTER}"
done
```
Expand All @@ -125,7 +123,7 @@ Later when you want to destroy the infrastructure.
Then run the following:

```bash
for CLUSTER in "${SERVICE_CLUSTER}" "${WORKLOAD_CLUSTERS[@]}"; do
for CLUSTER in sc wc; do
pushd "kubespray/inventory/${CLUSTER}"
terraform destroy \
-var-file cluster.tfvars \
Expand Down

0 comments on commit adf88fe

Please sign in to comment.