diff --git a/README.md b/README.md index d79f47afd..b80232f4f 100644 --- a/README.md +++ b/README.md @@ -16,13 +16,14 @@ Kubemarine is an open source, lightweight and powerful management tool built for - [upgrade](documentation/Maintenance.md#upgrade-procedure) - [backup](documentation/Maintenance.md#backup-procedure) - [restore](documentation/Maintenance.md#restore-procedure) + - [reconfigure](documentation/Maintenance.md#reconfigure-procedure) - [check_iaas](documentation/Kubecheck.md#iaas-procedure) - [check_paas](documentation/Kubecheck.md#paas-procedure) - [migrate_kubemarine](documentation/Maintenance.md#kubemarine-migration-procedure) - [manage_psp](documentation/Maintenance.md#manage-psp-procedure) - [manage_pss](documentation/Maintenance.md#manage-pss-procedure) - [cert_renew](documentation/Maintenance.md#certificate-renew-procedure) - - [migrate_cri](documentation/Maintenance.md#migration-cri-procedure) + - [migrate_cri](documentation/Maintenance.md#cri-migration-procedure) - [Single cluster inventory](documentation/Installation.md#configuration) for all operations, highly customizable - Default values of all parameters in configurations with a minimum of required parameters - [Control planes balancing](documentation/Installation.md#full-ha-scheme) with external balancers and VRRP diff --git a/documentation/Installation.md b/documentation/Installation.md index cc4a35341..20b15726d 100644 --- a/documentation/Installation.md +++ b/documentation/Installation.md @@ -505,7 +505,7 @@ For more information about the structure of the inventory and how to specify the * [Minimal All-in-one Inventory Example](../examples/cluster.yaml/allinone-cluster.yaml) - It provides the minimum set of parameters for deploying All-in-one scheme. * [Minimal Mini-HA Inventory Example](../examples/cluster.yaml/miniha-cluster.yaml) - It provides the minimum set of parameters for deploying Mini-HA scheme. -#### Inventory validation +#### Inventory validation When configuring the inventory, you can use your favorite IDE supporting YAML validation by JSON schema. JSON schema for inventory file can be used by [URL](../kubemarine/resources/schemas/cluster.json?raw=1). @@ -1094,23 +1094,25 @@ In the `services.kubeadm` section, you can override the original settings for ku For more information about these settings, refer to the official Kubernetes documentation at [https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file). By default, the installer uses the following parameters: -|Parameter| Default Value | -|---|----------------------------------------------------------| -|kubernetesVersion| `v1.26.11` | -|controlPlaneEndpoint| `{{ cluster_name }}:6443` | -|networking.podSubnet| `10.128.0.0/14` for IPv4 or `fd02::/48` for IPv6 | -|networking.serviceSubnet| `172.30.0.0/16` for IPv4 or `fd03::/112` for IPv6 | -|apiServer.certSANs| List with all nodes internal IPs, external IPs and names | -|apiServer.extraArgs.enable-admission-plugins| `NodeRestriction` | -|apiServer.extraArgs.profiling| `false` | -|apiServer.extraArgs.audit-log-path| `/var/log/kubernetes/audit/audit.log` | -|apiServer.extraArgs.audit-policy-file| `/etc/kubernetes/audit-policy.yaml` | -|apiServer.extraArgs.audit-log-maxage| `30` | -|apiServer.extraArgs.audit-log-maxbackup| `10` | -|apiServer.extraArgs.audit-log-maxsize| `100` | -|scheduler.extraArgs.profiling| `false` | -|controllerManager.extraArgs.profiling| `false` | -|controllerManager.extraArgs.terminated-pod-gc-threshold| `1000` | +| Parameter | Default Value | Description | +|-------------------------------------------------------|----------------------------------------------------------|--------------------------------------------------------------------------------------------------| +|kubernetesVersion | `v1.26.11` | | +|controlPlaneEndpoint | `{{ cluster_name }}:6443` | | +|networking.podSubnet | `10.128.0.0/14` for IPv4 or `fd02::/48` for IPv6 | | +|networking.serviceSubnet | `172.30.0.0/16` for IPv4 or `fd03::/112` for IPv6 | | +|apiServer.certSANs | List with all nodes internal IPs, external IPs and names | Custom SANs are only appended to, but do not override the default list | +|apiServer.extraArgs.enable-admission-plugins | `NodeRestriction` | `PodSecurityPolicy` plugin is added if [Admission psp](#admission-psp) is enabled | +|apiServer.extraArgs.feature-gates | | `PodSecurity=true` is added for Kubernetes < v1.28 if [Admission pss](#admission-pss) is enabled | +|apiServer.extraArgs.admission-control-config-file | `/etc/kubernetes/pki/admission.yaml` | Provided default value **overrides** custom value if [Admission pss](#admission-pss) is enabled. | +|apiServer.extraArgs.profiling | `false` | | +|apiServer.extraArgs.audit-log-path | `/var/log/kubernetes/audit/audit.log` | | +|apiServer.extraArgs.audit-policy-file | `/etc/kubernetes/audit-policy.yaml` | | +|apiServer.extraArgs.audit-log-maxage | `30` | | +|apiServer.extraArgs.audit-log-maxbackup | `10` | | +|apiServer.extraArgs.audit-log-maxsize | `100` | | +|scheduler.extraArgs.profiling | `false` | | +|controllerManager.extraArgs.profiling | `false` | | +|controllerManager.extraArgs.terminated-pod-gc-threshold| `1000` | | The following is an example of kubeadm defaults override: @@ -1147,10 +1149,7 @@ services: **Note**: Those parameters remain in manifests files after Kubernetes upgrade. That is the proper way to preserve custom settings for system services. -**Warning**: These kubeadm parameters are configurable only during installation, currently. -Kubemarine currently do not provide special procedure to change these parameters after installation. -To reconfigure the parameters manually, refer to the official documentation at [https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure). -For more information about generic approach of the cluster maintenance, refer to [Maintenance Basics](Maintenance.md#basics). +**Note**: These kubeadm parameters can be reconfigured after installation using [Reconfigure Procedure](Maintenance.md#reconfigure-procedure). During init, join, upgrade procedures kubeadm runs `preflight` procedure to do some preliminary checks. In case of any error kubeadm stops working. Sometimes it is necessary to ignore some preflight errors to deploy or upgrade successfully. @@ -1510,6 +1509,8 @@ By default, the installer uses the following parameters: `serializeImagePulls` parameter defines whether the images will be pulled in parallel (false) or one at a time. +**Note**: Some of the parameters can be reconfigured after installation using [Reconfigure Procedure](Maintenance.md#reconfigure-procedure). + **Warning**: If you want to change the values of variables `podPidsLimit` and `maxPods`, you have to update the value of the `pid_max` (this value should not less than result of next expression: `maxPods * podPidsLimit + 2048`), which can be done using task `prepare.system.sysctl`. To get more info about `pid_max` you can go to [sysctl](#sysctl) section. The following is an example of kubeadm defaults override: @@ -1544,6 +1545,8 @@ By default, the installer uses the following parameters: `conntrack.min` inherits the `services.sysctl.net.netfilter.nf_conntrack_max` value from [sysctl](#sysctl). +**Note**: These parameters can be reconfigured after installation using [Reconfigure Procedure](Maintenance.md#reconfigure-procedure). + #### kubeadm_patches *Installation task*: `deploy.kubernetes` @@ -1599,7 +1602,11 @@ services: By default Kubemarine sets `bind-address` parameter of `kube-apiserver` to `node.internal_address` via patches at every control-plane node. -**Note**: If a parameter of control-plane pods is defined in `kubeadm..extraArgs` or is set by default by kubeadm and then redefined in `kubeadm.paches`, the pod manifest file will contain the same flag twice and the running pod will take into account the last mentioned value (taken from `kubeadm.patches`). This behaviour persists at the moment: https://github.com/kubernetes/kubeadm/issues/1601. +**Note**: These parameters can be reconfigured after installation using [Reconfigure Procedure](Maintenance.md#reconfigure-procedure). + +**Note**: If a parameter of control-plane pods is defined in `kubeadm..extraArgs` or is set by default by kubeadm and then redefined in `services.kubeadm_patches`, +the pod manifest file will contain the same flag twice and the running pod will take into account the last mentioned value (taken from `services.kubeadm_patches`). +This behaviour persists at the moment: https://github.com/kubernetes/kubeadm/issues/1601. #### kernel_security @@ -5650,6 +5657,13 @@ plugins: Application of the list merge strategy is allowed in the following sections: * `plugins.installation.procedures` * `services.kubeadm.apiServer.extraVolumes` +* `services.kubeadm.controllerManager.extraVolumes` +* `services.kubeadm.scheduler.extraVolumes` +* `services.kubeadm_patches.apiServer` +* `services.kubeadm_patches.controllerManager` +* `services.kubeadm_patches.etcd` +* `services.kubeadm_patches.kubelet` +* `services.kubeadm_patches.scheduler` * `services.kernel_security.permissive` * `services.modprobe` * `services.etc_hosts` diff --git a/documentation/Kubecheck.md b/documentation/Kubecheck.md index 2d86b883d..2d8f203f5 100644 --- a/documentation/Kubecheck.md +++ b/documentation/Kubecheck.md @@ -41,6 +41,8 @@ This section provides information about the Kubecheck functionality. - [201 Kubelet Status](#201-kubelet-status) - [202 Nodes pid_max](#202-nodes-pid_max) - [203 Kubelet Version](#203-kubelet-version) + - [233 Kubelet Configuration](#233-kubelet-configuration) + - [234 kube-proxy Configuration](#234-kube-proxy-configuration) - [205 System Packages Versions](#205-system-packages-version) - [205 CRI Versions](#205-cri-versions) - [205 HAproxy Version](#205-haproxy-version) @@ -388,8 +390,11 @@ The task tree is as follows: * configuration * kubelet * status - * configuration + * pid_max * version + * configuration + kube-proxy: + * configuration * packages * system * recommended_versions @@ -462,7 +467,7 @@ This test checks the status of the Kubelet service on all hosts in the cluster w ##### 202 Nodes pid_max -*Task*: `services.kubelet.configuration` +*Task*: `services.kubelet.pid_max` This test checks that kubelet `maxPods` and `podPidsLimit` are correctly aligned with kernel `pid_max`. @@ -472,6 +477,19 @@ This test checks that kubelet `maxPods` and `podPidsLimit` are correctly aligned This test checks the Kubelet version on all hosts in a cluster. +##### 233 Kubelet Configuration + +*Task*: `services.kubelet.configuration` + +This test checks the consistency of the /var/lib/kubelet/config.yaml configuration +with `kubelet-config` ConfigMap and with the inventory. + +##### 234 kube-proxy Configuration + +*Task*: `services.kube-proxy.configuration` + +This test checks the consistency of the `kube-proxy` ConfigMap with the inventory. + ##### 204 Container Runtime Configuration Check *Task*: `services.container_runtime.configuration` @@ -647,13 +665,15 @@ This test verifies ETCD health. *Task*: `control_plane.configuration_status` -This test verifies the consistency of the configuration (image version, `extra_args`, `extra_volumes`) of static pods of Control Plain like `kube-apiserver`, `kube-controller-manager` and `kube-scheduler`. +This test verifies the consistency of the configuration of static pods of Control Plain +for `kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, and `etcd`. ##### 221 Control Plane Health Status *Task*: `control_plane.health_status` -This test verifies the health of static pods `kube-apiserver`, `kube-controller-manager` and `kube-scheduler`. +This test verifies the health of static pods `kube-apiserver`, `kube-controller-manager`, +`kube-scheduler`, and `etcd`. ##### 222 Default Services Configuration Status diff --git a/documentation/Maintenance.md b/documentation/Maintenance.md index 299b16b3f..aa052319e 100644 --- a/documentation/Maintenance.md +++ b/documentation/Maintenance.md @@ -11,6 +11,7 @@ This section describes the features and steps for performing maintenance procedu - [Add Node Procedure](#add-node-procedure) - [Operating System Migration](#operating-system-migration) - [Remove Node Procedure](#remove-node-procedure) + - [Reconfigure Procedure](#reconfigure-procedure) - [Manage PSP Procedure](#manage-psp-procedure) - [Manage PSS Procedure](#manage-pss-procedure) - [Reboot Procedure](#reboot-procedure) @@ -970,6 +971,173 @@ To change the operating system on an already running cluster: **Warning**: In case when you use custom associations, you need to specify them simultaneously for all types of operating systems. For more information, refer to the [associations](Installation.md#associations) section in the _Kubemarine Installation Procedure_. +## Reconfigure Procedure + +This procedure is aimed to reconfigure the cluster. + +It is supposed to reconfigure the cluster as a generalized concept described by the inventory file. +Though, currently the procedure supports to reconfigure only Kubeadm-managed settings. +If you are looking for how to reconfigure other settings, consider the following: + +- Probably some other [maintenance procedure](#provided-procedures) can do the task. +- Some [installation tasks](Installation.md#tasks-list-redefinition) can reconfigure some system settings without full redeploy of the cluster. + +**Basic prerequisites**: + +- Make sure to follow the [Basics](#basics). +- Before starting the procedure, consider making a backup. For more information, see the section [Backup Procedure](#backup-procedure). + +### Reconfigure Procedure Parameters + +The procedure accepts required positional argument with the path to the procedure inventory file. +You can find description and examples of the accepted parameters in the next sections. + +The JSON schema for procedure inventory is available by [URL](../kubemarine/resources/schemas/reconfigure.json?raw=1). +For more information, see [Validation by JSON Schemas](Installation.md#inventory-validation). + +#### Reconfigure Kubeadm + +The following Kubeadm-managed sections can be reconfigured: + +- `services.kubeadm.apiServer` +- `services.kubeadm.apiServer.certSANs` +- `services.kubeadm.scheduler` +- `services.kubeadm.controllerManager` +- `services.kubeadm.etcd.local.extraArgs` +- `services.kubeadm_kubelet` +- `services.kubeadm_kube-proxy` +- `services.kubeadm_patches` + +For more information, refer to the description of these sections: + +- [kubeadm](Installation.md#kubeadm) +- [kubeadm_kubelet](Installation.md#kubeadm_kubelet) +- [kubeadm_kube-proxy](Installation.md#kubeadm_kube-proxy) +- [kubeadm_patches](Installation.md#kubeadm_patches) + +Example of procedure inventory that reconfigures all the supported sections: + +
+ Click to expand + +```yaml +services: + kubeadm: + apiServer: + certSANs: + - k8s-lb + extraArgs: + enable-admission-plugins: NodeRestriction,PodNodeSelector + profiling: "false" + audit-log-path: /var/log/kubernetes/audit/audit.log + audit-policy-file: /etc/kubernetes/audit-policy.yaml + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "100" + scheduler: + extraArgs: + profiling: "false" + controllerManager: + extraArgs: + profiling: "false" + terminated-pod-gc-threshold: "1000" + etcd: + local: + extraArgs: + heartbeat-interval: "1000" + election-timeout: "10000" + kubeadm_kubelet: + protectKernelDefaults: true + kubeadm_kube-proxy: + conntrack: + min: 1000000 + kubeadm_patches: + apiServer: + - groups: [control-plane] + patch: + max-requests-inflight: 500 + - nodes: [master-3] + patch: + max-requests-inflight: 600 + etcd: + - nodes: [master-1] + patch: + snapshot-count: 110001 + - nodes: [master-2] + patch: + snapshot-count: 120001 + - nodes: [master-3] + patch: + snapshot-count: 130001 + controllerManager: + - groups: [control-plane] + patch: + authorization-webhook-cache-authorized-ttl: 30s + scheduler: + - nodes: [master-2,master-3] + patch: + profiling: true + kubelet: + - nodes: [worker5] + patch: + maxPods: 100 + - nodes: [worker6] + patch: + maxPods: 200 +``` + +
+ +The above configuration is merged with the corresponding sections in the main `cluster.yaml`, +and the related Kubernetes components are reconfigured based on the resulting inventory. + +In this way it is not possible to delete some property, +allowing the corresponding Kubernetes component to fall back to the default behaviour. +This can be worked around by manual changing of the `cluster.yaml` +and running the `reconfigure` procedure with **empty** necessary section. +For example, you can delete `services.kubeadm.etcd.local.extraArgs.election-timeout` from `cluster.yaml` +and then run the procedure with the following procedure inventory: + +```yaml +services: + kubeadm: + etcd: {} +``` + +**Note**: It is not possible to delete default parameters offered by Kubemarine. + +**Note**: The mentioned hint to delete custom properties is not enough for `services.kubeadm_kube-proxy` due to existing restrictions of Kubeadm CLI tool. +One should additionally edit the `kube-proxy` ConfigMap and set the value that is considered the default. + +**Note**: Passing of empty `services.kubeadm.apiServer` section reconfigures the `kube-apiserver`, +but does not write new certificate. +To **additionally** write new certificate, pass the desirable extra SANs in `services.kubeadm.apiServer.certSANs`. + +**Restrictions**: + +- Very few options of `services.kubeadm_kubelet` section can be reconfigured currently. + To learn exact set of options, refer to the JSON schema. +- Some properties cannot be fully redefined. + For example, this relates to some settings in `services.kubeadm.apiServer`. + For details, refer to the description of the corresponding sections in the installation guide. + +**Basic flow**: + +If the procedure affects the particular set of Kubernetes components, all the components are reconfigured on each relevant node one by one. +The flow proceeds to the next nodes only after the affected components are considered up and ready on the reconfigured node. +Control plane nodes are reconfigured first. + +Working `kube-apiserver` is not required to reconfigure control plane components (more specifically, to change their static manifests), +but required to reconfigure kubelet and kube-proxy. + +### Reconfigure Procedure Tasks Tree + +The `reconfigure` procedure executes the following sequence of tasks: + +- deploy + - kubernetes + - reconfigure + ## Manage PSP Procedure The manage PSP procedure allows you to change PSP configuration on an already installed cluster. Using this procedure, you can: @@ -1032,12 +1200,10 @@ To avoid this, you need to specify custom policy and bind it using `ClusterRoleB The `manage_psp` procedure executes the following sequence of tasks: -1. check_inventory 1. delete_custom 2. add_custom -3. reconfigure_oob -4. reconfigure_plugin -5. restart_pods +3. reconfigure_psp +4. restart_pods ## Manage PSS Procedure @@ -1113,10 +1279,8 @@ application is stateless or stateful. Also shouldn't use `restart-pod: true` opt The `manage_pss procedure executes the following sequence of tasks: -1. check_inventory -2. delete_default_pss -3. apply_default_pss -4. restart_pods +1. manage_pss +2. restart_pods ## Reboot Procedure diff --git a/kubemarine.spec b/kubemarine.spec index 27a0560ac..8bfc50023 100644 --- a/kubemarine.spec +++ b/kubemarine.spec @@ -18,6 +18,7 @@ a = Analysis(['./kubemarine/__main__.py'], 'kubemarine.procedures.migrate_cri', 'kubemarine.procedures.manage_psp', 'kubemarine.procedures.manage_pss', + 'kubemarine.procedures.reconfigure', 'kubemarine.procedures.remove_node', 'kubemarine.procedures.upgrade', 'kubemarine.procedures.cert_renew', diff --git a/kubemarine/__main__.py b/kubemarine/__main__.py index fc6fa09ba..04ef0cd65 100755 --- a/kubemarine/__main__.py +++ b/kubemarine/__main__.py @@ -83,6 +83,10 @@ 'description': "Remove existing nodes from cluster", 'group': 'maintenance' }, + 'reconfigure': { + 'description': "Reconfigure managed Kubernetes cluster", + 'group': 'maintenance' + }, 'manage_psp': { 'description': "Manage PSP on Kubernetes cluster", 'group': 'maintenance' diff --git a/kubemarine/admission.py b/kubemarine/admission.py index f784dd881..8598c8ca1 100644 --- a/kubemarine/admission.py +++ b/kubemarine/admission.py @@ -27,6 +27,7 @@ from kubemarine.core.cluster import KubernetesCluster from kubemarine.core.group import NodeGroup, RunnersGroupResult from kubemarine.core.yaml_merger import default_merger +from kubemarine.kubernetes import components from kubemarine.plugins import builtin privileged_policy_filename = "privileged.yaml" @@ -48,17 +49,17 @@ loaded_oob_policies = {} +ERROR_INCONSISTENT_INVENTORIES = "Procedure config and cluster config are inconsistent. Please check 'admission' option" + # TODO: When KubeMarine is not support Kubernetes version lower than 1.25, the PSP implementation code should be deleted def support_pss_only(cluster: KubernetesCluster) -> bool: - kubernetes_version = cluster.inventory["services"]["kubeadm"]["kubernetesVersion"] - return utils.version_key(kubernetes_version)[0:2] >= utils.minor_version_key("v1.25") + return components.kubernetes_minor_release_at_least(cluster.inventory, "v1.25") def is_pod_security_unconditional(cluster: KubernetesCluster) -> bool: - kubernetes_version = cluster.inventory["services"]["kubeadm"]["kubernetesVersion"] - return utils.version_key(kubernetes_version)[0:2] >= utils.minor_version_key("v1.28") + return components.kubernetes_minor_release_at_least(cluster.inventory, "v1.28") def enrich_inventory_psp(inventory: dict, _: KubernetesCluster) -> dict: @@ -141,6 +142,9 @@ def manage_psp_enrichment(inventory: dict, cluster: KubernetesCluster) -> dict: if final_security_state == "disabled" and procedure_config.get("oob-policies"): raise Exception("OOB policies can not be configured when security is disabled") + cluster.context['initial_pod_security'] = current_security_state + current_config["pod-security"] = final_security_state + return inventory @@ -286,7 +290,19 @@ def add_custom_task(cluster: KubernetesCluster) -> None: manage_scope=cluster.procedure_inventory["psp"]["add-policies"]) -def reconfigure_oob_task(cluster: KubernetesCluster) -> None: +def reconfigure_psp_task(cluster: KubernetesCluster) -> None: + if is_security_enabled(cluster.inventory): + reconfigure_oob_policies(cluster) + reconfigure_plugin(cluster) + else: + # If target state is disabled, firstly remove PodSecurityPolicy admission plugin thus disabling the security. + # It is necessary because while security on one control-plane is disabled, + # it is still enabled on the other control-planes, and the policies should still work. + reconfigure_plugin(cluster) + reconfigure_oob_policies(cluster) + + +def reconfigure_oob_policies(cluster: KubernetesCluster) -> None: target_security_state = cluster.procedure_inventory["psp"].get("pod-security") oob_policies = cluster.procedure_inventory["psp"].get("oob-policies") @@ -316,21 +332,14 @@ def reconfigure_oob_task(cluster: KubernetesCluster) -> None: first_control_plane.call(manage_policies, manage_type="apply", manage_scope=resolve_oob_scope(policies_to_recreate, "all")) -def reconfigure_plugin_task(cluster: KubernetesCluster) -> None: +def reconfigure_plugin(cluster: KubernetesCluster) -> None: target_state = cluster.procedure_inventory["psp"].get("pod-security") if not target_state: cluster.log.debug("Security plugin will not be reconfigured") return - first_control_plane = cluster.nodes["control-plane"].get_first_member() - - cluster.log.debug("Updating kubeadm config map") - final_admission_plugins_list = first_control_plane.call(update_kubeadm_configmap, target_state=target_state) - - # update api-server config on all control-planes - cluster.log.debug("Updating kube-apiserver configs on control-planes") - cluster.nodes["control-plane"].call(update_kubeapi_config, options_list=final_admission_plugins_list) + cluster.nodes['control-plane'].call(components.reconfigure_components, components=['kube-apiserver']) def restart_pods_task(cluster: KubernetesCluster) -> None: @@ -357,71 +366,7 @@ def restart_pods_task(cluster: KubernetesCluster) -> None: first_control_plane.sudo("kubectl rollout restart ds %s -n %s" % (ds["metadata"]["name"], ds["metadata"]["namespace"])) # we do not know to wait for, only for system pods maybe - cluster.log.debug("Waiting for system pods...") - kubernetes.wait_for_any_pods(cluster, first_control_plane) - - -def update_kubeadm_configmap_psp(first_control_plane: NodeGroup, target_state: str) -> str: - # load kubeadm config map and retrieve cluster config - kubeadm_cm = kubernetes.KubernetesObject(first_control_plane.cluster, 'ConfigMap', 'kubeadm-config', 'kube-system') - kubeadm_cm.reload(first_control_plane) - cluster_config = yaml.safe_load(kubeadm_cm.obj["data"]["ClusterConfiguration"]) - - # resolve resulting admission plugins list - final_plugins_string = resolve_final_plugins_list(cluster_config, target_state) - - # update kubeadm config map with updated plugins list - cluster_config["apiServer"]["extraArgs"]["enable-admission-plugins"] = final_plugins_string - kubeadm_cm.obj["data"]["ClusterConfiguration"] = yaml.dump(cluster_config) - - # apply updated kubeadm config map - kubeadm_cm.apply(first_control_plane) - - return final_plugins_string - - -def update_kubeadm_configmap(first_control_plane: NodeGroup, target_state: str) -> str: - admission_impl = first_control_plane.cluster.inventory['rbac']['admission'] - if admission_impl == "psp": - return update_kubeadm_configmap_psp(first_control_plane, target_state) - else: # admission_impl == "pss": - return update_kubeadm_configmap_pss(first_control_plane, target_state) - - -def update_kubeapi_config_psp(control_planes: NodeGroup, plugins_list: str) -> None: - yaml = ruamel.yaml.YAML() - - for control_plane in control_planes.get_ordered_members_list(): - result = control_plane.sudo("cat /etc/kubernetes/manifests/kube-apiserver.yaml") - - # update kube-apiserver config with updated plugins list - conf = yaml.load(list(result.values())[0].stdout) - new_command = [cmd for cmd in conf["spec"]["containers"][0]["command"] if "enable-admission-plugins" not in cmd] - new_command.append("--enable-admission-plugins=%s" % plugins_list) - conf["spec"]["containers"][0]["command"] = new_command - - # place updated config on control-plane - buf = io.StringIO() - yaml.dump(conf, buf) - control_plane.put(buf, "/etc/kubernetes/manifests/kube-apiserver.yaml", sudo=True) - - # force kube-apiserver pod restart, then wait for api to become available - if control_planes.cluster.inventory['services']['cri']['containerRuntime'] == 'containerd': - control_plane.call(utils.wait_command_successful, - command="crictl rm -f $(sudo crictl ps --name kube-apiserver -q)") - else: - control_plane.call(utils.wait_command_successful, - command="docker stop $(sudo docker ps -q -f 'name=k8s_kube-apiserver'" - " | awk '{print $1}')") - control_plane.call(utils.wait_command_successful, command="kubectl get pod -n kube-system") - - -def update_kubeapi_config(control_planes: NodeGroup, options_list: str) -> None: - admission_impl = control_planes.cluster.inventory['rbac']['admission'] - if admission_impl == "psp": - update_kubeapi_config_psp(control_planes, options_list) - elif admission_impl == "pss": - update_kubeapi_config_pss(control_planes, options_list) + kube_nodes.call(components.wait_for_pods) def is_security_enabled(inventory: dict) -> bool: @@ -445,32 +390,9 @@ def delete_privileged_policy(group: NodeGroup) -> RunnersGroupResult: def apply_admission(group: NodeGroup) -> None: admission_impl = group.cluster.inventory['rbac']['admission'] - if is_security_enabled(group.cluster.inventory): - if admission_impl == "psp": - group.cluster.log.debug("Setting up privileged psp...") - apply_privileged_policy(group) - elif admission_impl == "pss": - group.cluster.log.debug("Setting up default pss...") - apply_default_pss(group.cluster) - - -def apply_default_pss(cluster: KubernetesCluster) -> None: - if cluster.context.get('initial_procedure') == 'manage_pss': - procedure_config = cluster.procedure_inventory["pss"] - current_config = cluster.inventory["rbac"]["pss"] - if procedure_config["pod-security"] == "enabled" and current_config["pod-security"] == "enabled": - manage_pss(cluster, "apply") - elif procedure_config["pod-security"] == "enabled" and current_config["pod-security"] == "disabled": - manage_pss(cluster, "install") - else: - manage_pss(cluster, "init") - - -def delete_default_pss(cluster: KubernetesCluster) -> None: - procedure_config = cluster.procedure_inventory["pss"] - current_config = cluster.inventory["rbac"]["pss"] - if procedure_config["pod-security"] == "disabled" and current_config["pod-security"] == "enabled": - return manage_pss(cluster, "delete") + if admission_impl == "psp" and is_security_enabled(group.cluster.inventory): + group.cluster.log.debug("Setting up privileged psp...") + apply_privileged_policy(group) def manage_privileged_from_file(group: NodeGroup, filename: str, manage_type: str) -> RunnersGroupResult: @@ -553,27 +475,6 @@ def collect_policies_template(psp_list: Optional[List[dict]], return buf.getvalue() -def resolve_final_plugins_list(cluster_config: dict, target_state: str) -> str: - if "enable-admission-plugins" not in cluster_config["apiServer"]["extraArgs"]: - if target_state == "enabled": - return "PodSecurityPolicy" - else: - return "" - else: - current_plugins = cluster_config["apiServer"]["extraArgs"]["enable-admission-plugins"] - if "PodSecurityPolicy" not in current_plugins: - if target_state == "enabled": - resulting_list = "%s,%s" % (current_plugins, "PodSecurityPolicy") - else: - resulting_list = current_plugins - elif target_state == "disabled": - resulting_list = current_plugins.replace("PodSecurityPolicy", "") - else: - resulting_list = current_plugins - - return resulting_list.replace(",,", ",").strip(",") - - def install(cluster: KubernetesCluster) -> None: admission_impl = cluster.inventory['rbac']['admission'] if admission_impl == "psp": @@ -586,14 +487,14 @@ def manage_pss_enrichment(inventory: dict, cluster: KubernetesCluster) -> dict: procedure_config = cluster.procedure_inventory["pss"] kubernetes_version = cluster.inventory["services"]["kubeadm"]["kubernetesVersion"] - minor_version_key = utils.version_key(kubernetes_version)[0:2] - + if not is_security_enabled(inventory) and procedure_config["pod-security"] == "disabled": raise Exception("both 'pod-security' in procedure config and current config are 'disabled'. There is nothing to change") # check flags, profiles; enrich inventory - if minor_version_key < utils.minor_version_key("v1.23"): - raise Exception("PSS is not supported properly in Kubernetes version before v1.23") + cluster.context['initial_pod_security'] = inventory["rbac"]["pss"]["pod-security"] + inventory["rbac"]["pss"]["pod-security"] = procedure_config["pod-security"] + if "defaults" in procedure_config: for item in procedure_config["defaults"]: if item.endswith("version"): @@ -625,6 +526,7 @@ def enrich_default_admission(inventory: dict, cluster: KubernetesCluster) -> dic def manage_enrichment(inventory: dict, cluster: KubernetesCluster) -> dict: + check_inventory(cluster) admission_impl = inventory['rbac']['admission'] if admission_impl == "psp": return manage_psp_enrichment(inventory, cluster) @@ -634,165 +536,30 @@ def manage_enrichment(inventory: dict, cluster: KubernetesCluster) -> dict: return inventory -def manage_pss(cluster: KubernetesCluster, manage_type: str) -> None: - first_control_plane = cluster.nodes["control-plane"].get_first_member() +def manage_pss(cluster: KubernetesCluster) -> None: control_planes = cluster.nodes["control-plane"] - # 'apply' - change options in admission.yaml, PSS is enabled - if manage_type == "apply": - # set labels for predifined plugins namespaces and namespaces defined in procedure config - label_namespace_pss(cluster, manage_type) - # copy admission config on control-planes - copy_pss(control_planes) - for control_plane in control_planes.get_ordered_members_list(): - # force kube-apiserver pod restart, then wait for api to become available - if control_plane.cluster.inventory['services']['cri']['containerRuntime'] == 'containerd': - control_plane.call(utils.wait_command_successful, command="crictl rm -f " - "$(sudo crictl ps --name kube-apiserver -q)") - else: - control_plane.call(utils.wait_command_successful, command="docker stop " - "$(sudo docker ps -f 'name=k8s_kube-apiserver'" - " | awk '{print $1}')") - control_plane.call(utils.wait_command_successful, command="kubectl get pod -n kube-system") - # 'install' - enable PSS - elif manage_type == "install": - # set labels for predifined plugins namespaces and namespaces defined in procedure config - label_namespace_pss(cluster, manage_type) - # copy admission config on control-planes - copy_pss(cluster.nodes["control-plane"]) - - cluster.log.debug("Updating kubeadm config map") - final_features_list = first_control_plane.call(update_kubeadm_configmap_pss, target_state="enabled") - - # update api-server config on all control-planes - cluster.log.debug("Updating kube-apiserver configs on control-planes") - cluster.nodes["control-plane"].call(update_kubeapi_config_pss, features_list=final_features_list) - # 'init' make changes during init Kubernetes cluster - elif manage_type == "init": - cluster.log.debug("Updating kubeadm config map") - first_control_plane.call(update_kubeadm_configmap_pss, target_state="enabled") - # 'delete' - disable PSS - elif manage_type == "delete": - # set labels for predifined plugins namespaces and namespaces defined in procedure config - label_namespace_pss(cluster, manage_type) - - final_features_list = first_control_plane.call(update_kubeadm_configmap, target_state="disabled") - - # update api-server config on all control-planes - cluster.log.debug("Updating kube-apiserver configs on control-planes") - cluster.nodes["control-plane"].call(update_kubeapi_config_pss, features_list=final_features_list) - - # erase PSS admission config - cluster.log.debug("Erase admission configuration... %s" % admission_path) - group = cluster.nodes["control-plane"] - group.sudo("rm -f %s" % admission_path, warn=True) + target_state = cluster.inventory["rbac"]["pss"]["pod-security"] -def update_kubeapi_config_pss(control_planes: NodeGroup, features_list: str) -> None: - yaml = ruamel.yaml.YAML() + # set labels for predefined plugins namespaces and namespaces defined in procedure config + label_namespace_pss(cluster) - for control_plane in control_planes.get_ordered_members_list(): - result = control_plane.sudo("cat /etc/kubernetes/manifests/kube-apiserver.yaml") - # update kube-apiserver config with updated features list or delete '--feature-gates' and '--admission-control-config-file' - conf = yaml.load(list(result.values())[0].stdout) - new_command = [cmd for cmd in conf["spec"]["containers"][0]["command"]] - if len(features_list) != 0: - if not is_pod_security_unconditional(control_plane.cluster): - if 'PodSecurity=true' in features_list: - new_command.append("--admission-control-config-file=%s" % admission_path) - else: - new_command.append("--admission-control-config-file=''") - new_command.append("--feature-gates=%s" % features_list) - else: - new_command.append("--admission-control-config-file=%s" % admission_path) - if control_plane.cluster.context['initial_procedure'] == 'upgrade': - if any(argument in "--feature-gates=PodSecurity=true" for argument in new_command): - new_command.remove("--feature-gates=PodSecurity=true") - else: - for item in conf["spec"]["containers"][0]["command"]: - if item.startswith("--"): - key = item.split('=')[0] - value = item[len(key)+1:] - if key in ["--feature-gates", "--admission-control-config-file"]: - del_option = "%s=%s" % (key, value) - new_command.remove(del_option) - - conf["spec"]["containers"][0]["command"] = new_command - - # place updated config on control-plane - buf = io.StringIO() - yaml.dump(conf, buf) - control_plane.put(buf, "/etc/kubernetes/manifests/kube-apiserver.yaml", sudo=True) - - # force kube-apiserver pod restart, then wait for api to become available - if control_plane.cluster.inventory['services']['cri']['containerRuntime'] == 'containerd': - control_plane.call(utils.wait_command_successful, command="crictl rm -f " - "$(sudo crictl ps --name kube-apiserver -q)") - else: - control_plane.call(utils.wait_command_successful, command="docker stop " - "$(sudo docker ps -f 'name=k8s_kube-apiserver'" - " | awk '{print $1}')") - control_plane.call(utils.wait_command_successful, command="kubectl get pod -n kube-system") - - -def update_kubeadm_configmap_pss(first_control_plane: NodeGroup, target_state: str) -> str: - cluster: KubernetesCluster = first_control_plane.cluster - - final_feature_list = "" - - # load kubeadm config map and retrieve cluster config - kubeadm_cm = kubernetes.KubernetesObject(cluster, 'ConfigMap', 'kubeadm-config', 'kube-system') - kubeadm_cm.reload(first_control_plane) - cluster_config = yaml.safe_load(kubeadm_cm.obj["data"]["ClusterConfiguration"]) - - # update kubeadm config map with feature list - if target_state == "enabled": - if not is_pod_security_unconditional(cluster): - if "feature-gates" in cluster_config["apiServer"]["extraArgs"]: - enabled_admissions = cluster_config["apiServer"]["extraArgs"]["feature-gates"] - if 'PodSecurity=true' not in enabled_admissions: - enabled_admissions = "%s,PodSecurity=true" % enabled_admissions - cluster_config["apiServer"]["extraArgs"]["feature-gates"] = enabled_admissions - cluster_config["apiServer"]["extraArgs"]["admission-control-config-file"] = admission_path - final_feature_list = enabled_admissions - else: - cluster_config["apiServer"]["extraArgs"]["admission-control-config-file"] = admission_path - final_feature_list = enabled_admissions - else: - cluster_config["apiServer"]["extraArgs"]["feature-gates"] = "PodSecurity=true" - cluster_config["apiServer"]["extraArgs"]["admission-control-config-file"] = admission_path - final_feature_list = "PodSecurity=true" - else: - cluster_config["apiServer"]["extraArgs"]["admission-control-config-file"] = admission_path - if cluster.context['initial_procedure'] == 'upgrade': - if cluster_config["apiServer"]["extraArgs"].get("feature-gates"): - del cluster_config["apiServer"]["extraArgs"]["feature-gates"] - final_feature_list = "PodSecurity deprecated in %s" % cluster_config['kubernetesVersion'] - elif target_state == "disabled": - if not is_pod_security_unconditional(cluster): - feature_list = cluster_config["apiServer"]["extraArgs"]["feature-gates"].replace("PodSecurity=true", "") - final_feature_list = feature_list.replace(",,", ",") - if len(final_feature_list) == 0: - del cluster_config["apiServer"]["extraArgs"]["feature-gates"] - del cluster_config["apiServer"]["extraArgs"]["admission-control-config-file"] - else: - cluster_config["apiServer"]["extraArgs"]["feature-gates"] = final_feature_list - del cluster_config["apiServer"]["extraArgs"]["admission-control-config-file"] - else: - if cluster_config["apiServer"]["extraArgs"].get("feature-gates"): - if len(cluster_config["apiServer"]["extraArgs"]["feature-gates"]) == 0: - del cluster_config["apiServer"]["extraArgs"]["feature-gates"] - del cluster_config["apiServer"]["extraArgs"]["admission-control-config-file"] - else: - del cluster_config["apiServer"]["extraArgs"]["admission-control-config-file"] - else: - del cluster_config["apiServer"]["extraArgs"]["admission-control-config-file"] + # copy admission config on control-planes + copy_pss(control_planes) - kubeadm_cm.obj["data"]["ClusterConfiguration"] = yaml.dump(cluster_config) + # Admission configuration may change. + # Force kube-apiserver pod restart, then wait for API server to become available. + force_restart = True - # apply updated kubeadm config map - kubeadm_cm.apply(first_control_plane) + # Extra args of API may change, need to reconfigure the API server. + # See enrich_inventory_pss() + control_planes.call(kubernetes.components.reconfigure_components, + components=['kube-apiserver'], force_restart=force_restart) - return final_feature_list + if target_state == 'disabled': + # erase PSS admission config + cluster.log.debug("Erase admission configuration... %s" % admission_path) + control_planes.sudo("rm -f %s" % admission_path, warn=True) def finalize_inventory(cluster: KubernetesCluster, inventory_to_finalize: dict, @@ -827,12 +594,6 @@ def finalize_inventory_pss(cluster: KubernetesCluster, inventory_to_finalize: di # update PSP/PSS fields in the inventory dumped to cluster_finalized.yaml def update_finalized_inventory(cluster: KubernetesCluster, inventory_to_finalize: dict) -> dict: - if cluster.context.get('initial_procedure') == 'manage_pss': - current_config = inventory_to_finalize.setdefault("rbac", {}).setdefault("pss", {}) - current_config["pod-security"] = cluster.procedure_inventory["pss"].get("pod-security", current_config.get("pod-security", "enabled")) - elif cluster.context.get('initial_procedure') == 'manage_psp': - current_config = inventory_to_finalize.setdefault("rbac", {}).setdefault("psp", {}) - current_config["pod-security"] = cluster.procedure_inventory["psp"].get("pod-security", current_config.get("pod-security", "enabled")) # remove PSP section from cluster_finalyzed.yaml if support_pss_only(cluster): del inventory_to_finalize["rbac"]["psp"] @@ -840,24 +601,23 @@ def update_finalized_inventory(cluster: KubernetesCluster, inventory_to_finalize return inventory_to_finalize +def generate_pss(cluster: KubernetesCluster) -> str: + defaults = cluster.inventory["rbac"]["pss"]["defaults"] + exemptions = cluster.inventory["rbac"]["pss"]["exemptions"] + return Template(utils.read_internal(admission_template))\ + .render(defaults=defaults, exemptions=exemptions) + + def copy_pss(group: NodeGroup) -> Optional[RunnersGroupResult]: if group.cluster.inventory['rbac']['admission'] != "pss": return None - if group.cluster.context.get('initial_procedure') == 'manage_pss': - if not is_security_enabled(group.cluster.inventory) and \ - group.cluster.procedure_inventory["pss"]["pod-security"] != "enabled": - group.cluster.log.debug("Pod security disabled, skipping pod admission installation...") - return None - if group.cluster.context.get('initial_procedure') == 'install': - if not is_security_enabled(group.cluster.inventory): - group.cluster.log.debug("Pod security disabled, skipping pod admission installation...") - return None - - defaults = group.cluster.inventory["rbac"]["pss"]["defaults"] - exemptions = group.cluster.inventory["rbac"]["pss"]["exemptions"] + + if not is_security_enabled(group.cluster.inventory): + group.cluster.log.debug("Pod security disabled, skipping pod admission installation...") + return None + # create admission config from template and cluster.yaml - admission_config = Template(utils.read_internal(admission_template))\ - .render(defaults=defaults,exemptions=exemptions) + admission_config = generate_pss(group.cluster) # put admission config on every control-planes group.cluster.log.debug(f"Copy admission config to {admission_path}") @@ -881,16 +641,17 @@ def get_labels_to_ensure_profile(inventory: dict, profile: str) -> Dict[str, str return {} -def label_namespace_pss(cluster: KubernetesCluster, manage_type: str) -> None: +def label_namespace_pss(cluster: KubernetesCluster) -> None: + security_enabled = is_security_enabled(cluster.inventory) first_control_plane = cluster.nodes["control-plane"].get_first_member() # set/delete labels on predifined plugins namsespaces for ns_name, profile in builtin.get_namespace_to_necessary_pss_profiles(cluster).items(): target_labels = get_labels_to_ensure_profile(cluster.inventory, profile) - if manage_type in ["apply", "install"] and target_labels: + if security_enabled and target_labels: cluster.log.debug(f"Set PSS labels for profile {profile} on namespace {ns_name}") command = "kubectl label ns {namespace} {lk}={lv} --overwrite" - else: # manage_type == "delete" or default labels are not necessary + else: # Pod Security is disabled or default labels are not necessary cluster.log.debug(f"Delete PSS labels from namespace {ns_name}") command = "kubectl label ns {namespace} {lk}- || true" target_labels = _get_default_labels(profile) @@ -914,7 +675,7 @@ def label_namespace_pss(cluster: KubernetesCluster, manage_type: str) -> None: ns_name = list(namespace.keys())[0] else: ns_name = namespace - if manage_type in ["apply", "install"]: + if security_enabled: if default_modes: # set labels that are set in default section cluster.log.debug(f"Set PSS labels on {ns_name} namespace from defaults") @@ -927,7 +688,7 @@ def label_namespace_pss(cluster: KubernetesCluster, manage_type: str) -> None: for item in list(namespace[ns_name]): first_control_plane.sudo(f"kubectl label ns {ns_name} " f"pod-security.kubernetes.io/{item}={namespace[ns_name][item]} --overwrite") - elif manage_type == "delete": + else: # delete labels that are set in default section if default_modes: cluster.log.debug(f"Delete PSS labels on {ns_name} namespace from defaults") @@ -945,4 +706,4 @@ def check_inventory(cluster: KubernetesCluster) -> None: # check if 'admission' option in cluster.yaml and procedure.yaml are inconsistent if cluster.context.get('initial_procedure') == 'manage_pss' and cluster.inventory["rbac"]["admission"] != "pss" or \ cluster.context.get('initial_procedure') == 'manage_psp' and cluster.inventory["rbac"]["admission"] != "psp": - raise Exception("Procedure config and cluster config are inconsistent. Please check 'admission' option") + raise Exception(ERROR_INCONSISTENT_INVENTORIES) diff --git a/kubemarine/core/defaults.py b/kubemarine/core/defaults.py index a994c639f..178329157 100755 --- a/kubemarine/core/defaults.py +++ b/kubemarine/core/defaults.py @@ -41,6 +41,7 @@ "kubemarine.kubernetes.enrich_restore_inventory", "kubemarine.core.defaults.compile_inventory", "kubemarine.core.defaults.manage_primitive_values", + "kubemarine.kubernetes.enrich_reconfigure_inventory", "kubemarine.plugins.enrich_upgrade_inventory", "kubemarine.packages.enrich_inventory", "kubemarine.packages.enrich_upgrade_inventory", diff --git a/kubemarine/core/group.py b/kubemarine/core/group.py index 7b1d88ee7..98beaafb1 100755 --- a/kubemarine/core/group.py +++ b/kubemarine/core/group.py @@ -27,7 +27,7 @@ from kubemarine.core import utils, log, errors from kubemarine.core.connections import ConnectionPool from kubemarine.core.executor import ( - RawExecutor, Token, GenericResult, RunnersResult, HostToResult, Callback, TokenizedResult, + RawExecutor, Token, GenericResult, RunnersResult, HostToResult, Callback, TokenizedResult, UnexpectedExit, ) NodeConfig = Dict[str, Any] @@ -710,6 +710,62 @@ def _await_rebooted_nodes(self, timeout: int = None, initial_boot_history: Runne executor = RawExecutor(self.cluster) return executor.wait_for_boot(self.get_hosts(), timeout, initial_boot_history) + def wait_command_successful(self, command: str, + *, + retries: int = 15, timeout: int = 5, + hide: bool = False) -> None: + logger = self.cluster.log + + def attempt() -> bool: + result = self.sudo(command, warn=True, hide=hide) + if hide: + logger.verbose(result) + for host in result.get_failed_hosts_list(): + stderr = result[host].stderr.rstrip('\n') + if stderr: + logger.debug(f"{host}: {stderr}") + + return not result.is_any_failed() + + utils.wait_command_successful(logger, attempt, retries, timeout) + + def wait_commands_successful(self, commands: List[str], + *, + retries: int = 15, timeout: int = 5, + sudo: bool = True) -> None: + if len(self.nodes) != 1: + raise Exception("Waiting for few commands is currently supported only for single node") + + logger = self.cluster.log + remained_commands = list(commands) + + def attempt() -> bool: + defer = self.new_defer() + for command in remained_commands: + if sudo: + defer.sudo(command) + else: + defer.run(command) + + try: + defer.flush() + except RemoteGroupException as e: + results = e.results[self.get_host()] + for result in results: + if isinstance(result, UnexpectedExit): + logger.debug(result.result.stderr) + break + elif isinstance(result, Exception): + raise + + del remained_commands[0] + + return False + + return True + + utils.wait_command_successful(logger, attempt, retries, timeout) + def get_local_file_sha1(self, filename: str) -> str: return utils.get_local_file_sha1(filename) diff --git a/kubemarine/core/schema.py b/kubemarine/core/schema.py index 9e8c2fd4b..f874bca4b 100644 --- a/kubemarine/core/schema.py +++ b/kubemarine/core/schema.py @@ -324,9 +324,9 @@ def _friendly_msg(validator: jsonschema.Draft7Validator, error: jsonschema.Valid # In current 3rd-party version the error should not appear, but let's still fall back to default behaviour pass - if _validated_by(error, 'not') and isinstance(error.validator_value, dict) \ - and error.validator_value.get('enum', {}) == ['<<'] and "propertyNames" in error.schema_path: - return "Property name '<<' is unexpected" + if (_validated_by(error, 'not') and isinstance(error.validator_value, dict) + and set(error.validator_value) == {'enum'} and "propertyNames" in error.schema_path): + return f"Property name {error.instance!r} is unexpected" if _validated_by(error, 'enum'): if "propertyNames" not in error.schema_path: diff --git a/kubemarine/core/utils.py b/kubemarine/core/utils.py index cb156b029..2f803f537 100755 --- a/kubemarine/core/utils.py +++ b/kubemarine/core/utils.py @@ -11,6 +11,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import difflib import hashlib import io import ipaddress @@ -22,7 +23,7 @@ import time import tarfile -from typing import Tuple, Callable, List, TextIO, cast, Union, TypeVar, Dict, Sequence +from typing import Tuple, Callable, List, TextIO, cast, Union, TypeVar, Dict, Sequence, Optional import deepdiff # type: ignore[import-untyped] import yaml @@ -170,7 +171,7 @@ def make_ansible_inventory(location: str, c: object) -> None: def get_current_timestamp_formatted() -> str: - return datetime.now().strftime("%Y%m%d-%H%M%S") + return datetime.now().strftime("%Y-%m-%d-%H-%M-%S") def get_final_inventory(c: object, initial_inventory: dict = None, procedure_initial_inventory: dict = None) -> dict: @@ -199,6 +200,7 @@ def get_final_inventory(c: object, initial_inventory: dict = None, procedure_ini remove_node.remove_node_finalize_inventory, lambda cluster, inventory, _: kubernetes.restore_finalize_inventory(cluster, inventory), lambda cluster, inventory, _: kubernetes.upgrade_finalize_inventory(cluster, inventory), + kubernetes.reconfigure_finalize_inventory, thirdparties.restore_finalize_inventory, thirdparties.upgrade_finalize_inventory, thirdparties.migrate_cri_finalize_inventory, @@ -275,26 +277,13 @@ def get_dump_filepath(context: dict, filename: str) -> str: return get_external_resource_path(os.path.join(context['execution_arguments']['dump_location'], 'dump', filename)) -def wait_command_successful(g: object, command: str, - retries: int = 15, timeout: int = 5, - warn: bool = True, hide: bool = False) -> None: - from kubemarine.core.group import NodeGroup - group = cast(NodeGroup, g) - - log = group.cluster.log - +def wait_command_successful(logger: log.EnhancedLogger, attempt: Callable[[], bool], + retries: int, timeout: int) -> None: while retries > 0: - log.debug("Waiting for command to succeed, %s retries left" % retries) - result = group.sudo(command, warn=warn, hide=hide) - if hide: - log.verbose(result) - for host in result.get_failed_hosts_list(): - stderr = result[host].stderr.rstrip('\n') - if stderr: - log.debug(f"{host}: {stderr}") - - if not result.is_any_failed(): - log.debug("Command succeeded") + logger.debug("Waiting for command to succeed, %s retries left" % retries) + result = attempt() + if result: + logger.debug("Command succeeded") return retries = retries - 1 time.sleep(timeout) @@ -492,6 +481,25 @@ def print_diff(logger: log.EnhancedLogger, diff: deepdiff.DeepDiff) -> None: logger.debug(yaml.safe_dump(yaml.safe_load(diff.to_json()))) +def get_unified_diff(old: str, new: str, fromfile: str = '', tofile: str = '') -> Optional[str]: + diff = list(difflib.unified_diff( + old.splitlines(), new.splitlines(), + fromfile=fromfile, tofile=tofile, + lineterm='')) + + if diff: + return '\n'.join(diff) + + return None + + +def get_yaml_diff(old: str, new: str, fromfile: str = '', tofile: str = '') -> Optional[str]: + if yaml.safe_load(old) == yaml.safe_load(new): + return None + + return get_unified_diff(old, new, fromfile, tofile) + + def isipv(address: str, versions: List[int]) -> bool: return ipaddress.ip_network(address).version in versions diff --git a/kubemarine/core/yaml_merger.py b/kubemarine/core/yaml_merger.py index f1f715e62..908c2f569 100644 --- a/kubemarine/core/yaml_merger.py +++ b/kubemarine/core/yaml_merger.py @@ -53,4 +53,13 @@ def list_merger(config: Merger, path: list, base: list, nxt: list) -> list: ], ["override"], ["override"] -) \ No newline at end of file +) + +override_merger: Merger = Merger( + [ + (list, ["override"]), + (dict, ["merge"]) + ], + ["override"], + ["override"] +) diff --git a/kubemarine/demo.py b/kubemarine/demo.py index e554af8c0..f8e1add3a 100644 --- a/kubemarine/demo.py +++ b/kubemarine/demo.py @@ -104,9 +104,19 @@ def is_called(self, host: str, do_type: str, args: List[str]) -> bool: :param args: Required command arguments :return: Boolean """ + return self.called_times(host, do_type, args) > 0 + + def called_times(self, host: str, do_type: str, args: List[str]) -> int: + """ + Returns number of times the specified command was executed in FakeShell for the specified connection. + :param host: host to check, for which the desirable command should have been executed. + :param do_type: The type of required command + :param args: Required command arguments + :return: number of calls + """ found_entries = self.history_find(host, do_type, args) total_used_times: int = sum(found_entry['used_times'] for found_entry in found_entries) - return total_used_times > 0 + return total_used_times class FakeFS: diff --git a/kubemarine/k8s_certs.py b/kubemarine/k8s_certs.py index 1917bb489..9d3891d2a 100644 --- a/kubemarine/k8s_certs.py +++ b/kubemarine/k8s_certs.py @@ -48,21 +48,8 @@ def renew_apply(control_planes: NodeGroup) -> None: kubernetes.copy_admin_config(log, control_planes) # for some reason simple pod delete do not work for certs update - we need to delete containers themselves - control_planes.call(force_restart_control_plane) - - for control_plane in control_planes.get_ordered_members_list(): - kubernetes.wait_for_any_pods(control_planes.cluster, control_plane, apply_filter=control_plane.get_node_name()) - - -def force_restart_control_plane(control_planes: NodeGroup) -> None: - cri_impl = control_planes.cluster.inventory['services']['cri']['containerRuntime'] - restart_containers = ["etcd", "kube-scheduler", "kube-apiserver", "kube-controller-manager"] - c_filter = "grep -e %s" % " -e ".join(restart_containers) - - if cri_impl == "docker": - control_planes.sudo("sudo docker container rm -f $(sudo docker ps -a | %s | awk '{ print $1 }')" % c_filter, warn=True) - else: - control_planes.sudo("sudo crictl rm -f $(sudo crictl ps -a | %s | awk '{ print $1 }')" % c_filter, warn=True) + control_planes.call(kubernetes.components.restart_components, + components=kubernetes.components.CONTROL_PLANE_COMPONENTS) def verify_all_is_absent_or_single(cert_list: List[str]) -> None: diff --git a/kubemarine/kubernetes/__init__.py b/kubemarine/kubernetes/__init__.py index 13f8248d9..f696ce2d4 100644 --- a/kubemarine/kubernetes/__init__.py +++ b/kubemarine/kubernetes/__init__.py @@ -19,23 +19,21 @@ import uuid from contextlib import contextmanager from copy import deepcopy -from typing import List, Dict, Iterator, Any, Optional, Callable +from typing import List, Dict, Iterator, Any, Optional import yaml from jinja2 import Template import ipaddress -from kubemarine import system, plugins, admission, etcd, packages +from kubemarine import system, admission, etcd, packages from kubemarine.core import utils, static, summary, log, errors from kubemarine.core.cluster import KubernetesCluster from kubemarine.core.executor import Token -from kubemarine.core.group import ( - NodeGroup, AbstractGroup, DeferredGroup, - NodeConfig, RunnersGroupResult, RunResult, CollectorCallback -) +from kubemarine.core.group import NodeGroup, DeferredGroup, RunnersGroupResult, CollectorCallback from kubemarine.core.errors import KME +from kubemarine.core.yaml_merger import default_merger from kubemarine.cri import containerd -from kubemarine.kubernetes.object import KubernetesObject +from kubemarine.kubernetes import components ERROR_DOWNGRADE='Kubernetes old version \"%s\" is greater than new one \"%s\"' ERROR_SAME='Kubernetes old version \"%s\" is the same as new one \"%s\"' @@ -43,15 +41,9 @@ ERROR_MINOR_RANGE_EXCEEDED='Minor version \"%s\" rises to new \"%s\" more than one' ERROR_NOT_LATEST_PATCH='New version \"%s\" is not the latest supported patch version \"%s\"' - -def is_container_runtime_not_configurable(cluster: KubernetesCluster) -> bool: - kubernetes_version = cluster.inventory["services"]["kubeadm"]["kubernetesVersion"] - return utils.version_key(kubernetes_version)[0:2] >= utils.minor_version_key("v1.27") - - -def kube_proxy_overwrites_higher_system_values(cluster: KubernetesCluster) -> bool: - kubernetes_version = cluster.inventory["services"]["kubeadm"]["kubernetesVersion"] - return utils.version_key(kubernetes_version)[0:2] >= utils.minor_version_key("v1.29") +ERROR_KUBELET_PATCH_NOT_KUBERNETES_NODE = "%s patch can be uploaded only to control-plane or worker nodes" +ERROR_CONTROL_PLANE_PATCH_NOT_CONTROL_PLANE_NODE = "%s patch can be uploaded only to control-plane nodes" +ERROR_KUBEADM_DOES_NOT_SUPPORT_PATCHES_KUBELET = "Patches for kubelet are not supported in Kubernetes {version}" def add_node_enrichment(inventory: dict, cluster: KubernetesCluster, procedure_inventory: dict = None) -> dict: @@ -152,7 +144,27 @@ def restore_finalize_inventory(cluster: KubernetesCluster, inventory: dict) -> d return inventory -def enrich_inventory(inventory: dict, _: KubernetesCluster) -> dict: +def enrich_reconfigure_inventory(inventory: dict, cluster: KubernetesCluster) -> dict: + return reconfigure_finalize_inventory(cluster, inventory) + + +def reconfigure_finalize_inventory(cluster: KubernetesCluster, inventory: dict, procedure_inventory: dict = None) -> dict: + if procedure_inventory is None: + procedure_inventory = cluster.procedure_inventory + + if cluster.context.get("initial_procedure") != "reconfigure": + return inventory + + kubeadm_sections = {s: v for s, v in procedure_inventory.get('services', {}).items() + if s in ('kubeadm', 'kubeadm_kubelet', 'kubeadm_kube-proxy', 'kubeadm_patches')} + + if kubeadm_sections: + default_merger.merge(inventory.setdefault('services', {}), deepcopy(kubeadm_sections)) + + return inventory + + +def enrich_inventory(inventory: dict, cluster: KubernetesCluster) -> dict: kubeadm = inventory['services']['kubeadm'] kubeadm['dns'].setdefault('imageRepository', f"{kubeadm['imageRepository']}/coredns") @@ -173,7 +185,7 @@ def enrich_inventory(inventory: dict, _: KubernetesCluster) -> dict: if inventory.get("public_cluster_ip"): enriched_certsans.append(inventory["public_cluster_ip"]) - certsans = inventory["services"]["kubeadm"]['apiServer']['certSANs'] + certsans = kubeadm['apiServer']['certSANs'] # do not overwrite apiServer.certSANs, but append - may be user specified something already there? for name in enriched_certsans: @@ -202,19 +214,22 @@ def enrich_inventory(inventory: dict, _: KubernetesCluster) -> dict: # Validate the provided podSubnet and serviceSubnet IP addresses for subnet in ('podSubnet', 'serviceSubnet'): - utils.isipv(inventory['services']['kubeadm']['networking'][subnet], [4, 6]) + utils.isipv(kubeadm['networking'][subnet], [4, 6]) # validate nodes in kubeadm_patches (groups are validated with JSON schema) - for node in inventory["nodes"]: - for control_plane_item in inventory["services"]["kubeadm_patches"]: - for i in inventory["services"]["kubeadm_patches"][control_plane_item]: - if i.get('nodes') is not None: - for n in i['nodes']: - if node['name'] == n: - if control_plane_item == 'kubelet' and 'control-plane' not in node['roles'] and 'worker' not in node['roles']: - raise Exception("%s patch can be uploaded only to control-plane or worker nodes" % control_plane_item) - if control_plane_item != 'kubelet' and ('control-plane' not in node['roles']): - raise Exception("%s patch can be uploaded only to control-plane nodes" % control_plane_item) + for control_plane_item, patches in inventory["services"]["kubeadm_patches"].items(): + for patch in patches: + if control_plane_item == 'kubelet' and not components.kubelet_supports_patches(cluster): + raise Exception(ERROR_KUBEADM_DOES_NOT_SUPPORT_PATCHES_KUBELET.format(version=kubeadm['kubernetesVersion'])) + + if 'nodes' not in patch: + continue + + for node in cluster.get_nodes_by_names(patch['nodes']): + if control_plane_item == 'kubelet' and 'control-plane' not in node['roles'] and 'worker' not in node['roles']: + raise Exception(ERROR_KUBELET_PATCH_NOT_KUBERNETES_NODE % control_plane_item) + if control_plane_item != 'kubelet' and ('control-plane' not in node['roles']): + raise Exception(ERROR_CONTROL_PLANE_PATCH_NOT_CONTROL_PLANE_NODE % control_plane_item) if not any_worker_found: raise KME("KME0004") @@ -394,45 +409,10 @@ def join_new_control_plane(group: NodeGroup) -> None: def join_control_plane(cluster: KubernetesCluster, node: NodeGroup, join_dict: dict) -> None: log = cluster.log - node_config = node.get_config() node_name = node.get_node_name() - defer = node.new_defer() - - join_config: dict = { - 'apiVersion': cluster.inventory["services"]["kubeadm"]['apiVersion'], - 'kind': 'JoinConfiguration', - 'discovery': { - 'bootstrapToken': { - 'apiServerEndpoint': cluster.inventory["services"]["kubeadm"]['controlPlaneEndpoint'], - 'token': join_dict['token'], - 'caCertHashes': [ - join_dict['discovery-token-ca-cert-hash'] - ] - } - }, - 'controlPlane': { - 'certificateKey': join_dict['certificate-key'], - 'localAPIEndpoint': { - 'advertiseAddress': node_config['internal_address'], - } - }, - 'patches': {'directory': '/etc/kubernetes/patches'}, - } - - if cluster.inventory['services']['kubeadm']['controllerManager']['extraArgs'].get( - 'external-cloud-volume-plugin'): - join_config['nodeRegistration'] = { - 'kubeletExtraArgs': { - 'cloud-provider': 'external' - } - } - - if 'worker' in node_config['roles']: - join_config.setdefault('nodeRegistration', {})['taints'] = [] - - configure_container_runtime(cluster, join_config) - config = get_kubeadm_config(cluster.inventory) + "---\n" + yaml.dump(join_config, default_flow_style=False) + join_config = components.get_init_config(cluster, node, init=False, join_dict=join_dict) + config = components.get_kubeadm_config(cluster, join_config) utils.dump_file(cluster, config, 'join-config_%s.yaml' % node_name) @@ -441,7 +421,7 @@ def join_control_plane(cluster: KubernetesCluster, node: NodeGroup, join_dict: d node.put(io.StringIO(config), '/etc/kubernetes/join-config.yaml', sudo=True) # put control-plane patches - create_kubeadm_patches_for_node(cluster, node) + components.create_kubeadm_patches_for_node(cluster, node) # copy admission config to control-plane admission.copy_pss(node) @@ -458,11 +438,9 @@ def join_control_plane(cluster: KubernetesCluster, node: NodeGroup, join_dict: d " --ignore-preflight-errors='" + cluster.inventory['services']['kubeadm_flags']['ignorePreflightErrors'] + "'" " --v=5", hide=False) - defer.sudo("systemctl restart kubelet") - copy_admin_config(log, defer) - defer.flush() + copy_admin_config(log, node) - wait_for_any_pods(cluster, node, apply_filter=node_name) + components.wait_for_pods(node) @contextmanager @@ -486,7 +464,7 @@ def local_admin_config(nodes: NodeGroup) -> Iterator[str]: nodes.sudo(f'rm -f {temp_filepath}') -def copy_admin_config(logger: log.EnhancedLogger, nodes: AbstractGroup[RunResult]) -> None: +def copy_admin_config(logger: log.EnhancedLogger, nodes: NodeGroup) -> None: logger.debug("Setting up admin-config...") command = "mkdir -p /root/.kube && sudo cp -f /etc/kubernetes/admin.conf /root/.kube/config" nodes.sudo(command) @@ -539,32 +517,10 @@ def init_first_control_plane(group: NodeGroup) -> None: log = cluster.log first_control_plane = group.get_first_member() - node_config = first_control_plane.get_config() node_name = first_control_plane.get_node_name() - init_config: dict = { - 'apiVersion': cluster.inventory["services"]["kubeadm"]['apiVersion'], - 'kind': 'InitConfiguration', - 'localAPIEndpoint': { - 'advertiseAddress': node_config['internal_address'] - }, - 'patches': {'directory': '/etc/kubernetes/patches'}, - } - - if cluster.inventory['services']['kubeadm']['controllerManager']['extraArgs'].get( - 'external-cloud-volume-plugin'): - init_config['nodeRegistration'] = { - 'kubeletExtraArgs': { - 'cloud-provider': 'external' - } - } - - if 'worker' in node_config['roles']: - init_config.setdefault('nodeRegistration', {})['taints'] = [] - - configure_container_runtime(cluster, init_config) - - config = get_kubeadm_config(cluster.inventory) + "---\n" + yaml.dump(init_config, default_flow_style=False) + init_config = components.get_init_config(cluster, first_control_plane, init=True) + config = components.get_kubeadm_config(cluster, init_config) utils.dump_file(cluster, config, 'init-config_%s.yaml' % node_name) @@ -573,7 +529,7 @@ def init_first_control_plane(group: NodeGroup) -> None: first_control_plane.put(io.StringIO(config), '/etc/kubernetes/init-config.yaml', sudo=True) # put control-plane patches - create_kubeadm_patches_for_node(cluster, first_control_plane) + components.create_kubeadm_patches_for_node(cluster, first_control_plane) # copy admission config to first control-plane first_control_plane.call(admission.copy_pss) @@ -616,37 +572,20 @@ def init_first_control_plane(group: NodeGroup) -> None: join_dict["worker_join_command"] = worker_join_command cluster.context["join_dict"] = join_dict - wait_for_any_pods(cluster, first_control_plane, apply_filter=node_name) + components.wait_for_pods(first_control_plane) # refresh cluster installation status in cluster context is_cluster_installed(cluster) -def wait_for_any_pods(cluster: KubernetesCluster, connection: NodeGroup, apply_filter: str = None) -> None: - wait_for_pods(cluster, connection, [ - 'kube-apiserver', - 'kube-controller-manager', - 'kube-proxy', - 'kube-scheduler', - 'etcd' - ], apply_filter=apply_filter) - - -def wait_for_pods(cluster: KubernetesCluster, connection: NodeGroup, - pods_list: List[str], apply_filter: str = None) -> None: - plugins.expect_pods(cluster, pods_list, node=connection, apply_filter=apply_filter, - timeout=cluster.inventory['globals']['expect']['pods']['kubernetes']['timeout'], - retries=cluster.inventory['globals']['expect']['pods']['kubernetes']['retries']) - - def wait_uncordon(node: NodeGroup) -> None: cluster = node.cluster timeout_config = cluster.inventory['globals']['expect']['pods']['kubernetes'] # This forces to use local API server and waits till it is up. with local_admin_config(node) as kubeconfig: - utils.wait_command_successful(node, f"kubectl --kubeconfig {kubeconfig} uncordon {node.get_node_name()}", - hide=False, - timeout=timeout_config['timeout'], - retries=timeout_config['retries']) + node.wait_command_successful(f"kubectl --kubeconfig {kubeconfig} uncordon {node.get_node_name()}", + hide=False, + timeout=timeout_config['timeout'], + retries=timeout_config['retries']) def wait_for_nodes(group: NodeGroup) -> None: @@ -697,35 +636,14 @@ def wait_for_nodes(group: NodeGroup) -> None: def init_workers(group: NodeGroup) -> None: + if group.is_empty(): + return + cluster: KubernetesCluster = group.cluster join_dict = cluster.context.get("join_dict", get_join_dict(group)) - join_config = { - 'apiVersion': group.cluster.inventory["services"]["kubeadm"]['apiVersion'], - 'kind': 'JoinConfiguration', - 'discovery': { - 'bootstrapToken': { - 'apiServerEndpoint': cluster.inventory["services"]["kubeadm"]['controlPlaneEndpoint'], - 'token': join_dict['token'], - 'caCertHashes': [ - join_dict['discovery-token-ca-cert-hash'] - ] - } - }, - 'patches': {'directory': '/etc/kubernetes/patches'}, - } - - if cluster.inventory['services']['kubeadm']['controllerManager']['extraArgs'].get( - 'external-cloud-volume-plugin'): - join_config['nodeRegistration'] = { - 'kubeletExtraArgs': { - 'cloud-provider': 'external' - } - } - - configure_container_runtime(cluster, join_config) - - config = yaml.dump(join_config, default_flow_style=False) + join_config = components.get_init_config(cluster, group, init=False, join_dict=join_dict) + config = yaml.dump(join_config) utils.dump_file(cluster, config, 'join-config-workers.yaml') @@ -734,7 +652,7 @@ def init_workers(group: NodeGroup) -> None: # put control-plane patches for node in group.get_ordered_members_list(): - create_kubeadm_patches_for_node(cluster, node) + components.create_kubeadm_patches_for_node(cluster, node) cluster.log.debug('Joining workers...') @@ -745,6 +663,8 @@ def init_workers(group: NodeGroup) -> None: " --v=5", hide=False) + components.wait_for_pods(node) + def apply_labels(group: NodeGroup) -> RunnersGroupResult: cluster: KubernetesCluster = group.cluster @@ -766,7 +686,6 @@ def apply_labels(group: NodeGroup) -> RunnersGroupResult: log.debug("Successfully applied additional labels") return control_plane.sudo("kubectl get nodes --show-labels") - # TODO: Add wait for pods on worker nodes def apply_taints(group: NodeGroup) -> RunnersGroupResult: @@ -788,7 +707,7 @@ def apply_taints(group: NodeGroup) -> RunnersGroupResult: return control_plane.sudo( "kubectl get nodes -o=jsonpath=" - "'{range .items[*]}{\"node: \"}{.metadata.name}{\"\\ntaints: \"}{.spec.taints}{\"\\n\"}'") + "'{range .items[*]}{\"node: \"}{.metadata.name}{\"\\ntaints: \"}{.spec.taints}{\"\\n\"}{end}'") def is_cluster_installed(cluster: KubernetesCluster) -> bool: @@ -809,40 +728,6 @@ def is_cluster_installed(cluster: KubernetesCluster) -> bool: return False -def get_kubeadm_config(inventory: dict) -> str: - kubeadm_kubelet = yaml.dump(inventory["services"]["kubeadm_kubelet"], default_flow_style=False) - kubeadm_kube_proxy = yaml.dump(inventory["services"]["kubeadm_kube-proxy"], default_flow_style=False) - kubeadm = yaml.dump(inventory["services"]["kubeadm"], default_flow_style=False) - return f'{kubeadm_kube_proxy}---\n{kubeadm_kubelet}---\n{kubeadm}' - - -def reconfigure_kube_proxy_configmap(control_plane: NodeGroup, mutate_func: Callable[[dict], dict]) -> None: - cluster: KubernetesCluster = control_plane.cluster - - # Load kube-proxy config map and retrieve config - kube_proxy_cm = KubernetesObject(cluster, 'ConfigMap', 'kube-proxy', 'kube-system') - kube_proxy_cm.reload(control_plane) - cluster_config: dict = yaml.safe_load(kube_proxy_cm.obj["data"]["config.conf"]) - - # Always perform the reconfiguration entirely even if nothing is changed. - # This is necessary because the operation is not atomic, but idempotent. - cluster_config = mutate_func(cluster_config) - kube_proxy_cm.obj["data"]["config.conf"] = yaml.dump(cluster_config) - - # Apply updated kube-proxy config map - kube_proxy_cm.apply(control_plane) - - for node in cluster.make_group_from_roles(['control-plane', 'worker']).get_ordered_members_list(): - node_name = node.get_node_name() - control_plane.sudo( - f"kubectl delete pod -n kube-system $(" - f" sudo kubectl describe node {node_name} " - f" | awk '/kube-system\\s+kube-proxy-[a-z,0-9]{{5}}/{{print $2}}'" - f")") - - wait_for_pods(cluster, control_plane,['kube-proxy'], apply_filter=node_name) - - def upgrade_first_control_plane(upgrade_group: NodeGroup, cluster: KubernetesCluster, **drain_kwargs: Any) -> None: version = cluster.inventory["services"]["kubeadm"]["kubernetesVersion"] first_control_plane = cluster.nodes['control-plane'].get_first_member() @@ -855,7 +740,7 @@ def upgrade_first_control_plane(upgrade_group: NodeGroup, cluster: KubernetesClu cluster.log.debug("Upgrading first control-plane \"%s\"" % node_name) # put control-plane patches - create_kubeadm_patches_for_node(cluster, first_control_plane) + components.create_kubeadm_patches_for_node(cluster, first_control_plane) flags = "-f --certificate-renewal=true --ignore-preflight-errors='%s' --patches=/etc/kubernetes/patches" % cluster.inventory['services']['kubeadm_flags']['ignorePreflightErrors'] @@ -873,7 +758,7 @@ def upgrade_first_control_plane(upgrade_group: NodeGroup, cluster: KubernetesClu copy_admin_config(cluster.log, first_control_plane) expect_kubernetes_version(cluster, version, apply_filter=node_name) - wait_for_any_pods(cluster, first_control_plane, apply_filter=node_name) + components.wait_for_pods(first_control_plane) exclude_node_from_upgrade_list(first_control_plane, node_name) @@ -892,7 +777,7 @@ def upgrade_other_control_planes(upgrade_group: NodeGroup, cluster: KubernetesCl cluster.log.debug("Upgrading control-plane \"%s\"" % node_name) # put control-plane patches - create_kubeadm_patches_for_node(cluster, node) + components.create_kubeadm_patches_for_node(cluster, node) drain_cmd = prepare_drain_command(cluster, node_name, **drain_kwargs) node.sudo(drain_cmd, hide=False) @@ -908,7 +793,7 @@ def upgrade_other_control_planes(upgrade_group: NodeGroup, cluster: KubernetesCl expect_kubernetes_version(cluster, version, apply_filter=node_name) copy_admin_config(cluster.log, node) - wait_for_any_pods(cluster, node, apply_filter=node_name) + components.wait_for_pods(node) exclude_node_from_upgrade_list(first_control_plane, node_name) @@ -927,7 +812,7 @@ def upgrade_workers(upgrade_group: NodeGroup, cluster: KubernetesCluster, **drai cluster.log.debug("Upgrading worker \"%s\"" % node_name) # put control-plane patches - create_kubeadm_patches_for_node(cluster, node) + components.create_kubeadm_patches_for_node(cluster, node) drain_cmd = prepare_drain_command(cluster, node_name, **drain_kwargs) first_control_plane.sudo(drain_cmd, hide=False) @@ -945,6 +830,8 @@ def upgrade_workers(upgrade_group: NodeGroup, cluster: KubernetesCluster, **drai # workers do not have system pods to wait for their start exclude_node_from_upgrade_list(first_control_plane, node_name) + components.wait_for_pods(node) + def prepare_drain_command(cluster: KubernetesCluster, node_name: str, *, @@ -1128,22 +1015,6 @@ def recalculate_proper_timeout(cluster: KubernetesCluster, timeout: int) -> int: return timeout * 10 * cluster.nodes['all'].nodes_amount() -def configure_container_runtime(cluster: KubernetesCluster, kubeadm_config: dict) -> None: - if cluster.inventory['services']['cri']['containerRuntime'] == "containerd": - if 'nodeRegistration' not in kubeadm_config: - kubeadm_config['nodeRegistration'] = {} - if 'kubeletExtraArgs' not in kubeadm_config['nodeRegistration']: - kubeadm_config['nodeRegistration']['kubeletExtraArgs'] = {} - - kubeadm_config['nodeRegistration']['criSocket'] = '/var/run/containerd/containerd.sock' - - if not is_container_runtime_not_configurable(cluster): - kubeadm_config['nodeRegistration']['kubeletExtraArgs']['container-runtime'] = 'remote' - - kubeadm_config['nodeRegistration']['kubeletExtraArgs']['container-runtime-endpoint'] = \ - 'unix:///run/containerd/containerd.sock' - - def exclude_node_from_upgrade_list(first_control_plane: NodeGroup, node_name: str) -> None: first_control_plane.sudo('sed -i \'/%s/d\' /etc/kubernetes/nodes-k8s-versions.txt' % node_name, warn=True) @@ -1265,19 +1136,14 @@ def images_prepull(group: DeferredGroup, collector: CollectorCallback) -> Token: """ Prepull kubeadm images on group. - :param group: NodeGroup where prepull should be performed. + :param group: single-node NodeGroup where prepull should be performed. :param collector: CollectorCallback instance :return: NodeGroupResult from all nodes in presented group. """ - config = get_kubeadm_config(group.cluster.inventory) - kubeadm_init: dict = { - 'apiVersion': group.cluster.inventory["services"]["kubeadm"]['apiVersion'], - 'kind': 'InitConfiguration', - } - - configure_container_runtime(group.cluster, kubeadm_init) - config = f'{config}---\n{yaml.dump(kubeadm_init, default_flow_style=False)}' + cluster: KubernetesCluster = group.cluster + kubeadm_init = components.get_init_config(cluster, group, init=True) + config = components.get_kubeadm_config(cluster, kubeadm_init) group.put(io.StringIO(config), '/etc/kubernetes/prepull-config.yaml', sudo=True) @@ -1349,56 +1215,6 @@ def get_nodes_conditions(nodes_description: dict) -> Dict[str, Dict[str, dict]]: return result -# function to get dictionary of flags to be patched for a given control plane item and a given node -def get_patched_flags_for_control_plane_item(inventory: dict, control_plane_item: str, node: NodeConfig) -> Dict[str, str]: - flags = {} - - for n in inventory['services']['kubeadm_patches'][control_plane_item]: - if n.get('groups') is not None and list(set(node['roles']) & set(n['groups'])): - if n.get('patch') is not None: - for arg, value in n['patch'].items(): - flags[arg] = value - if n.get('nodes') is not None and node['name'] in n['nodes']: - if n.get('patch') is not None: - for arg, value in n['patch'].items(): - flags[arg] = value - - # we always set binding-address to the node's internal address for apiServer - if control_plane_item == 'apiServer' and 'control-plane' in node['roles']: - flags['bind-address'] = node['internal_address'] - - return flags - - -# function to create kubeadm patches and put them to a node -def create_kubeadm_patches_for_node(cluster: KubernetesCluster, node: NodeGroup) -> None: - cluster.log.verbose(f"Create and upload kubeadm patches to %s..." % node.get_node_name()) - node.sudo('sudo rm -rf /etc/kubernetes/patches ; sudo mkdir -p /etc/kubernetes/patches', warn=True) - - control_plane_patch_files = { - 'apiServer' : 'kube-apiserver+json.json', - 'etcd' : 'etcd+json.json', - 'controllerManager' : 'kube-controller-manager+json.json', - 'scheduler' : 'kube-scheduler+json.json', - 'kubelet' : 'kubeletconfiguration.yaml' - } - - # read patches content from inventory and upload patch files to a node - node_config = node.get_config() - for control_plane_item in cluster.inventory['services']['kubeadm_patches']: - patched_flags = get_patched_flags_for_control_plane_item(cluster.inventory, control_plane_item, node_config) - if patched_flags: - if control_plane_item == 'kubelet': - template_filename = 'templates/patches/kubelet.yaml.j2' - else: - template_filename = 'templates/patches/control-plane-pod.json.j2' - - control_plane_patch = Template(utils.read_internal(template_filename)).render(flags=patched_flags) - patch_file = '/etc/kubernetes/patches/' + control_plane_patch_files[control_plane_item] - node.put(io.StringIO(control_plane_patch + "\n"), patch_file, sudo=True) - node.sudo(f'chmod 644 {patch_file}') - - def fix_flag_kubelet(group: NodeGroup) -> bool: kubeadm_flags_file = "/var/lib/kubelet/kubeadm-flags.env" cluster = group.cluster @@ -1415,7 +1231,7 @@ def fix_flag_kubelet(group: NodeGroup) -> bool: for node in exe.group.get_ordered_members_list(): kubeadm_flags = collector.result[node.get_host()].stdout updated_kubeadm_flags = kubeadm_flags - if is_container_runtime_not_configurable(cluster): + if components.is_container_runtime_not_configurable(cluster): # remove the deprecated kubelet flag for versions starting from 1.27.0 updated_kubeadm_flags = updated_kubeadm_flags.replace(container_runtime_flag, '') diff --git a/kubemarine/kubernetes/components.py b/kubemarine/kubernetes/components.py new file mode 100644 index 000000000..8390e5e78 --- /dev/null +++ b/kubemarine/kubernetes/components.py @@ -0,0 +1,1097 @@ +import io +import re +import uuid +from copy import deepcopy +from textwrap import dedent +from typing import List, Optional, Dict, Callable, Sequence + +import yaml +from jinja2 import Template +from ordered_set import OrderedSet + +from kubemarine import plugins, system +from kubemarine.core import utils, log +from kubemarine.core.cluster import KubernetesCluster +from kubemarine.core.group import NodeConfig, NodeGroup, DeferredGroup, CollectorCallback, AbstractGroup, RunResult +from kubemarine.core.yaml_merger import override_merger +from kubemarine.kubernetes.object import KubernetesObject + +ERROR_WAIT_FOR_PODS_NOT_SUPPORTED = "Waiting for pods of {components} components is currently not supported" +ERROR_RESTART_NOT_SUPPORTED = "Restart of {components} components is currently not supported" +ERROR_RECONFIGURE_NOT_SUPPORTED = "Reconfiguration of {components} components is currently not supported" + +COMPONENTS_CONSTANTS: dict = { + 'kube-apiserver/cert-sans': { + 'sections': [ + ['services', 'kubeadm', 'apiServer', 'certSANs'], + ], + 'init_phase': 'certs apiserver', + }, + 'kube-apiserver': { + 'sections': [ + ['services', 'kubeadm', 'apiServer'], + ['services', 'kubeadm_patches', 'apiServer'], + ], + 'patch': { + 'section': 'apiServer', + 'target_template': r'kube-apiserver\*', + 'file': 'kube-apiserver+json.json' + }, + 'init_phase': 'control-plane apiserver', + }, + 'kube-scheduler': { + 'sections': [ + ['services', 'kubeadm', 'scheduler'], + ['services', 'kubeadm_patches', 'scheduler'], + ], + 'patch': { + 'section': 'scheduler', + 'target_template': r'kube-scheduler\*', + 'file': 'kube-scheduler+json.json' + }, + 'init_phase': 'control-plane scheduler', + }, + 'kube-controller-manager': { + 'sections': [ + ['services', 'kubeadm', 'controllerManager'], + ['services', 'kubeadm_patches', 'controllerManager'], + ], + 'patch': { + 'section': 'controllerManager', + 'target_template': r'kube-controller-manager\*', + 'file': 'kube-controller-manager+json.json' + }, + 'init_phase': 'control-plane controller-manager', + }, + 'etcd': { + 'sections': [ + ['services', 'kubeadm', 'etcd'], + ['services', 'kubeadm_patches', 'etcd'], + ], + 'patch': { + 'section': 'etcd', + 'target_template': r'etcd\*', + 'file': 'etcd+json.json' + }, + 'init_phase': 'etcd local', + }, + 'kubelet': { + 'sections': [ + ['services', 'kubeadm_kubelet'], + ['services', 'kubeadm_patches', 'kubelet'], + ], + 'patch': { + 'section': 'kubelet', + 'target_template': r'kubeletconfiguration\*', + 'file': 'kubeletconfiguration.yaml' + } + }, + 'kube-proxy': { + 'sections': [ + ['services', 'kubeadm_kube-proxy'], + ] + }, +} + +CONFIGMAPS_CONSTANTS = { + 'kubeadm-config': { + 'section': 'kubeadm', + 'key': 'ClusterConfiguration', + 'init_phase': 'upload-config kubeadm', + }, + 'kubelet-config': { + 'section': 'kubeadm_kubelet', + 'key': 'kubelet', + 'init_phase': 'upload-config kubelet', + }, + 'kube-proxy': { + 'section': 'kubeadm_kube-proxy', + 'key': 'config.conf', + }, +} + + +CONTROL_PLANE_COMPONENTS = ["kube-apiserver", "kube-scheduler", "kube-controller-manager", "etcd"] +CONTROL_PLANE_SPECIFIC_COMPONENTS = ['kube-apiserver/cert-sans'] + CONTROL_PLANE_COMPONENTS +NODE_COMPONENTS = ["kubelet", "kube-proxy"] +ALL_COMPONENTS = CONTROL_PLANE_SPECIFIC_COMPONENTS + NODE_COMPONENTS +COMPONENTS_SUPPORT_PATCHES = CONTROL_PLANE_COMPONENTS + ['kubelet'] + + +class KubeadmConfig: + def __init__(self, cluster: KubernetesCluster): + self.cluster = cluster + + inventory = cluster.inventory + self.maps: Dict[str, dict] = { + configmap: inventory["services"][constants['section']] + for configmap, constants in CONFIGMAPS_CONSTANTS.items()} + + self.loaded_maps: Dict[str, KubernetesObject] = {} + + def is_loaded(self, configmap: str) -> bool: + return configmap in self.loaded_maps + + def load(self, configmap: str, control_plane: NodeGroup, edit_func: Callable[[dict], dict] = None) -> dict: + """ + Load ConfigMap as object, retrieve plain config from it, and apply `edit_func` to the plain config if provided. + + :param configmap: name of ConfigMap + :param edit_func: function to apply changes + :param control_plane: Use this control plane node to fetch the ConfigMap. + """ + configmap_name = _get_configmap_name(self.cluster, configmap) + + configmap_obj = KubernetesObject(self.cluster, 'ConfigMap', configmap_name, 'kube-system') + configmap_obj.reload(control_plane) + + self.loaded_maps[configmap] = configmap_obj + + key = CONFIGMAPS_CONSTANTS[configmap]['key'] + config: dict = yaml.safe_load(configmap_obj.obj["data"][key]) + + if edit_func is not None: + config = edit_func(config) + configmap_obj.obj["data"][key] = yaml.dump(config) + + self.maps[configmap] = config + return config + + def apply(self, configmap: str, control_plane: NodeGroup) -> None: + """ + Apply ConfigMap that was previously changed using edit(). + + :param configmap: name of ConfigMap + :param control_plane: Use this control plane node to apply the ConfigMap. + :return: + """ + if not self.is_loaded(configmap): + raise ValueError(f"To apply changed {configmap} ConfigMap, it is necessary to fetch it first") + + self.loaded_maps[configmap].apply(control_plane) + + def to_yaml(self, init_config: dict) -> str: + configs = list(self.maps.values()) + configs.append(init_config) + return yaml.dump_all(configs) + + def merge_with_inventory(self, configmap: str) -> Callable[[dict], dict]: + def merge_func(config_: dict) -> dict: + patch_config: dict = KubeadmConfig(self.cluster).maps[configmap] + # It seems that all default lists are always overridden with custom instead of appending, + # and so override merger seems the most suitable. + config_ = override_merger.merge(config_, deepcopy(patch_config)) + return config_ + + return merge_func + + +def kubelet_config_unversioned(cluster: KubernetesCluster) -> bool: + return kubernetes_minor_release_at_least(cluster.inventory, "v1.24") + + +def kubelet_supports_patches(cluster: KubernetesCluster) -> bool: + return kubernetes_minor_release_at_least(cluster.inventory, "v1.25") + + +def kubeadm_extended_dryrun(cluster: KubernetesCluster) -> bool: + return kubernetes_minor_release_at_least(cluster.inventory, "v1.26") + + +def is_container_runtime_not_configurable(cluster: KubernetesCluster) -> bool: + return kubernetes_minor_release_at_least(cluster.inventory, "v1.27") + + +def kube_proxy_overwrites_higher_system_values(cluster: KubernetesCluster) -> bool: + return kubernetes_minor_release_at_least(cluster.inventory, "v1.29") + + +def kubernetes_minor_release_at_least(inventory: dict, minor_version: str) -> bool: + kubernetes_version = inventory["services"]["kubeadm"]["kubernetesVersion"] + return utils.version_key(kubernetes_version)[0:2] >= utils.minor_version_key(minor_version) + + +def get_init_config(cluster: KubernetesCluster, group: AbstractGroup[RunResult], *, + init: bool, join_dict: dict = None) -> dict: + inventory = cluster.inventory + + if join_dict is None: + join_dict = {} + + init_kind = 'InitConfiguration' if init else 'JoinConfiguration' + + control_plane_spec = {} + if group.nodes_amount() > 1: + control_planes = group.new_group(lambda node: 'control-plane' in node['roles']) + if not control_planes.is_empty(): + raise Exception("Init/Join configuration for control planes should be unique") + + control_plane = False + worker = True + else: + node_config = group.get_config() + control_plane = 'control-plane' in node_config['roles'] + worker = 'worker' in node_config['roles'] + if control_plane: + control_plane_spec = {'localAPIEndpoint': { + 'advertiseAddress': node_config['internal_address'] + }} + + init_config: dict = { + 'apiVersion': inventory["services"]["kubeadm"]['apiVersion'], + 'kind': init_kind, + 'patches': {'directory': '/etc/kubernetes/patches'}, + } + if init: + if control_plane: + init_config.update(control_plane_spec) + else: + if control_plane: + control_plane_spec['certificateKey'] = join_dict['certificate-key'] + init_config['controlPlane'] = control_plane_spec + + init_config['discovery'] = { + 'bootstrapToken': { + 'apiServerEndpoint': inventory["services"]["kubeadm"]['controlPlaneEndpoint'], + 'token': join_dict['token'], + 'caCertHashes': [ + join_dict['discovery-token-ca-cert-hash'] + ] + } + } + + if inventory['services']['kubeadm']['controllerManager']['extraArgs'].get('external-cloud-volume-plugin'): + init_config['nodeRegistration'] = { + 'kubeletExtraArgs': { + 'cloud-provider': 'external' + } + } + + if control_plane and worker: + init_config.setdefault('nodeRegistration', {})['taints'] = [] + + _configure_container_runtime(cluster, init_config) + + return init_config + + +def get_kubeadm_config(cluster: KubernetesCluster, init_config: dict) -> str: + return KubeadmConfig(cluster).to_yaml(init_config) + + +def _configure_container_runtime(cluster: KubernetesCluster, kubeadm_config: dict) -> None: + if cluster.inventory['services']['cri']['containerRuntime'] == "containerd": + kubelet_extra_args = kubeadm_config.setdefault('nodeRegistration', {}).setdefault('kubeletExtraArgs', {}) + + kubeadm_config['nodeRegistration']['criSocket'] = '/var/run/containerd/containerd.sock' + + if not is_container_runtime_not_configurable(cluster): + kubelet_extra_args['container-runtime'] = 'remote' + + kubelet_extra_args['container-runtime-endpoint'] = 'unix:///run/containerd/containerd.sock' + + +def reconfigure_components(group: NodeGroup, components: List[str], + *, + edit_functions: Dict[str, Callable[[dict], dict]] = None, + force_restart: bool = False) -> None: + """ + Reconfigure the specified `components` on `group` of nodes. + Control-plane nodes are reconfigured first. + The cluster is not required to be working to update control plane manifests. + + :param group: nodes to reconfigure components on + :param components: List of control plane components or `kube-proxy`, or `kubelet` to reconfigure. + :param edit_functions: Callables that edit the specified kubeadm-managed ConfigMaps in a custom way. + The ConfigMaps are fetched from the first control plane, + instead of being generated. + This implies necessity of working API server. + :param force_restart: Restart the given `components` even if nothing has changed in their configuration. + """ + not_supported = list(OrderedSet[str](components) - set(ALL_COMPONENTS)) + if not_supported: + raise Exception(ERROR_RECONFIGURE_NOT_SUPPORTED.format(components=not_supported)) + + if edit_functions is None: + edit_functions = {} + + cluster: KubernetesCluster = group.cluster + logger = cluster.log + + control_planes = (cluster.nodes['control-plane'] + .intersection_group(group)) + workers = (cluster.make_group_from_roles(['worker']) + .exclude_group(control_planes) + .intersection_group(group)) + + group = control_planes.include_group(workers) + if group.is_empty(): + logger.debug("No Kubernetes nodes to reconfigure components") + return + + first_control_plane = cluster.nodes['control-plane'].get_first_member() + if not control_planes.is_empty(): + first_control_plane = control_planes.get_first_member() + + timestamp = utils.get_current_timestamp_formatted() + backup_dir = '/etc/kubernetes/tmp/kubemarine-backup-' + timestamp + logger.debug(f"Using backup directory {backup_dir}") + + kubeadm_config = KubeadmConfig(cluster) + for configmap, func in edit_functions.items(): + kubeadm_config.load(configmap, first_control_plane, func) + + # This configuration will be used for `kubeadm init phase upload-config` phase. + # InitConfiguration is necessary to specify patches directory. + # Use patches from the first control plane, to make it the same as during regular installation. + # Patches are used to upload KubeletConfiguration for some reason (which is likely a gap) + # https://github.com/kubernetes/kubernetes/issues/123090 + upload_config = '/etc/kubernetes/upload-config.yaml' + upload_config_uploaded = False + + def prepare_upload_config() -> None: + nonlocal upload_config_uploaded + if upload_config_uploaded: + return + + logger.debug(f"Uploading cluster config to control plane: {first_control_plane.get_node_name()}") + _upload_config(cluster, first_control_plane, kubeadm_config, upload_config) + + upload_config_uploaded = True + + # This configuration will be generated for control plane nodes, + # and will be used to `kubeadm init phase control-plane / etcd / certs`. + reconfigure_config = '/etc/kubernetes/reconfigure-config.yaml' + + _prepare_nodes_to_reconfigure_components(cluster, group, components, + kubeadm_config, reconfigure_config, backup_dir) + + kubeadm_config_updated = False + kubelet_config_updated = False + kube_proxy_config_updated = False + kube_proxy_changed = False + for node in (control_planes.get_ordered_members_list() + workers.get_ordered_members_list()): + _components = _choose_components(node, components) + if not _components: + continue + + # Firstly reconfigure components that do not require working API server + control_plane_components = list(OrderedSet[str](_components) & set(CONTROL_PLANE_SPECIFIC_COMPONENTS)) + if control_plane_components: + _reconfigure_control_plane_components(cluster, node, control_plane_components, force_restart, + reconfigure_config, backup_dir) + + # Upload kubeadm-config after control plane components are successfully reconfigured on the first node. + if not kubeadm_config_updated: + prepare_upload_config() + _update_configmap(cluster, first_control_plane, 'kubeadm-config', + _configmap_init_phase_uploader('kubeadm-config', upload_config), + backup_dir) + kubeadm_config_updated = True + + node_components = list(OrderedSet[str](_components) & set(NODE_COMPONENTS)) + if node_components: + if 'kubelet' in node_components and not kubelet_config_updated: + prepare_upload_config() + _update_configmap(cluster, first_control_plane, 'kubelet-config', + _configmap_init_phase_uploader('kubelet-config', upload_config), + backup_dir) + + kubelet_config_updated = True + + if 'kube-proxy' in node_components and not kube_proxy_config_updated: + kube_proxy_changed = _update_configmap(cluster, first_control_plane, 'kube-proxy', + _kube_proxy_configmap_uploader(cluster, kubeadm_config), + backup_dir) + + kube_proxy_config_updated = True + + _reconfigure_node_components(cluster, node, node_components, force_restart, first_control_plane, + kube_proxy_changed, backup_dir) + + +def restart_components(group: NodeGroup, components: List[str]) -> None: + """ + Currently it is supported to restart only + 'kube-apiserver', 'kube-scheduler', 'kube-controller-manager', 'etcd'. + + :param group: nodes to restart components on + :param components: Kubernetes components to restart + """ + not_supported = list(OrderedSet[str](components) - set(CONTROL_PLANE_COMPONENTS)) + if not_supported: + raise Exception(ERROR_RESTART_NOT_SUPPORTED.format(components=not_supported)) + + cluster: KubernetesCluster = group.cluster + + for node in group.get_ordered_members_list(): + _components = _choose_components(node, components) + _restart_containers(cluster, node, _components) + wait_for_pods(node, _components) + + +def wait_for_pods(group: NodeGroup, components: Sequence[str] = None) -> None: + """ + Wait for pods of Kubernetes components on the given `group` of nodes. + All relevant components are waited for unless specific list of `components` is given. + For nodes that are not control planes, only 'kube-proxy' can be waited for. + + :param group: nodes to wait pods on + :param components: Kubernetes components to wait for. + """ + if components is not None: + if not components: + return + + not_supported = list(OrderedSet[str](components) - set(CONTROL_PLANE_COMPONENTS) - {'kube-proxy'}) + if not_supported: + raise Exception(ERROR_WAIT_FOR_PODS_NOT_SUPPORTED.format(components=not_supported)) + + cluster: KubernetesCluster = group.cluster + first_control_plane = cluster.nodes['control-plane'].get_first_member() + expect_config = cluster.inventory['globals']['expect']['pods']['kubernetes'] + + for node in group.get_ordered_members_list(): + node_name = node.get_node_name() + is_control_plane = 'control-plane' in node.get_config()['roles'] + cluster.log.debug(f"Waiting for system pods on node: {node_name}") + + if components is not None: + _components = list(components) + else: + _components = ['kube-proxy'] + if is_control_plane: + _components.extend(CONTROL_PLANE_COMPONENTS) + + _components = _choose_components(node, _components) + if not _components: + continue + + control_plane = node + if not is_control_plane: + control_plane = first_control_plane + + plugins.expect_pods(cluster, _components, namespace='kube-system', + control_plane=control_plane, node_name=node_name, + timeout=expect_config['timeout'], + retries=expect_config['retries']) + + +# function to create kubeadm patches and put them to a node +def create_kubeadm_patches_for_node(cluster: KubernetesCluster, node: NodeGroup) -> None: + cluster.log.verbose(f"Create and upload kubeadm patches to %s..." % node.get_node_name()) + node.sudo("mkdir -p /etc/kubernetes/patches") + defer = node.new_defer() + for component in COMPONENTS_SUPPORT_PATCHES: + _create_kubeadm_patches_for_component_on_node(cluster, defer, component) + + defer.flush() + + +def _upload_config(cluster: KubernetesCluster, control_plane: AbstractGroup[RunResult], + kubeadm_config: KubeadmConfig, remote_path: str, + *, patches_dir: str = '/etc/kubernetes/patches') -> None: + name = remote_path.rstrip('.yaml').split('/')[-1] + + init_config = get_init_config(cluster, control_plane, init=True) + init_config['patches']['directory'] = patches_dir + config = kubeadm_config.to_yaml(init_config) + utils.dump_file(cluster, config, f"{name}_{control_plane.get_node_name()}.yaml") + + control_plane.put(io.StringIO(config), remote_path, sudo=True) + + +def _update_configmap(cluster: KubernetesCluster, control_plane: NodeGroup, configmap: str, + uploader: Callable[[DeferredGroup], None], backup_dir: str) -> bool: + logger = cluster.log + + logger.debug(f"Updating {configmap} ConfigMap") + defer = control_plane.new_defer() + collector = CollectorCallback(cluster) + + configmap_name = _get_configmap_name(cluster, configmap) + + key = CONFIGMAPS_CONSTANTS[configmap]['key'].replace('.', r'\.') + configmap_cmd = f'sudo kubectl get cm -n kube-system {configmap_name} -o=jsonpath="{{.data.{key}}}"' + + # backup + defer.run(f'sudo mkdir -p {backup_dir}') + backup_file = f'{backup_dir}/{configmap_name}.yaml' + defer.run(f'(set -o pipefail && {configmap_cmd} | sudo tee {backup_file}) > /dev/null') + + # update + uploader(defer) + + # compare + defer.run(f'sudo cat {backup_file}', callback=collector) + defer.run(configmap_cmd, callback=collector) + + defer.flush() + + results = collector.results[defer.get_host()] + return _detect_changes(logger, results[0].stdout, results[1].stdout, + fromfile=backup_file, tofile=f'{configmap_name} ConfigMap') + + +def _get_configmap_name(cluster: KubernetesCluster, configmap: str) -> str: + configmap_name = configmap + if configmap == 'kubelet-config' and not kubelet_config_unversioned(cluster): + kubernetes_version = cluster.inventory["services"]["kubeadm"]["kubernetesVersion"] + configmap_name += '-' + utils.minor_version(kubernetes_version)[1:] + + return configmap_name + + +def _configmap_init_phase_uploader(configmap: str, upload_config: str) -> Callable[[DeferredGroup], None]: + def upload(control_plane: DeferredGroup) -> None: + init_phase = CONFIGMAPS_CONSTANTS[configmap]['init_phase'] + control_plane.run(f'sudo kubeadm init phase {init_phase} --config {upload_config}') + + return upload + + +def _kube_proxy_configmap_uploader(cluster: KubernetesCluster, kubeadm_config: KubeadmConfig) \ + -> Callable[[DeferredGroup], None]: + + # Unfortunately, there is no suitable kubeadm command to upload the generated ConfigMap. + # This makes it impossible to reset some property to default by deleting it from `services.kubeadm_kube-proxy`. + + def upload(control_plane_deferred: DeferredGroup) -> None: + control_plane_deferred.flush() + control_plane = cluster.make_group(control_plane_deferred.get_hosts()) + + # reconfigure_components() can be called with custom editing function. + # The ConfigMap is already fetched and changed. + if not kubeadm_config.is_loaded('kube-proxy'): + kubeadm_config.load('kube-proxy', control_plane, kubeadm_config.merge_with_inventory('kube-proxy')) + + # Apply updated kube-proxy ConfigMap + kubeadm_config.apply('kube-proxy', control_plane) + + return upload + + +def _choose_components(node: AbstractGroup[RunResult], components: List[str]) -> List[str]: + roles = node.get_config()['roles'] + + return [c for c in components if c in NODE_COMPONENTS and set(roles) & {'control-plane', 'worker'} + or 'control-plane' in roles and c in CONTROL_PLANE_SPECIFIC_COMPONENTS] + + +def _prepare_nodes_to_reconfigure_components(cluster: KubernetesCluster, group: NodeGroup, components: List[str], + kubeadm_config: KubeadmConfig, + reconfigure_config: str, backup_dir: str) -> None: + logger = cluster.log + defer = group.new_defer() + for node in defer.get_ordered_members_list(): + _components = _choose_components(node, components) + if not _components: + continue + + if set(_components) & set(CONTROL_PLANE_SPECIFIC_COMPONENTS): + logger.debug(f"Uploading config for control plane components on node: {node.get_node_name()}") + _upload_config(cluster, node, kubeadm_config, reconfigure_config) + + if set(_components) & set(COMPONENTS_SUPPORT_PATCHES): + node.sudo("mkdir -p /etc/kubernetes/patches") + node.sudo(f'mkdir -p {backup_dir}/patches') + + if set(_components) & set(CONTROL_PLANE_COMPONENTS): + node.sudo(f'mkdir -p {backup_dir}/manifests') + + for component in _components: + if component not in COMPONENTS_SUPPORT_PATCHES: + continue + + _create_kubeadm_patches_for_component_on_node(cluster, node, component, backup_dir) + + defer.flush() + + +def _reconfigure_control_plane_components(cluster: KubernetesCluster, node: NodeGroup, components: List[str], + force_restart: bool, + reconfigure_config: str, backup_dir: str) -> None: + logger = cluster.log + logger.debug(f"Reconfiguring control plane components {components} on node: {node.get_node_name()}") + + defer = node.new_defer() + + containers_restart = OrderedSet[str]() + pods_wait = OrderedSet[str]() + for component in components: + if component == 'kube-apiserver/cert-sans': + _reconfigure_apiserver_certsans(defer, reconfigure_config, backup_dir) + containers_restart.add('kube-apiserver') + pods_wait.add('kube-apiserver') + continue + + # Let's anyway wait for pods as the component may be broken due to previous runs. + pods_wait.add(component) + if (_reconfigure_control_plane_component(cluster, defer, component, reconfigure_config, backup_dir) + or force_restart): + containers_restart.add(component) + else: + # Manifest file may be changed in formatting but not in meaningful content. + # Kubelet is not observed to restart the container in this case, neither will we. + pass + + defer.flush() + _restart_containers(cluster, node, containers_restart) + wait_for_pods(node, pods_wait) + + +def _reconfigure_apiserver_certsans(node: DeferredGroup, reconfigure_config: str, backup_dir: str) -> None: + apiserver_certs = r'find /etc/kubernetes/pki/ -name apiserver.\*' + + # backup + node.sudo(f'mkdir -p {backup_dir}/pki') + node.sudo(f'{apiserver_certs} -exec cp {{}} {backup_dir}/pki \\;') + + # create cert + node.sudo(f'{apiserver_certs} -delete') + init_phase = COMPONENTS_CONSTANTS['kube-apiserver/cert-sans']['init_phase'] + node.sudo(f'kubeadm init phase {init_phase} --config {reconfigure_config}') + + +def _reconfigure_control_plane_component(cluster: KubernetesCluster, node: DeferredGroup, component: str, + reconfigure_config: str, backup_dir: str) -> bool: + manifest = f'/etc/kubernetes/manifests/{component}.yaml' + backup_file = f'{backup_dir}/manifests/{component}.yaml' + collector = CollectorCallback(cluster) + + # backup + node.sudo(f"cp {manifest} {backup_file}") + + # update + init_phase = COMPONENTS_CONSTANTS[component]['init_phase'] + node.sudo(f'kubeadm init phase {init_phase} --config {reconfigure_config}') + + # compare + node.sudo(f'cat {backup_file}', callback=collector) + node.sudo(f'cat {manifest}', callback=collector) + + node.flush() + + results = collector.results[node.get_host()] + return _detect_changes(cluster.log, results[0].stdout, results[1].stdout, + fromfile=backup_file, tofile=manifest) + + +def _reconfigure_node_components(cluster: KubernetesCluster, node: NodeGroup, components: List[str], + force_restart: bool, control_plane: NodeGroup, + kube_proxy_changed: bool, backup_dir: str) -> None: + logger = cluster.log + + is_control_plane = 'control-plane' in node.get_config()['roles'] + kube_proxy_restart = kube_proxy_changed or force_restart + containers_restart = OrderedSet[str]() + pods_wait = OrderedSet[str]() + if 'kube-proxy' in components: + pods_wait.add('kube-proxy') + + if 'kubelet' in components: + logger.debug(f"Reconfiguring kubelet on node: {node.get_node_name()}") + + pods_wait.add('kube-proxy') + if is_control_plane: + pods_wait.update(CONTROL_PLANE_COMPONENTS) + + defer = node.new_defer() + if _reconfigure_kubelet(cluster, defer, backup_dir) or force_restart: + system.restart_service(defer, 'kubelet') + # It is not clear how to check that kubelet is healthy. + # Let's restart and check health of all components. + kube_proxy_restart = True + if is_control_plane: + # No need to manually kill container for kube-proxy. It is restarted as soon as pod is deleted. + containers_restart.update(CONTROL_PLANE_COMPONENTS) + + defer.flush() + + # Delete pod for 'kube-proxy' early while 'kube-apiserver' is expected to be available. + if kube_proxy_restart: + _delete_pods(cluster, node, control_plane, ['kube-proxy']) + + _restart_containers(cluster, node, containers_restart) + wait_for_pods(node, pods_wait) + + +def _reconfigure_kubelet(cluster: KubernetesCluster, node: DeferredGroup, + backup_dir: str) -> bool: + config = '/var/lib/kubelet/config.yaml' + backup_file = f'{backup_dir}/kubelet/config.yaml' + collector = CollectorCallback(cluster) + + # backup + node.sudo(f'mkdir -p {backup_dir}/kubelet') + node.sudo(f"cp {config} {backup_file}") + + # update + patches_flag = ' --patches=/etc/kubernetes/patches' if kubelet_supports_patches(cluster) else '' + node.sudo(f'kubeadm upgrade node phase kubelet-config{patches_flag}') + + # compare + node.sudo(f'cat {backup_file}', callback=collector) + node.sudo(f'cat {config}', callback=collector) + + node.flush() + + results = collector.results[node.get_host()] + return _detect_changes(cluster.log, results[0].stdout, results[1].stdout, + fromfile=backup_file, tofile=config) + + +def compare_manifests(cluster: KubernetesCluster, *, with_inventory: bool) \ + -> Dict[str, Dict[str, Optional[str]]]: + """ + Generate manifests in dry-run mode for all control plane components on all control plane nodes, + and compare with already present manifests. + + :param cluster: KubernetesCluster instance + :param with_inventory: flag if cluster configuration should be generated from the inventory + :return: mapping host -> component -> diff string + """ + kubeadm_config = KubeadmConfig(cluster) + if not with_inventory: + kubeadm_config.load('kubeadm-config', cluster.nodes['control-plane'].get_first_member()) + + control_planes = cluster.nodes['control-plane'].new_defer() + temp_config = "/tmp/%s" % uuid.uuid4().hex + patches_dir = '/etc/kubernetes/patches' + if with_inventory: + patches_dir = "/tmp/%s" % uuid.uuid4().hex + + components = [c for c in CONTROL_PLANE_COMPONENTS + if c != 'etcd' or kubeadm_extended_dryrun(cluster)] + + tmp_dirs_cmd = "sh -c 'sudo ls /etc/kubernetes/tmp/ | grep dryrun 2>/dev/null || true'" + old_tmp_dirs = CollectorCallback(cluster) + new_tmp_dirs = CollectorCallback(cluster) + for defer in control_planes.get_ordered_members_list(): + _upload_config(cluster, defer, kubeadm_config, temp_config, patches_dir=patches_dir) + if with_inventory: + defer.sudo(f'mkdir -p {patches_dir}') + + for component in components: + if with_inventory: + _create_kubeadm_patches_for_component_on_node(cluster, defer, component, + patches_dir=patches_dir, reset=False) + + defer.sudo(tmp_dirs_cmd, callback=old_tmp_dirs) + + init_phase = COMPONENTS_CONSTANTS[component]['init_phase'] + defer.sudo(f'kubeadm init phase {init_phase} --dry-run --config {temp_config}') + + defer.sudo(tmp_dirs_cmd, callback=new_tmp_dirs) + + control_planes.flush() + + stored_manifest = CollectorCallback(cluster) + generated_manifest = CollectorCallback(cluster) + for defer in control_planes.get_ordered_members_list(): + old_tmp_dirs_results = old_tmp_dirs.results[defer.get_host()] + new_tmp_dirs_results = new_tmp_dirs.results[defer.get_host()] + for i, component in enumerate(components): + tmp_dir = next(iter( + set(new_tmp_dirs_results[i].stdout.split()) + - set(old_tmp_dirs_results[i].stdout.split()) + )) + defer.sudo(f'cat /etc/kubernetes/manifests/{component}.yaml', callback=stored_manifest) + defer.sudo(f'cat /etc/kubernetes/tmp/{tmp_dir}/{component}.yaml', callback=generated_manifest) + + control_planes.flush() + + result: Dict[str, Dict[str, Optional[str]]] = {} + for host in control_planes.get_hosts(): + stored_manifest_results = stored_manifest.results[host] + generated_manifest_results = generated_manifest.results[host] + for i, component in enumerate(components): + tofile = (f"{component}.yaml generated from 'services.kubeadm' section" + if with_inventory + else f"{component}.yaml generated from kubeadm-config ConfigMap") + stored = stored_manifest_results[i].stdout + generated = generated_manifest_results[i].stdout + if component == 'etcd': + stored = _filter_etcd_initial_cluster_args(stored) + generated = _filter_etcd_initial_cluster_args(generated) + + diff = utils.get_yaml_diff(stored, generated, + fromfile=f'/etc/kubernetes/manifests/{component}.yaml', + tofile=tofile) + + result.setdefault(host, {})[component] = diff + + return result + + +def compare_kubelet_config(cluster: KubernetesCluster, *, with_inventory: bool) \ + -> Dict[str, Optional[str]]: + """ + Generate /var/lib/kubelet/config.yaml in dry-run mode on all nodes, + and compare with already present configurations. + + :param cluster: KubernetesCluster instance + :param with_inventory: flag if patches should be taken from the inventory + :return: mapping host -> diff string + """ + nodes = cluster.make_group_from_roles(['control-plane', 'worker']).new_defer() + patches_dir = '/etc/kubernetes/patches' + if with_inventory: + patches_dir = "/tmp/%s" % uuid.uuid4().hex + + tmp_dirs_cmd = "sh -c 'sudo ls /etc/kubernetes/tmp/ | grep dryrun 2>/dev/null || true'" + old_tmp_dirs = CollectorCallback(cluster) + new_tmp_dirs = CollectorCallback(cluster) + for defer in nodes.get_ordered_members_list(): + if with_inventory and kubelet_supports_patches(cluster): + defer.sudo(f'mkdir -p {patches_dir}') + _create_kubeadm_patches_for_component_on_node( + cluster, defer, 'kubelet', patches_dir=patches_dir, reset=False) + + defer.sudo(tmp_dirs_cmd, callback=old_tmp_dirs) + + patches_flag = f' --patches={patches_dir}' if kubelet_supports_patches(cluster) else '' + defer.sudo(f'kubeadm upgrade node phase kubelet-config --dry-run{patches_flag}') + + defer.sudo(tmp_dirs_cmd, callback=new_tmp_dirs) + + nodes.flush() + + stored_config = CollectorCallback(cluster) + generated_config = CollectorCallback(cluster) + for defer in nodes.get_ordered_members_list(): + old_tmp_dirs_results = old_tmp_dirs.results[defer.get_host()] + new_tmp_dirs_results = new_tmp_dirs.results[defer.get_host()] + tmp_dir = next(iter( + set(new_tmp_dirs_results[0].stdout.split()) + - set(old_tmp_dirs_results[0].stdout.split()) + )) + defer.sudo(f'cat /var/lib/kubelet/config.yaml', callback=stored_config) + defer.sudo(f'cat /etc/kubernetes/tmp/{tmp_dir}/config.yaml', callback=generated_config) + + nodes.flush() + + result = {} + for host in nodes.get_hosts(): + tofile = (f"config.yaml with patches from inventory" + if with_inventory + else f"config.yaml generated from kubelet-config ConfigMap") + stored = stored_config.results[host][0].stdout + generated = generated_config.results[host][0].stdout + + diff = utils.get_yaml_diff(stored, generated, + fromfile='/var/lib/kubelet/config.yaml', + tofile=tofile) + + result[host] = diff + + return result + + +def compare_configmap(cluster: KubernetesCluster, configmap: str) -> Optional[str]: + control_plane = cluster.nodes['control-plane'].get_first_member() + kubeadm_config = KubeadmConfig(cluster) + + if configmap == 'kubelet-config': + # Do not check kubelet-config ConfigMap, because some properties may be deleted from KubeletConfiguration + # if set to default, for example readOnlyPort: 0, protectKernelDefaults: false + # Otherwise, the check would require to take into account all such default properties. + if not kubeadm_extended_dryrun(cluster): + return None + + # Use upload-config kubelet --dry-run to catch all inserted/updated/deleted properties. + + temp_config = "/tmp/%s" % uuid.uuid4().hex + patches_dir = "/tmp/%s" % uuid.uuid4().hex + + defer = control_plane.new_defer() + collector = CollectorCallback(cluster) + + _upload_config(cluster, defer, kubeadm_config, temp_config, patches_dir=patches_dir) + defer.sudo(f'mkdir -p {patches_dir}') + _create_kubeadm_patches_for_component_on_node( + cluster, defer, 'kubelet', patches_dir=patches_dir, reset=False) + + init_phase = CONFIGMAPS_CONSTANTS[configmap]['init_phase'] + defer.sudo(f'kubeadm init phase {init_phase} --dry-run --config {temp_config}', + callback=collector) + + defer.flush() + output = collector.result[control_plane.get_host()].stdout + + split_logs = re.compile(r'^\[.*].*$\n', flags=re.M) + cfg = next(filter(lambda ln: 'kind: KubeletConfiguration' in ln, split_logs.split(output))) + cfg = dedent(cfg) + + key = CONFIGMAPS_CONSTANTS[configmap]['key'] + generated_config = yaml.safe_load(cfg)['data'][key] + + kubeadm_config.load(configmap, control_plane) + stored_config = kubeadm_config.loaded_maps[configmap].obj["data"][key] + + if yaml.safe_load(generated_config) == yaml.safe_load(stored_config): + return None + + return utils.get_unified_diff(stored_config, generated_config, + fromfile=f'{configmap} ConfigMap', + tofile="generated from 'services.kubeadm_kubelet' section") + + else: + # Merge with inventory and check. + # This way it is possible to check only new or changed properties in the inventory + # that are still not reflected in the remote ConfigMap. + + stored_config = kubeadm_config.load(configmap, control_plane) + + generated_config = kubeadm_config.merge_with_inventory(configmap)\ + (deepcopy(stored_config)) + + if generated_config == stored_config: + return None + + section = CONFIGMAPS_CONSTANTS[configmap]['section'] + return utils.get_unified_diff(yaml.dump(stored_config), yaml.dump(generated_config), + fromfile=f'{configmap} ConfigMap', + tofile=f"{configmap} ConfigMap merged 'services.{section}' section") + + +def _detect_changes(logger: log.EnhancedLogger, old: str, new: str, fromfile: str, tofile: str) -> bool: + diff = utils.get_yaml_diff(old, new, fromfile, tofile) + if diff is not None: + logger.debug(f"Detected changes in {tofile}") + logger.verbose(diff) + return True + + return False + + +def _filter_etcd_initial_cluster_args(content: str) -> str: + return '\n'.join(filter(lambda ln: '--initial-cluster' not in ln, content.splitlines())) + + +def _restart_containers(cluster: KubernetesCluster, node: NodeGroup, components: Sequence[str]) -> None: + if not components: + return + + logger = cluster.log + node_name = node.get_node_name() + logger.debug(f"Restarting containers for components {list(components)} on node: {node_name}") + + commands = [] + + cri_impl = cluster.inventory['services']['cri']['containerRuntime'] + # Take into account probably missed container because kubelet may be restarting them at this moment. + # Though still ensure the command to delete the container successfully if it is present. + if cri_impl == 'containerd': + restart_container = ("(set -o pipefail && sudo crictl ps --name {component} -q " + "| xargs -I CONTAINER sudo crictl rm -f CONTAINER)") + else: + restart_container = ("(set -o pipefail && sudo docker ps -q -f 'name=k8s_{component}' " + "| xargs -I CONTAINER sudo docker rm -f CONTAINER)") + + for component in components: + commands.append(restart_container.format(component=component)) + + if cri_impl == 'containerd': + get_container_from_cri = "sudo crictl ps --name {component} -q" + else: + get_container_from_cri = ( + "sudo docker ps --no-trunc -f 'name=k8s_{component}' " + "| grep k8s_{component} | awk '{{{{ print $1 }}}}'") + + get_container_from_pod = ( + "sudo kubectl get pods -n kube-system {component}-{node} " + "-o 'jsonpath={{.status.containerStatuses[0].containerID}}{{\"\\n\"}}' " + "| sed 's|.\\+://\\(.\\+\\)|\\1|'") + + # Wait for kubelet to refresh container status in pods. + # It is expected that Ready status will be refreshed at the same time, + # so we can safely wait_for_pods(). + test_refreshed_container = ( + f"(" + f"CONTAINER=$({get_container_from_cri}); " + f"if [ -z \"$CONTAINER\" ]; then " + f" echo \"container '{{component}}' is not created yet\" >&2 ; exit 1; " + f"fi " + f"&& " + f"if [ \"$CONTAINER\" != \"$({get_container_from_pod})\" ]; " + f" then echo \"Pod '{{component}}-{{node}}' is not refreshed yet\" >&2; exit 1; " + f"fi " + f")") + + for component in components: + commands.append(test_refreshed_container.format(component=component, node=node_name)) + + expect_config = cluster.inventory['globals']['expect']['pods']['kubernetes'] + node.wait_commands_successful(commands, + timeout=expect_config['timeout'], + retries=expect_config['retries'], + sudo=False) + + +def _delete_pods(cluster: KubernetesCluster, node: AbstractGroup[RunResult], + control_plane: NodeGroup, components: Sequence[str]) -> None: + if not components: + return + + node_name = node.get_node_name() + cluster.log.debug(f"Deleting pods for components {list(components)} on node: {node_name}") + + restart_pod = ( + "kubectl delete pod -n kube-system $(" + " sudo kubectl get pods -n kube-system -o=wide " + " | grep '{pod}' | grep '{node}' | awk '{{ print $1 }}'" + ")" + ) + + defer = control_plane.new_defer() + for component in components: + defer.sudo(restart_pod.format(pod=component, node=node_name)) + + defer.flush() + + +# function to get dictionary of flags to be patched for a given control plane item and a given node +def _get_patched_flags_for_section(inventory: dict, patch_section: str, node: NodeConfig) -> Dict[str, str]: + flags = {} + + for n in inventory['services']['kubeadm_patches'][patch_section]: + if n.get('groups') is not None and list(set(node['roles']) & set(n['groups'])): + for arg, value in n['patch'].items(): + flags[arg] = value + if n.get('nodes') is not None and node['name'] in n['nodes']: + for arg, value in n['patch'].items(): + flags[arg] = value + + # we always set binding-address to the node's internal address for apiServer + if patch_section == 'apiServer' and 'control-plane' in node['roles']: + flags['bind-address'] = node['internal_address'] + + return flags + + +def _create_kubeadm_patches_for_component_on_node(cluster: KubernetesCluster, node: DeferredGroup, component: str, + backup_dir: Optional[str] = None, + *, patches_dir: str = '/etc/kubernetes/patches', reset: bool = True) -> None: + patch_constants = COMPONENTS_CONSTANTS[component]['patch'] + component_patches = f"find {patches_dir} -name {patch_constants['target_template']}" + + if backup_dir is not None: + node.sudo(f'{component_patches} -exec cp {{}} {backup_dir}/patches \\;') + + if reset: + node.sudo(f'{component_patches} -delete') + + # read patch content from inventory and upload patch files to a node + node_config = node.get_config() + patched_flags = _get_patched_flags_for_section(cluster.inventory, patch_constants['section'], node_config) + if patched_flags: + if component == 'kubelet': + template_filename = 'templates/patches/kubelet.yaml.j2' + else: + template_filename = 'templates/patches/control-plane-pod.json.j2' + + control_plane_patch = Template(utils.read_internal(template_filename)).render(flags=patched_flags) + patch_file = patches_dir + '/' + patch_constants['file'] + node.put(io.StringIO(control_plane_patch + "\n"), patch_file, sudo=True) + node.sudo(f'chmod 644 {patch_file}') diff --git a/kubemarine/plugins/__init__.py b/kubemarine/plugins/__init__.py index f7b8cf770..415991800 100755 --- a/kubemarine/plugins/__init__.py +++ b/kubemarine/plugins/__init__.py @@ -48,6 +48,8 @@ oob_plugins = list(static.DEFAULTS["plugins"].keys()) LOADED_MODULES: Dict[str, ModuleType] = {} +ERROR_PODS_NOT_READY = 'In the expected time, the pods did not become ready' + def verify_inventory(inventory: dict, cluster: KubernetesCluster) -> dict: for plugin_name, plugin_item in inventory["plugins"].items(): @@ -432,7 +434,7 @@ def expect_deployment(cluster: KubernetesCluster, def expect_pods(cluster: KubernetesCluster, pods: List[str], namespace: str = None, timeout: int = None, retries: int = None, - node: NodeGroup = None, apply_filter: str = None) -> None: + control_plane: NodeGroup = None, node_name: str = None) -> None: if timeout is None: timeout = cluster.inventory['globals']['expect']['pods']['plugins']['timeout'] @@ -446,20 +448,20 @@ def expect_pods(cluster: KubernetesCluster, pods: List[str], namespace: str = No failures = 0 - if node is None: - node = cluster.nodes['control-plane'].get_first_member() + if control_plane is None: + control_plane = cluster.nodes['control-plane'].get_first_member() namespace_filter = '-A' if namespace is not None: namespace_filter = "-n " + namespace command = f"kubectl get pods {namespace_filter} -o=wide" - if apply_filter is not None: - command += ' | grep %s' % apply_filter + if node_name is not None: + command += ' | grep %s' % node_name while retries > 0: - result = node.sudo(command, warn=True) + result = control_plane.sudo(command, warn=True) stdout = list(result.values())[0].stdout running_pods_stdout = '' @@ -468,15 +470,12 @@ def expect_pods(cluster: KubernetesCluster, pods: List[str], namespace: str = No for stdout_line in iter(stdout.splitlines()): - stdout_line_allowed = False - # is current line has requested pod for verification? # we do not have to fail on pods with bad status which was not requested for pod in pods: - if pod + "-" in stdout_line: - stdout_line_allowed = True + if pod + "-" not in stdout_line: + continue - if stdout_line_allowed: if is_critical_state_in_stdout(cluster, stdout_line): cluster.log.verbose("Failed pod detected: %s\n" % stdout_line) @@ -488,9 +487,11 @@ def expect_pods(cluster: KubernetesCluster, pods: List[str], namespace: str = No if failures > cluster.globals['pods']['allowed_failures']: raise Exception('Pod entered a state of error, further proceeding is impossible') else: - # we have to take into account any pod in not a critical state + # we have to take into account any pod in not a critical state running_pods_stdout += stdout_line + '\n' + break + pods_ready = False if running_pods_stdout and running_pods_stdout != "" and "0/1" not in running_pods_stdout: pods_ready = True @@ -510,7 +511,7 @@ def expect_pods(cluster: KubernetesCluster, pods: List[str], namespace: str = No cluster.log.debug(running_pods_stdout) time.sleep(timeout) - raise Exception('In the expected time, the pods did not become ready') + raise Exception(ERROR_PODS_NOT_READY) def is_critical_state_in_stdout(cluster: KubernetesCluster, stdout: str) -> bool: diff --git a/kubemarine/plugins/calico.py b/kubemarine/plugins/calico.py index 60ab3a4c3..894374c5d 100755 --- a/kubemarine/plugins/calico.py +++ b/kubemarine/plugins/calico.py @@ -20,6 +20,7 @@ from kubemarine import plugins, kubernetes from kubemarine.core import utils, log from kubemarine.core.cluster import KubernetesCluster +from kubemarine.core.group import NodeGroup from kubemarine.kubernetes import secrets from kubemarine.plugins.manifest import Processor, EnrichmentFunction, Manifest, Identity @@ -124,7 +125,7 @@ def renew_apiserver_certificate(cluster: KubernetesCluster) -> None: # Try to access some projectcalico.org resource using each instance of the Kubernetes API server. expect_config = cluster.inventory['plugins']['calico']['apiserver']['expect']['apiservice'] with kubernetes.local_admin_config(control_planes) as kubeconfig: - control_planes.call(utils.wait_command_successful, + control_planes.call(NodeGroup.wait_command_successful, command=f"kubectl --kubeconfig {kubeconfig} get ippools.projectcalico.org", hide=True, retries=expect_config['retries'], timeout=expect_config['timeout']) diff --git a/kubemarine/procedures/add_node.py b/kubemarine/procedures/add_node.py index d135c1cf5..3f0d00da0 100755 --- a/kubemarine/procedures/add_node.py +++ b/kubemarine/procedures/add_node.py @@ -22,6 +22,7 @@ from kubemarine.core.action import Action from kubemarine.core.cluster import KubernetesCluster from kubemarine.core.resources import DynamicResources +from kubemarine.kubernetes import components from kubemarine.plugins.nginx_ingress import redeploy_ingress_nginx_is_needed from kubemarine.procedures import install @@ -31,6 +32,8 @@ def deploy_kubernetes_join(cluster: KubernetesCluster) -> None: group = cluster.make_group_from_roles(['control-plane', 'worker']).get_new_nodes() if group.is_empty(): + # If balancers are added, it is necessary to reconfigure apiServer.certSANs + write_new_apiserver_certsans(cluster) cluster.log.debug("No kubernetes nodes to perform") return @@ -40,6 +43,8 @@ def deploy_kubernetes_join(cluster: KubernetesCluster) -> None: cluster.nodes['worker'].get_new_nodes().exclude_group(cluster.nodes['control-plane']) \ .call(kubernetes.init_workers) + write_new_apiserver_certsans(cluster) + group.call_batch([ kubernetes.apply_labels, kubernetes.apply_taints @@ -50,6 +55,17 @@ def deploy_kubernetes_join(cluster: KubernetesCluster) -> None: kubernetes.schedule_running_nodes_report(cluster) +def write_new_apiserver_certsans(cluster: KubernetesCluster) -> None: + # If balancer or control plane is added, apiServer.certSANs are changed. + # See kubernetes.enrich_inventory() + new_nodes_require_sans = cluster.make_group_from_roles(['control-plane', 'balancer']).get_new_nodes() + if new_nodes_require_sans.is_empty(): + return + + cluster.log.debug("Write new certificates for kube-apiserver") + components.reconfigure_components(cluster.nodes['control-plane'], ['kube-apiserver/cert-sans']) + + def redeploy_plugins_if_needed(cluster: KubernetesCluster) -> None: # redeploy ingress-nginx-controller if needed if redeploy_ingress_nginx_is_needed(cluster): diff --git a/kubemarine/procedures/check_paas.py b/kubemarine/procedures/check_paas.py index ec14819c4..0eb36da90 100755 --- a/kubemarine/procedures/check_paas.py +++ b/kubemarine/procedures/check_paas.py @@ -12,7 +12,6 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -import difflib import io import sys import time @@ -26,15 +25,18 @@ import ipaddress import uuid +from ordered_set import OrderedSet + from kubemarine import ( packages as pckgs, system, selinux, etcd, thirdparties, apparmor, kubernetes, sysctl, audit, plugins, modprobe, admission ) from kubemarine.core.action import Action from kubemarine.core.cluster import KubernetesCluster -from kubemarine.core.group import NodeConfig, NodeGroup, CollectorCallback +from kubemarine.core.group import NodeGroup from kubemarine.core.resources import DynamicResources from kubemarine.cri import containerd +from kubemarine.kubernetes import components from kubemarine.plugins import calico, builtin, manifest from kubemarine.procedures import check_iaas from kubemarine.core import flow, static, utils @@ -273,6 +275,92 @@ def kubelet_version(cluster: KubernetesCluster) -> None: "versions." % cluster.inventory['services']['kubeadm']['kubernetesVersion']) +def kubelet_config(cluster: KubernetesCluster) -> None: + """ + This test checks the consistency of the /var/lib/kubelet/config.yaml configuration + with `kubelet-config` ConfigMap and with the inventory. + """ + with TestCase(cluster, '233', "Services", "Kubelet Configuration") as tc: + messages = [] + + cluster.log.debug("Checking configuration consistency with kubelet-config ConfigMap") + failed_nodes = _compare_kubelet_config(cluster, with_inventory=False) + if failed_nodes: + messages.append(f"/var/lib/kubelet/config.yaml is not consistent with kubelet-config ConfigMap " + f"on nodes {', '.join(failed_nodes)}") + + if components.kubelet_supports_patches(cluster): + cluster.log.debug("Checking configuration consistency with patches from inventory") + failed_nodes = _compare_kubelet_config(cluster, with_inventory=True) + if failed_nodes: + messages.append(f"/var/lib/kubelet/config.yaml is not consistent with patches from inventory " + f"on nodes {', '.join(failed_nodes)}") + + cluster.log.debug("Checking kubelet-config ConfigMap consistency with services.kubeadm_kubelet section") + diff = components.compare_configmap(cluster, 'kubelet-config') + if diff is not None: + msg = "kubelet-config ConfigMap is not consistent with services.kubeadm_kubelet section" + messages.append(msg) + cluster.log.debug(msg) + cluster.log.debug(diff + '\n') + + if not messages: + tc.success(results='valid') + else: + messages.append( + "Check DEBUG logs for details. To have the check passed, " + "you may need to call `kubemarine reconfigure --tasks deploy.kubernetes.reconfigure`, " + "or change the inventory file accordingly." + ) + raise TestFailure('invalid', hint=yaml.safe_dump(messages)) + + +def _compare_kubelet_config(cluster: KubernetesCluster, *, with_inventory: bool) -> List[str]: + config_diffs = components.compare_kubelet_config(cluster, with_inventory=with_inventory) + failed_nodes = OrderedSet[str]() + for host, diff in config_diffs.items(): + if diff is None: + continue + + node_name = cluster.get_node_name(host) + cluster.log.debug(f"/var/lib/kubelet/config.yaml is not actual on node {node_name}") + if failed_nodes: + cluster.log.debug('') + cluster.log.verbose(diff + '\n') + else: + cluster.log.debug(diff + '\n') + + failed_nodes.add(node_name) + + return list(failed_nodes) + + +def kube_proxy_configmap(cluster: KubernetesCluster) -> None: + """ + This test checks the consistency of the `kube-proxy` ConfigMap with the inventory. + """ + with TestCase(cluster, '234', "Services", "kube-proxy Configuration") as tc: + messages = [] + + cluster.log.debug("Checking kube-proxy ConfigMap consistency with services.kubeadm_kube-proxy section") + diff = components.compare_configmap(cluster, 'kube-proxy') + if diff is not None: + msg = "kube-proxy ConfigMap is not consistent with services.kubeadm_kube-proxy section" + messages.append(msg) + cluster.log.debug(msg) + cluster.log.debug(diff + '\n') + + if not messages: + tc.success(results='valid') + else: + messages.append( + "Check DEBUG logs for details. To have the check passed, " + "you may need to call `kubemarine reconfigure --tasks deploy.kubernetes.reconfigure`, " + "or change the inventory file accordingly." + ) + raise TestFailure('invalid', hint=yaml.safe_dump(messages)) + + def thirdparties_hashes(cluster: KubernetesCluster) -> None: """ Task which is used to verify configured thirdparties hashes agains actual hashes on nodes. @@ -643,15 +731,13 @@ def kubernetes_audit_policy_configuration(cluster: KubernetesCluster) -> None: broken.append(f"{node_name}: {audit_file_name} is absent") else: actual_config = policy_result.stdout - diff = list(difflib.unified_diff( - actual_config.splitlines(), - expected_config.splitlines(), + diff = utils.get_yaml_diff( + actual_config, expected_config, fromfile=audit_file_name, - tofile="inventory['services']['audit']['cluster_policy']", - lineterm='')) - if diff: + tofile="inventory['services']['audit']['cluster_policy']") + if diff is not None: cluster.log.debug(f"Configuration of audit policy is not actual on {node_name} node") - cluster.log.debug('\n'.join(diff)) + cluster.log.debug(diff) broken.append(f"{node_name}: {audit_file_name} is not actual") if broken: @@ -996,124 +1082,70 @@ def container_runtime_configuration_check(cluster: KubernetesCluster) -> None: def control_plane_configuration_status(cluster: KubernetesCluster) -> None: ''' - This test verifies the consistency of the configuration (image version, `extra_args`, `extra_volumes`) of static pods of Control Plain like `kube-apiserver`, `kube-controller-manager` and `kube-scheduler` + This test verifies the consistency of the configuration of static pods of Control Plain + for `kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, and `etcd`. :param cluster: KubernetesCluster object :return: None ''' with TestCase(cluster, '220', "Control plane", "configuration status") as tc: - static_pod_names = {'kube-apiserver': 'apiServer', - 'kube-controller-manager': 'controllerManager', - 'kube-scheduler': 'scheduler'} - defer = cluster.nodes['control-plane'].new_defer() - collector = CollectorCallback(cluster) - for control_plane in defer.get_ordered_members_list(): - for static_pod_name in static_pod_names: - control_plane.sudo(f'cat /etc/kubernetes/manifests/{static_pod_name}.yaml', - warn=True, callback=collector) - - defer.flush() - - message = "" - for control_plane in defer.get_ordered_members_list(): - node_config = control_plane.get_config() - node_name = control_plane.get_node_name() - for i, static_pod_name in enumerate(static_pod_names): - kubeadm_section_name = static_pod_names[static_pod_name] - static_pod_result = collector.results[control_plane.get_host()][i] - if static_pod_result.ok: - static_pod = yaml.safe_load(static_pod_result.stdout) - correct_version = check_control_plane_version(cluster, static_pod_name, static_pod, node_config) - correct_args = check_extra_args(cluster, static_pod_name, kubeadm_section_name, static_pod, node_config) - correct_volumes = check_extra_volumes(cluster, static_pod_name, kubeadm_section_name, static_pod, node_config) - if correct_version and correct_args and correct_volumes: - cluster.log.verbose( - f'Control-plane {node_name} has correct configuration for {static_pod_name}.') - else: - message += f"Control-plane {node_name} has incorrect configuration for {static_pod_name}.\n" - else: - message += f"{static_pod_name} static pod is not present on control-plane {node_name}.\n" + messages = [] + + cluster.log.debug("Checking consistency with kubeadm-config ConfigMap") + failed_nodes = _control_plane_compare_manifests(cluster, with_inventory=False) + if failed_nodes: + messages.append(f"Static pod manifests are not consistent with kubeadm-config ConfigMap " + f"on control-planes {', '.join(failed_nodes)}") + + cluster.log.debug("Checking consistency with inventory") + failed_nodes = _control_plane_compare_manifests(cluster, with_inventory=True) + if failed_nodes: + messages.append(f"Static pod manifests are not consistent with inventory " + f"on control-planes {', '.join(failed_nodes)}") - if not message: + if not messages: tc.success(results='valid') else: - message += 'Check DEBUG logs for details.' - raise TestFailure('invalid', hint=message) + messages.append( + "Check DEBUG logs for details. To have the check passed, " + "you may need to call `kubemarine reconfigure --tasks deploy.kubernetes.reconfigure`, " + "or change the inventory file accordingly." + ) + raise TestFailure('invalid', hint=yaml.safe_dump(messages)) + +def _control_plane_compare_manifests(cluster: KubernetesCluster, *, with_inventory: bool) -> List[str]: + manifest_diffs = components.compare_manifests(cluster, with_inventory=with_inventory) + failed_nodes = OrderedSet[str]() + failed_manifests = set() + for host, manifests in manifest_diffs.items(): + for manifest_, diff in manifests.items(): + if diff is None: + continue + + node_name = cluster.get_node_name(host) + cluster.log.debug(f"Static pod manifest /etc/kubernetes/manifests/{manifest_}.yaml is not actual " + f"on control-plane {node_name}") + if manifest_ in failed_manifests: + cluster.log.debug('') + cluster.log.verbose(diff + '\n') + else: + failed_manifests.add(manifest_) + cluster.log.debug(diff + '\n') + + failed_nodes.add(node_name) -def check_control_plane_version(cluster: KubernetesCluster, static_pod_name: str, static_pod: dict, - node: NodeConfig) -> bool: - version = cluster.inventory["services"]["kubeadm"]["kubernetesVersion"] - actual_image = static_pod["spec"]["containers"][0].get("image", "") - if version not in actual_image: - cluster.log.debug(f"{static_pod_name} image {actual_image} for control-plane {node['name']} " - f"does not correspond to Kubernetes {version}") - return False - - return True - - -def check_extra_args(cluster: KubernetesCluster, static_pod_name: str, kubeadm_section_name: str, - static_pod: dict, node: NodeConfig) -> bool: - result = True - for arg, value in cluster.inventory["services"]["kubeadm"][kubeadm_section_name].get("extraArgs", {}).items(): - if arg == "bind-address": - # for "bind-address" we do not take default value into account, because its patched to node internal-address - value = node["internal_address"] - original_property = f'--{arg}={value}' - properties = static_pod["spec"]["containers"][0].get("command", []) - if original_property not in properties: - original_property_str = yaml.dump({arg: value}).rstrip('\n') - cluster.log.debug(f"{original_property_str!r} is present in services.kubeadm.{kubeadm_section_name}.extraArgs, " - f"but not found in {static_pod_name} manifest for control-plane {node['name']}") - result = False - - return result - - -def check_extra_volumes(cluster: KubernetesCluster, static_pod_name: str, kubeadm_section_name: str, - static_pod: dict, node: NodeConfig) -> bool: - logger = cluster.log - result = True - section = f"services.kubeadm.{kubeadm_section_name}.extraVolumes" - for original_volume in cluster.inventory["services"]["kubeadm"][kubeadm_section_name].get("extraVolumes", []): - volume_name = original_volume['name'] - volume_mounts = static_pod["spec"]["containers"][0].get("volumeMounts", {}) - volume_mount = next((vm for vm in volume_mounts if vm['name'] == volume_name), None) - if volume_mount is None: - logger.debug(f"Extra volume {volume_name!r} is present in {section}, " - f"but not found among volumeMounts in {static_pod_name} manifest for control-plane {node['name']}") - result = False - elif (volume_mount['mountPath'] != original_volume['mountPath'] - or volume_mount.get('readOnly', False) != original_volume.get('readOnly', False)): - logger.debug(f"Specification of extra volume {volume_name!r} in {section} " - f"does not match the specification of volumeMount in {static_pod_name} manifest " - f"for control-plane {node['name']}") - result = False - - volumes = static_pod["spec"].get("volumes", []) - volume = next((v for v in volumes if v['name'] == volume_name), None) - if volume is None: - logger.debug(f"Extra volume {volume_name!r} is present in {section}, " - f"but not found among volumes in {static_pod_name} manifest for control-plane {node['name']}") - result = False - elif (volume['hostPath']['path'] != original_volume['hostPath'] - or volume['hostPath'].get('type') != original_volume.get('pathType')): - logger.debug(f"Specification of extra volume {volume_name!r} in {section} " - f"does not match the specification of volume in {static_pod_name} manifest " - f"for control-plane {node['name']}") - result = False - - return result + return list(failed_nodes) def control_plane_health_status(cluster: KubernetesCluster) -> None: ''' - This test verifies the health of static pods `kube-apiserver`, `kube-controller-manager` and `kube-scheduler` + This test verifies the health of static pods `kube-apiserver`, `kube-controller-manager`, + `kube-scheduler`, and `etcd`. :param cluster: KubernetesCluster object :return: None ''' with TestCase(cluster, '221', "Control plane", "health status") as tc: - static_pods = ['kube-apiserver', 'kube-controller-manager', 'kube-scheduler'] + static_pods = ['kube-apiserver', 'kube-controller-manager', 'kube-scheduler', 'etcd'] static_pod_names = [] for control_plane in cluster.nodes['control-plane'].get_ordered_members_list(): @@ -1154,18 +1186,15 @@ def default_services_configuration_status(cluster: KubernetesCluster) -> None: original_coredns_cm = yaml.safe_load(generate_configmap(cluster.inventory)) result = first_control_plane.sudo('kubectl get cm coredns -n kube-system -oyaml') coredns_cm = yaml.safe_load(result.get_simple_out()) - diff = list(difflib.unified_diff( - coredns_cm['data']['Corefile'].splitlines(), - original_coredns_cm['data']['Corefile'].splitlines(), + diff = utils.get_unified_diff( + coredns_cm['data']['Corefile'], original_coredns_cm['data']['Corefile'], fromfile='kube-system/configmaps/coredns/data/Corefile', - tofile="inventory['services']['coredns']['configmap']['Corefile']", - lineterm='')) + tofile="inventory['services']['coredns']['configmap']['Corefile']") failed_messages = [] warn_messages = [] - if diff: - diff_str = '\n'.join(diff) - failed_messages.append(f"CoreDNS config is outdated: \n{diff_str}") + if diff is not None: + failed_messages.append(f"CoreDNS config is outdated: \n{diff}") software_compatibility = static.GLOBALS["compatibility_map"]["software"] version = cluster.inventory['services']['kubeadm']['kubernetesVersion'] @@ -1417,60 +1446,70 @@ def kubernetes_admission_status(cluster: KubernetesCluster) -> None: and 'kube-apiserver.yaml' and 'kubeadm-config' consistancy """ with TestCase(cluster, '225', "Kubernetes", "Pod Security Admissions") as tc: - first_control_plane = cluster.nodes['control-plane'].get_first_member() - profile_inv = "" - if cluster.inventory["rbac"]["admission"] == "pss" and \ - cluster.inventory["rbac"]["pss"]["pod-security"] == "enabled": - profile_inv = cluster.inventory["rbac"]["pss"]["defaults"]["enforce"] - profile = "" - kubeadm_cm = kubernetes.KubernetesObject(cluster, 'ConfigMap', 'kubeadm-config', 'kube-system') - kubeadm_cm.reload(first_control_plane) - cluster_config = yaml.safe_load(kubeadm_cm.obj["data"]["ClusterConfiguration"]) - api_result = first_control_plane.sudo("cat /etc/kubernetes/manifests/kube-apiserver.yaml") - api_conf = yaml.safe_load(list(api_result.values())[0].stdout) - ext_args = [cmd for cmd in api_conf["spec"]["containers"][0]["command"]] - admission_path = "" - for item in ext_args: - if item.startswith("--"): - key = item.split('=')[0] - value = item[len(key) + 1:] - if key == "--admission-control-config-file": - admission_path = value - adm_result = first_control_plane.sudo("cat %s" % admission_path) - adm_conf = yaml.safe_load(list(adm_result.values())[0].stdout) - profile = adm_conf["plugins"][0]["configuration"]["defaults"]["enforce"] - if admission.is_pod_security_unconditional(cluster): - kube_admission_status = 'PSS is "enabled", default profile is "%s"' % profile - cluster.log.debug(kube_admission_status) - tc.success(results='enabled') - if key == "--feature-gates": - features = value - if "PodSecurity=false" not in features: - kube_admission_status = 'PSS is "enabled", default profile is "%s"' % profile - cluster.log.debug(kube_admission_status) - tc.success(results='enabled') - feature_cm = cluster_config["apiServer"]["extraArgs"].get("feature-gates", "") - if features != feature_cm: - raise TestWarn('enable', - hint=f"Check if the '--feature-gates' option in 'kubeadm-config' " - f"is consistent with 'kube-apiserver.yaml") - admission_path_cm = cluster_config["apiServer"]["extraArgs"].get("admission-control-config-file","") - if admission_path != admission_path_cm: - raise TestWarn('enable', - hint=f"Check if the '--admission-control-config-file' option in 'kubeadm-config' " - f"is consistent with 'kube-apiserver.yaml") - else: - kube_admission_status = 'PSS is "disabled"' - cluster.log.debug(kube_admission_status) - tc.success(results='disabled') - if profile != profile_inv: + # Consistency of kube-apiserver flags is checked in control_plane.configuration_status + # Here we check only the admission configuration and global enabled / disabled status. + + control_planes = cluster.nodes['control-plane'] + first_control_plane = control_planes.get_first_member() + + expected_state = "disabled" + if cluster.inventory["rbac"]["admission"] == "pss": + expected_state = cluster.inventory["rbac"]["pss"]["pod-security"] + + kubeadm_config = components.KubeadmConfig(cluster) + cluster_config = kubeadm_config.load('kubeadm-config', first_control_plane) + apiserver_actual_args = cluster_config["apiServer"]["extraArgs"] + + actual_state = "disabled" + if "admission-control-config-file" in apiserver_actual_args and ( + "PodSecurity=true" in apiserver_actual_args.get("feature-gates", "") + or admission.is_pod_security_unconditional(cluster) + ): + actual_state = "enabled" + + if expected_state != actual_state: raise TestFailure('invalid', - hint=f"The 'cluster.yaml' does not match with the configuration " - f"that is applied on cluster in 'kube-apiserver.yaml' and 'admission.yaml'") - if not profile: + hint=f"Incorrect PSS state. Expected: {expected_state}, actual: {actual_state}") + + if expected_state == "disabled": kube_admission_status = 'PSS is "disabled"' cluster.log.debug(kube_admission_status) - tc.success(results='disabled') + return tc.success(results='disabled') + + expected_config = admission.generate_pss(cluster) + + apiserver_expected_args = cluster.inventory['services']['kubeadm']['apiServer']['extraArgs'] + config_file_name = apiserver_expected_args['admission-control-config-file'] + + broken = [] + result = control_planes.sudo(f"cat {config_file_name}", warn=True) + for node in control_planes.get_ordered_members_list(): + config_result = result[node.get_host()] + node_name = node.get_node_name() + if config_result.failed: + broken.append(f"{node_name}: {config_file_name} is absent") + else: + actual_config = config_result.stdout + diff = utils.get_yaml_diff( + actual_config, expected_config, + fromfile=config_file_name, + tofile="inventory['rbac']['pss']") + if diff is not None: + cluster.log.debug(f"Admission configuration is not actual on {node_name} node") + cluster.log.debug(diff) + broken.append(f"{node_name}: {config_file_name} is not actual") + + if broken: + broken.append( + "Check DEBUG logs for details. " + "To have the check passed, you may need to run `kubemarine manage_pss` with enabled pod-security, " + "or change the inventory file accordingly." + ) + raise TestFailure('invalid', hint=yaml.safe_dump(broken)) + + kube_admission_status = 'PSS is "enabled", default profile is "%s"' % cluster.inventory["rbac"]["pss"]["defaults"]["enforce"] + cluster.log.debug(kube_admission_status) + tc.success(results='enabled') def geo_check(cluster: KubernetesCluster) -> None: @@ -1656,8 +1695,12 @@ def verify_apparmor_config(cluster: KubernetesCluster) -> None: }, 'kubelet': { 'status': lambda cluster: services_status(cluster, 'kubelet'), - 'configuration': lambda cluster: nodes_pid_max(cluster), + 'pid_max': nodes_pid_max, 'version': kubelet_version, + 'configuration': kubelet_config, + }, + 'kube-proxy': { + 'configuration': kube_proxy_configmap, }, 'packages': { 'system': { diff --git a/kubemarine/procedures/install.py b/kubemarine/procedures/install.py index 256c2e52f..a063fb37a 100755 --- a/kubemarine/procedures/install.py +++ b/kubemarine/procedures/install.py @@ -18,9 +18,6 @@ from types import FunctionType from typing import Callable, List, Dict, cast -import yaml -import io - from kubemarine.core.action import Action from kubemarine.core.cluster import KubernetesCluster from kubemarine.core.errors import KME @@ -198,42 +195,7 @@ def deploy_kubernetes_audit(group: NodeGroup) -> None: return kubernetes.prepare_audit_policy(group) - - for control_plane in group.get_ordered_members_list(): - node_config = control_plane.get_config() - config_new = kubernetes.get_kubeadm_config(cluster.inventory) - - # we need InitConfiguration in audit-on-config.yaml file to take into account kubeadm patch for apiserver - init_config = { - 'apiVersion': cluster.inventory["services"]["kubeadm"]['apiVersion'], - 'kind': 'InitConfiguration', - 'localAPIEndpoint': { - 'advertiseAddress': node_config['internal_address'] - }, - 'patches': { - 'directory': '/etc/kubernetes/patches' - } - } - - config_new = config_new + "---\n" + yaml.dump(init_config, default_flow_style=False) - - control_plane.put(io.StringIO(config_new), '/etc/kubernetes/audit-on-config.yaml', sudo=True) - - kubernetes.create_kubeadm_patches_for_node(cluster, control_plane) - - control_plane.sudo(f"kubeadm init phase control-plane apiserver " - f"--config=/etc/kubernetes/audit-on-config.yaml ") - - if cluster.inventory['services']['cri']['containerRuntime'] == 'containerd': - control_plane.call(utils.wait_command_successful, - command="crictl rm -f $(sudo crictl ps --name kube-apiserver -q)") - else: - control_plane.call(utils.wait_command_successful, - command="docker stop $(sudo docker ps -q -f 'name=k8s_kube-apiserver'" - " | awk '{print $1}')") - control_plane.call(utils.wait_command_successful, command="kubectl get pod -n kube-system") - control_plane.sudo("kubeadm init phase upload-config kubeadm " - "--config=/etc/kubernetes/audit-on-config.yaml") + group.call(kubernetes.components.reconfigure_components, components=['kube-apiserver'], force_restart=True) @_applicable_for_new_nodes_with_roles('all') diff --git a/kubemarine/procedures/manage_psp.py b/kubemarine/procedures/manage_psp.py index d4282d0e1..a94bf5fbe 100755 --- a/kubemarine/procedures/manage_psp.py +++ b/kubemarine/procedures/manage_psp.py @@ -23,11 +23,9 @@ from kubemarine.core.resources import DynamicResources tasks = OrderedDict({ - "check_inventory": admission.check_inventory, "delete_custom": admission.delete_custom_task, "add_custom": admission.add_custom_task, - "reconfigure_oob": admission.reconfigure_oob_task, - "reconfigure_plugin": admission.reconfigure_plugin_task, + "reconfigure_psp": admission.reconfigure_psp_task, "restart_pods": admission.restart_pods_task }) diff --git a/kubemarine/procedures/manage_pss.py b/kubemarine/procedures/manage_pss.py index 0c230440d..f4ac51e85 100755 --- a/kubemarine/procedures/manage_pss.py +++ b/kubemarine/procedures/manage_pss.py @@ -23,9 +23,7 @@ from kubemarine.core.resources import DynamicResources tasks = OrderedDict({ - "check_inventory": admission.check_inventory, - "delete_default_pss": admission.delete_default_pss, - "apply_default_pss": admission.apply_default_pss, + "manage_pss": admission.manage_pss, "restart_pods": admission.restart_pods_task }) diff --git a/kubemarine/procedures/migrate_cri.py b/kubemarine/procedures/migrate_cri.py index 188e0b55d..3ff1b88b6 100755 --- a/kubemarine/procedures/migrate_cri.py +++ b/kubemarine/procedures/migrate_cri.py @@ -115,8 +115,8 @@ def _migrate_cri(cluster: KubernetesCluster, node_group: List[NodeGroup]) -> Non else: control_plane.sudo(f"kubectl uncordon {node_name}", hide=False) + kubernetes.components.wait_for_pods(node) if is_control_plane: - kubernetes.wait_for_any_pods(cluster, node, apply_filter=node_name) # check ETCD health etcd.wait_for_health(cluster, node) diff --git a/kubemarine/procedures/migrate_kubemarine.py b/kubemarine/procedures/migrate_kubemarine.py index 768f34e6f..e9016c82f 100644 --- a/kubemarine/procedures/migrate_kubemarine.py +++ b/kubemarine/procedures/migrate_kubemarine.py @@ -139,8 +139,8 @@ def upgrade_cri(self, group: NodeGroup, workers: bool) -> None: else: kubernetes.wait_uncordon(node) + kubernetes.components.wait_for_pods(node) if not workers: - kubernetes.wait_for_any_pods(cluster, node, apply_filter=node_name) etcd.wait_for_health(cluster, node) def associations_changed(self, res: DynamicResources) -> bool: diff --git a/kubemarine/procedures/reconfigure.py b/kubemarine/procedures/reconfigure.py new file mode 100644 index 000000000..4b226f492 --- /dev/null +++ b/kubemarine/procedures/reconfigure.py @@ -0,0 +1,70 @@ +from collections import OrderedDict +from typing import List, Union + +from ordered_set import OrderedSet + +from kubemarine import kubernetes +from kubemarine.core import flow +from kubemarine.core.action import Action +from kubemarine.core.cluster import KubernetesCluster +from kubemarine.core.resources import DynamicResources + + +def deploy_kubernetes_reconfigure(cluster: KubernetesCluster) -> None: + changed_components = OrderedSet[str]() + for component, constants in kubernetes.components.COMPONENTS_CONSTANTS.items(): + for section_names in constants['sections']: + section: Union[dict, list, None] = cluster.procedure_inventory + for name in section_names: + if isinstance(section, dict): + section = section.get(name) + + if section is not None: + changed_components.add(component) + + if changed_components: + cluster.log.debug(f"Detected changes in components: {', '.join(changed_components)}") + kubernetes_nodes = cluster.make_group_from_roles(['control-plane', 'worker']) + kubernetes_nodes.call(kubernetes.components.reconfigure_components, components=list(changed_components)) + else: + cluster.log.debug("No changes detected, skipping.") + + +tasks = OrderedDict({ + "deploy": { + "kubernetes": { + "reconfigure": deploy_kubernetes_reconfigure + } + } +}) + + +class ReconfigureAction(Action): + def __init__(self) -> None: + super().__init__('reconfigure', recreate_inventory=True) + + def run(self, res: DynamicResources) -> None: + flow.run_tasks(res, tasks) + res.make_final_inventory() + + +def create_context(cli_arguments: List[str] = None) -> dict: + cli_help = ''' + Script for generic reconfiguring of existing Kubernetes cluster. + + How to use: + + ''' + + parser = flow.new_procedure_parser(cli_help, tasks=tasks) + context = flow.create_context(parser, cli_arguments, procedure="reconfigure") + return context + + +def main(cli_arguments: List[str] = None) -> None: + context = create_context(cli_arguments) + flow.ActionsFlow([ReconfigureAction()]).run_flow(context) + + +if __name__ == '__main__': + main() diff --git a/kubemarine/procedures/upgrade.py b/kubemarine/procedures/upgrade.py index cc5f4fb66..88762201d 100755 --- a/kubemarine/procedures/upgrade.py +++ b/kubemarine/procedures/upgrade.py @@ -15,7 +15,7 @@ import copy from collections import OrderedDict from itertools import chain -from typing import List +from typing import List, Callable, Dict from kubemarine import kubernetes, plugins, admission from kubemarine.core import flow @@ -51,21 +51,35 @@ def prepull_images(cluster: KubernetesCluster) -> None: def kubernetes_upgrade(cluster: KubernetesCluster) -> None: initial_kubernetes_version = cluster.context['initial_kubernetes_version'] - first_control_plane = cluster.nodes["control-plane"].get_first_member() upgrade_group = kubernetes.get_group_for_upgrade(cluster) + preconfigure_components = [] + preconfigure_functions: Dict[str, Callable[[dict], dict]] = {} if (admission.is_pod_security_unconditional(cluster) and utils.version_key(initial_kubernetes_version)[0:2] < utils.minor_version_key("v1.28") and cluster.inventory['rbac']['pss']['pod-security'] == 'enabled'): - cluster.log.debug("Updating kubeadm config map") - final_features_list = first_control_plane.call(admission.update_kubeadm_configmap_pss, target_state="enabled") - cluster.log.debug("Updating kube-apiserver configs on control-planes") - cluster.nodes["control-plane"].call(admission.update_kubeapi_config_pss, features_list=final_features_list) + # Extra args of API server have changed, need to reconfigure the API server. + # See admission.enrich_inventory_pss() + # Still, should not reconfigure using generated ConfigMaps from inventory, + # because the inventory has already incremented kubernetesVersion, but the cluster is not upgraded yet. + # Instead, change only necessary apiServer args. + def reconfigure_feature_gates(cluster_config: dict) -> dict: + feature_gates = cluster.inventory["services"]["kubeadm"]["apiServer"]["extraArgs"].get("feature-gates") + if feature_gates is not None: + cluster_config["apiServer"]["extraArgs"]["feature-gates"] = feature_gates + else: + del cluster_config["apiServer"]["extraArgs"]["feature-gates"] - if (kubernetes.kube_proxy_overwrites_higher_system_values(cluster) + return cluster_config + + preconfigure_components.append('kube-apiserver') + preconfigure_functions['kubeadm-config'] = reconfigure_feature_gates + + if (kubernetes.components.kube_proxy_overwrites_higher_system_values(cluster) and utils.version_key(initial_kubernetes_version)[0:2] < utils.minor_version_key("v1.29")): - cluster.log.debug("Updating kube-proxy config map") + # Defaults of KubeProxyConfiguration have changed. + # See services.kubeadm_kube-proxy.conntrack.min section of defaults.yaml def edit_kube_proxy_conntrack_min(kube_proxy_cm: dict) -> dict: expected_conntrack: dict = cluster.inventory['services']['kubeadm_kube-proxy']['conntrack'] if 'min' not in expected_conntrack: @@ -77,7 +91,12 @@ def edit_kube_proxy_conntrack_min(kube_proxy_cm: dict) -> dict: return kube_proxy_cm - first_control_plane.call(kubernetes.reconfigure_kube_proxy_configmap, mutate_func=edit_kube_proxy_conntrack_min) + preconfigure_components.append('kube-proxy') + preconfigure_functions['kube-proxy'] = edit_kube_proxy_conntrack_min + + if preconfigure_components: + upgrade_group.call(kubernetes.components.reconfigure_components, + components=preconfigure_components, edit_functions=preconfigure_functions) drain_timeout = cluster.procedure_inventory.get('drain_timeout') grace_period = cluster.procedure_inventory.get('grace_period') @@ -97,14 +116,14 @@ def edit_kube_proxy_conntrack_min(kube_proxy_cm: dict) -> dict: if cluster.nodes.get('worker', []): kubernetes.upgrade_workers(upgrade_group, cluster, **drain_kwargs) - cluster.nodes['control-plane'].get_first_member().sudo('rm -f /etc/kubernetes/nodes-k8s-versions.txt') - cluster.context['cached_nodes_versions_cleaned'] = True + kubernetes_cleanup_nodes_versions(cluster) def kubernetes_cleanup_nodes_versions(cluster: KubernetesCluster) -> None: if not cluster.context.get('cached_nodes_versions_cleaned', False): cluster.log.verbose('Cached nodes versions required') cluster.nodes['control-plane'].get_first_member().sudo('rm -f /etc/kubernetes/nodes-k8s-versions.txt') + cluster.context['cached_nodes_versions_cleaned'] = True else: cluster.log.verbose('Cached nodes versions already cleaned') diff --git a/kubemarine/resources/configurations/defaults.yaml b/kubemarine/resources/configurations/defaults.yaml index 12d9f49c9..574edb8ad 100644 --- a/kubemarine/resources/configurations/defaults.yaml +++ b/kubemarine/resources/configurations/defaults.yaml @@ -81,10 +81,12 @@ services: scheduler: extraArgs: profiling: "false" + extraVolumes: [] controllerManager: extraArgs: profiling: "false" terminated-pod-gc-threshold: "1000" + extraVolumes: [] kubeadm_patches: # bind-address flag for apiServer is set in the code apiServer: [] diff --git a/kubemarine/resources/schemas/definitions/common/node_ref.json b/kubemarine/resources/schemas/definitions/common/node_ref.json index 76164017b..ce981f5d3 100644 --- a/kubemarine/resources/schemas/definitions/common/node_ref.json +++ b/kubemarine/resources/schemas/definitions/common/node_ref.json @@ -4,6 +4,12 @@ "Role": { "enum": ["worker", "control-plane", "master", "balancer"] }, + "Kubernetes": { + "enum": ["worker", "control-plane"] + }, + "ControlPlane": { + "enum": ["control-plane"] + }, "Roles": { "type": "array", "items": { @@ -12,6 +18,22 @@ "uniqueItems": true, "minItems": 1 }, + "KubernetesRoles": { + "type": "array", + "items": { + "$ref": "#/definitions/Kubernetes" + }, + "uniqueItems": true, + "minItems": 1 + }, + "ControlPlanes": { + "type": "array", + "items": { + "$ref": "#/definitions/ControlPlane" + }, + "uniqueItems": true, + "minItems": 1 + }, "Name": { "type": "string" }, @@ -22,6 +44,28 @@ }, "uniqueItems": true, "minItems": 1 + }, + "OneOfNodesGroupsSpec": { + "oneOf": [ + { + "type": "object", + "properties": { + "groups": { + "type": "array" + } + }, + "required": ["groups"] + }, + { + "type": "object", + "properties": { + "nodes": { + "type": "array" + } + }, + "required": ["nodes"] + } + ] } } } diff --git a/kubemarine/resources/schemas/definitions/services/kubeadm.json b/kubemarine/resources/schemas/definitions/services/kubeadm.json index dad6f7114..f9ca992ef 100644 --- a/kubemarine/resources/schemas/definitions/services/kubeadm.json +++ b/kubemarine/resources/schemas/definitions/services/kubeadm.json @@ -45,7 +45,7 @@ "$ref": "#/definitions/ControllerManager" }, "etcd": { - "$ref": "#/definitions/ETCD" + "$ref": "#/definitions/Etcd" }, "apiVersion": {"type": ["string"], "default": "kubeadm.k8s.io/v1beta2"}, "kind": {"enum": ["ClusterConfiguration"], "default": "ClusterConfiguration"} @@ -53,12 +53,12 @@ "definitions": { "ApiServer": { "type": "object", + "allOf": [{"$ref": "#/definitions/ControlPlaneComponentProperties"}], "properties": { "certSANs": { "$ref": "../common/utils.json#/definitions/ArrayOfStrings" }, "extraArgs": { - "type": "object", "properties": { "enable-admission-plugins": {"type": "string", "default": "NodeRestriction"}, "profiling": {"type": "string", "default": "'false'"}, @@ -72,48 +72,39 @@ "service-account-jwks-uri": {"type": "string"}, "service-account-signing-key-file": {"type": "string"}, "service-account-key-file": {"type": "string"} - }, - "additionalProperties": { - "type": "string" } }, - "extraVolumes": { - "type": "array", - "items": { - "oneOf": [ - {"$ref": "#/definitions/HostPathMount"}, - {"$ref": "../common/utils.json#/definitions/ListMergingSymbol"} - ] - } + "timeoutForControlPlane": { + "type": "string" } + }, + "propertyNames": { + "anyOf": [ + {"$ref": "#/definitions/ControlPlaneComponentPropertyNames"}, + {"enum": ["certSANs", "timeoutForControlPlane"]} + ] } }, "Scheduler": { "type": "object", + "allOf": [{"$ref": "#/definitions/ControlPlaneComponentProperties"}], "properties": { "extraArgs": { - "type": "object", "properties": { "profiling": {"type": "string", "default": "'false'"}, "feature-gates": {"type": "string"} - }, - "additionalProperties": { - "type": "string" - } - }, - "extraVolumes": { - "type": "array", - "items": { - "$ref": "#/definitions/HostPathMount" } } + }, + "propertyNames": { + "$ref": "#/definitions/ControlPlaneComponentPropertyNames" } }, "ControllerManager": { "type": "object", + "allOf": [{"$ref": "#/definitions/ControlPlaneComponentProperties"}], "properties": { "extraArgs": { - "type": "object", "properties": { "profiling": {"type": "string", "default": "'false'"}, "terminated-pod-gc-threshold": {"type": "string", "default": "'1000'"}, @@ -122,35 +113,54 @@ "description": "Plugin to enable the CPP support" }, "feature-gates": {"type": "string"} - }, - "additionalProperties": { - "type": "string" - } - }, - "extraVolumes": { - "type": "array", - "items": { - "$ref": "#/definitions/HostPathMount" } } + }, + "propertyNames": { + "$ref": "#/definitions/ControlPlaneComponentPropertyNames" } }, - "ETCD": { + "Etcd": { "type": "object", "properties": { "local": { "type": "object", "properties": { "extraArgs": { - "type": "object", - "additionalProperties": { - "type": "string" - } + "$ref": "#/definitions/ExtraArgs" } } } } }, + "ControlPlaneComponentProperties": { + "properties": { + "extraArgs": { + "$ref": "#/definitions/ExtraArgs" + }, + "extraVolumes": { + "$ref": "#/definitions/ExtraVolumes" + } + } + }, + "ControlPlaneComponentPropertyNames": { + "enum": ["extraArgs", "extraVolumes"] + }, + "ExtraArgs": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "ExtraVolumes": { + "type": "array", + "items": { + "oneOf": [ + {"$ref": "#/definitions/HostPathMount"}, + {"$ref": "../common/utils.json#/definitions/ListMergingSymbol"} + ] + } + }, "HostPathMount": { "type": "object", "properties": { diff --git a/kubemarine/resources/schemas/definitions/services/kubeadm_kube-proxy.json b/kubemarine/resources/schemas/definitions/services/kubeadm_kube-proxy.json index 0fd755d56..31db908c4 100644 --- a/kubemarine/resources/schemas/definitions/services/kubeadm_kube-proxy.json +++ b/kubemarine/resources/schemas/definitions/services/kubeadm_kube-proxy.json @@ -2,19 +2,27 @@ "$schema": "http://json-schema.org/draft-07/schema", "type": "object", "description": "Override the original settings for the kube-proxy", + "allOf": [{"$ref": "#/definitions/PayloadProperties"}], "properties": { - "conntrack": { + "apiVersion": {"enum": ["kubeproxy.config.k8s.io/v1alpha1"], "default": "kubeproxy.config.k8s.io/v1alpha1"}, + "kind": {"enum": ["KubeProxyConfiguration"], "default": "KubeProxyConfiguration"} + }, + "definitions": { + "PayloadProperties": { "type": "object", - "description": "Conntrack settings for the Kubernetes proxy server", "properties": { - "min": { - "type": ["string", "integer"], - "description": "Minimum value of connect-tracking records to allocate. By default, inherits the `services.sysctl.net.netfilter.nf_conntrack_max` value.", - "default": 1000000 + "conntrack": { + "type": "object", + "description": "Conntrack settings for the Kubernetes proxy server", + "properties": { + "min": { + "type": ["string", "integer"], + "description": "Minimum value of connect-tracking records to allocate. By default, inherits the `services.sysctl.net.netfilter.nf_conntrack_max` value.", + "default": 1000000 + } + } } } - }, - "apiVersion": {"enum": ["kubeproxy.config.k8s.io/v1alpha1"], "default": "kubeproxy.config.k8s.io/v1alpha1"}, - "kind": {"enum": ["KubeProxyConfiguration"], "default": "KubeProxyConfiguration"} + } } } diff --git a/kubemarine/resources/schemas/definitions/services/kubeadm_kubelet.json b/kubemarine/resources/schemas/definitions/services/kubeadm_kubelet.json index 2eda8ad6b..70d5132ca 100644 --- a/kubemarine/resources/schemas/definitions/services/kubeadm_kubelet.json +++ b/kubemarine/resources/schemas/definitions/services/kubeadm_kubelet.json @@ -4,11 +4,17 @@ "description": "Override the original settings for the kubelet", "properties": { "readOnlyPort": {"type": "integer", "default": 0}, - "protectKernelDefaults": {"type": "boolean", "default": true}, + "enableDebuggingHandlers": {"type": "boolean", "default": true}, + "protectKernelDefaults": {"$ref": "#/definitions/ProtectKernelDefaults"}, "podPidsLimit": {"type": "integer", "default": 4096}, "cgroupDriver": {"type": "string", "default": "systemd"}, "maxPods": {"type": "integer", "default": 110}, + "serializeImagePulls": {"$ref": "#/definitions/SerializeImagePulls"}, "apiVersion": {"type": ["string"], "default": "kubelet.config.k8s.io/v1beta1"}, "kind": {"enum": ["KubeletConfiguration"], "default": "KubeletConfiguration"} + }, + "definitions": { + "ProtectKernelDefaults": {"type": "boolean", "default": true}, + "SerializeImagePulls": {"type": "boolean", "default": false} } } diff --git a/kubemarine/resources/schemas/definitions/services/kubeadm_patches.json b/kubemarine/resources/schemas/definitions/services/kubeadm_patches.json index 2dee18aee..c73bb6964 100644 --- a/kubemarine/resources/schemas/definitions/services/kubeadm_patches.json +++ b/kubemarine/resources/schemas/definitions/services/kubeadm_patches.json @@ -26,50 +26,32 @@ "description": "Patches for control-plane pods", "minItems": 0, "items": { - "anyOf": [ - { - "type": "object", - "additionalProperties": false, - "properties": { - "groups": { - "type": "array", - "items": { - "enum": ["control-plane"] + "oneOf": [ + { + "allOf": [ + { + "$ref": "../common/node_ref.json#/definitions/OneOfNodesGroupsSpec" + }, + { + "type": "object", + "properties": { + "patch": { + "$ref": "#/definitions/patch" + }, + "groups": { + "$ref": "../common/node_ref.json#/definitions/ControlPlanes" + }, + "nodes": { + "$ref": "../common/node_ref.json#/definitions/Names" + } + }, + "required": ["patch"], + "additionalProperties": false } - }, - "patch": { - "type": "object", - "properties": { - "additionalProperties": { - "type": "string" - } - } - } - } - }, - { - "type": "object", - "additionalProperties": false, - "properties": { - "nodes": { - "type": "array", - "items": [ - { - "type": "string" - } - ] - }, - "patch": { - "type": "object", - "properties": { - "additionalProperties": { - "type": "string" - } - } - } - } - } - ] + ] + }, + {"$ref": "../common/utils.json#/definitions/ListMergingSymbol"} + ] } }, "kubelet-patch": { @@ -77,52 +59,39 @@ "description": "Patches for kubelet", "minItems": 0, "items": { - "anyOf": [ - { - "type": "object", - "additionalProperties": false, - "properties": { - "groups": { - "type": "array", - "items": { - "enum": ["control-plane", "worker"] - } - }, - "patch": { - "type": "object", - "properties": { - "additionalProperties": { - "type": "string" - } + "oneOf": [ + { + "allOf": [ + { + "$ref": "../common/node_ref.json#/definitions/OneOfNodesGroupsSpec" + }, + { + "type": "object", + "properties": { + "patch": { + "$ref": "#/definitions/patch" + }, + "groups": { + "$ref": "../common/node_ref.json#/definitions/KubernetesRoles" + }, + "nodes": { + "$ref": "../common/node_ref.json#/definitions/Names" + } + }, + "required": ["patch"], + "additionalProperties": false } - } - } - }, - { - "type": "object", - "additionalProperties": false, - "properties": { - "nodes": { - "type": "array", - "items": [ - { - "type": "string" - } - ] - }, - "patch": { - "type": "object", - "properties": { - "additionalProperties": { - "type": "string" - } - } - } - } - } - ] + ] + }, + {"$ref": "../common/utils.json#/definitions/ListMergingSymbol"} + ] + } + }, + "patch": { + "type": "object", + "additionalProperties": { + "type": ["string", "boolean", "integer"] } } } } - diff --git a/kubemarine/resources/schemas/reconfigure.json b/kubemarine/resources/schemas/reconfigure.json new file mode 100644 index 000000000..b5b7495e9 --- /dev/null +++ b/kubemarine/resources/schemas/reconfigure.json @@ -0,0 +1,65 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema", + "type": "object", + "properties": { + "services": { + "type": "object", + "description": "Configure settings of different services", + "properties": { + "kubeadm_kubelet": { + "type": "object", + "description": "Override the original settings for the kubelet", + "properties": { + "protectKernelDefaults": {"$ref": "definitions/services/kubeadm_kubelet.json#/definitions/ProtectKernelDefaults"}, + "serializeImagePulls": {"$ref": "definitions/services/kubeadm_kubelet.json#/definitions/SerializeImagePulls"} + }, + "additionalProperties": false + }, + "kubeadm_kube-proxy": { + "type": "object", + "description": "Override the original settings for the kube-proxy", + "allOf": [{"$ref": "definitions/services/kubeadm_kube-proxy.json#/definitions/PayloadProperties"}], + "propertyNames": { + "not": {"enum": ["apiVersion", "kind"]} + } + }, + "kubeadm_patches": { + "$ref": "definitions/services/kubeadm_patches.json" + }, + "kubeadm": { + "type": "object", + "description": "Override the original settings for the kubeadm", + "properties": { + "apiServer": { + "$ref": "definitions/services/kubeadm.json#/definitions/ApiServer" + }, + "scheduler": { + "$ref": "definitions/services/kubeadm.json#/definitions/Scheduler" + }, + "controllerManager": { + "$ref": "definitions/services/kubeadm.json#/definitions/ControllerManager" + }, + "etcd": { + "type": "object", + "properties": { + "local": { + "type": "object", + "properties": { + "extraArgs": { + "$ref": "definitions/services/kubeadm.json#/definitions/ExtraArgs" + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false + } + }, + "additionalProperties": false +} diff --git a/test/unit/core/test_schema.py b/test/unit/core/test_schema.py index 19755a0a2..19d753d71 100644 --- a/test/unit/core/test_schema.py +++ b/test/unit/core/test_schema.py @@ -215,6 +215,21 @@ def test_required_and_optional_properties_heuristic(self): with self.assertRaisesRegex(errors.FailException, r"'address' was unexpected"): demo.new_cluster(inventory) + def test_propertyNames_not_enum(self): + """ + 'services.kubeadm_kube-proxy' section is an example where propertyNames are configured as not(enum). + Specify unexpected property to check correctly generated error. + See kubemarine.core.schema._friendly_msg + """ + inventory = demo.generate_inventory(**demo.ALLINONE) + context = demo.create_silent_context(["fake.yaml"], procedure='reconfigure') + + reconfigure = demo.generate_procedure_inventory(procedure='reconfigure') + reconfigure.setdefault('services', {}).setdefault('kubeadm_kube-proxy', {})['kind'] = 'unsupported' + + with self.assertRaisesRegex(errors.FailException, "Property name 'kind' is unexpected"): + demo.new_cluster(inventory, procedure_inventory=reconfigure, context=context) + if __name__ == '__main__': unittest.main() diff --git a/test/unit/test_add_node.py b/test/unit/test_add_node.py index 215f9a89b..b9314cf6e 100644 --- a/test/unit/test_add_node.py +++ b/test/unit/test_add_node.py @@ -16,8 +16,9 @@ import tempfile import unittest -from kubemarine import demo +from kubemarine import demo, kubernetes from kubemarine.core import flow +from kubemarine.kubernetes import components from kubemarine.procedures.add_node import AddNodeAction from test.unit import utils as test_utils @@ -71,5 +72,57 @@ def test_enrich_inventory_generate_ansible_new_nodes_group(self): "Unexpected group with new nodes") +class RunTasks(unittest.TestCase): + def setUp(self): + self.inventory = {} + self.context = {} + + def _run_tasks(self, tasks_filter: str, added_node_name: str) -> demo.FakeResources: + context = demo.create_silent_context( + ['fake.yaml', '--tasks', tasks_filter], procedure='add_node') + + nodes_context = demo.generate_nodes_context(self.inventory) + + added_node_idx = next(i for i, node in enumerate(self.inventory['nodes']) + if node['name'] == added_node_name) + + added_node = self.inventory['nodes'].pop(added_node_idx) + procedure_inventory = demo.generate_procedure_inventory('add_node') + procedure_inventory['nodes'] = [added_node] + + resources = demo.FakeResources(context, self.inventory, + procedure_inventory=procedure_inventory, nodes_context=nodes_context) + flow.run_actions(resources, [AddNodeAction()]) + return resources + + def test_kubernetes_init_write_new_certificates(self): + for new_role, expected_called in (('worker', False), ('master', True), ('balancer', True)): + with self.subTest(f"Add: {new_role}"), \ + test_utils.mock_call(kubernetes.join_new_control_plane), \ + test_utils.mock_call(kubernetes.init_workers), \ + test_utils.mock_call(kubernetes.apply_labels), \ + test_utils.mock_call(kubernetes.apply_taints), \ + test_utils.mock_call(kubernetes.wait_for_nodes), \ + test_utils.mock_call(kubernetes.schedule_running_nodes_report), \ + test_utils.mock_call(components.reconfigure_components) as run: + + self.inventory = demo.generate_inventory(balancer=2, master=2, worker=2) + + new_node_name = f'{new_role}-2' + res = self._run_tasks('deploy.kubernetes.init', new_node_name) + + actual_called_components = run.call_args[0][1] if run.called else [] + expected_called_components = ['kube-apiserver/cert-sans'] if expected_called else [] + self.assertEqual(expected_called_components, actual_called_components, + f"New certificate was {'not' if expected_called else 'unexpectedly'} written") + + if expected_called: + self.assertEqual(['master-1', 'master-2'], run.call_args[0][0].get_nodes_names()) + + certsans = res.last_cluster.inventory['services']['kubeadm']['apiServer']['certSANs'] + self.assertEqual(expected_called, new_node_name in certsans, + "New certificate should be written if and only if new cert SAN appears") + + if __name__ == '__main__': unittest.main() diff --git a/test/unit/test_group.py b/test/unit/test_group.py index 1d99dcc8a..b96be10d5 100755 --- a/test/unit/test_group.py +++ b/test/unit/test_group.py @@ -184,6 +184,27 @@ def test_write_large_file(self): for host in all_nodes.get_hosts(): self.assertEqual('a' * 100000, self.cluster.fake_fs.read(host, '/fake/path')) + def test_wait_commands_successful_intermediate_failed(self): + node = self.cluster.nodes["all"].get_first_member() + + results = demo.create_hosts_result(node.get_hosts(), stdout='result1') + TestGroupCall.cluster.fake_shell.add(results, "run", ['command1'], usage_limit=1) + + results = demo.create_hosts_result(node.get_hosts(), stderr='error2', code=1) + TestGroupCall.cluster.fake_shell.add(results, "run", ['command2'], usage_limit=1) + + results = demo.create_hosts_result(node.get_hosts(), stdout='result2') + TestGroupCall.cluster.fake_shell.add(results, "run", ['command2'], usage_limit=1) + + results = demo.create_hosts_result(node.get_hosts(), stdout='result3') + TestGroupCall.cluster.fake_shell.add(results, "run", ['command3'], usage_limit=1) + + node.wait_commands_successful(['command1', 'command2', 'command3'], sudo=False, timeout=0) + + for cmd, expected_calls in (('command1', 1), ('command2', 2), ('command3', 1)): + actual_calls = TestGroupCall.cluster.fake_shell.called_times(node.get_host(), 'run', [cmd]) + self.assertEqual(expected_calls, actual_calls, "Number of calls is not expected") + if __name__ == '__main__': unittest.main() diff --git a/test/unit/test_inventory.py b/test/unit/test_inventory.py index 7bb9a3c49..1a17b824e 100755 --- a/test/unit/test_inventory.py +++ b/test/unit/test_inventory.py @@ -426,6 +426,76 @@ def test_enrich_certsans_with_custom(self): self.assertIn('custom', certsans) self.assertEqual(1, len([san for san in certsans if san == first_node_name])) + def test_enrich_psp_extra_args_plugins_list(self): + inventory = demo.generate_inventory(**demo.ALLINONE) + inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = 'v1.24.11' + inventory['rbac'] = { + 'admission': 'psp', + 'psp': {'pod-security': 'enabled'} + } + + admission_plugins_expected = 'NodeRestriction,PodSecurityPolicy' + + cluster = demo.new_cluster(inventory) + apiserver_extra_args = cluster.inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual(admission_plugins_expected, apiserver_extra_args.get('enable-admission-plugins')) + + test_utils.stub_associations_packages(cluster, {}) + finalized_inventory = test_utils.make_finalized_inventory(cluster) + apiserver_extra_args = finalized_inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual(admission_plugins_expected, apiserver_extra_args.get('enable-admission-plugins')) + + def test_conditional_enrich_pss_extra_args_feature_gates(self): + for k8s_version, feature_gates_enriched in (('v1.27.8', True), ('v1.28.4', False)): + with self.subTest(f"Kubernetes: {k8s_version}"): + inventory = demo.generate_inventory(**demo.ALLINONE) + inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = k8s_version + inventory['rbac'] = { + 'admission': 'pss', + 'pss': {'pod-security': 'enabled'} + } + + feature_gates_expected = 'PodSecurity=true' if feature_gates_enriched else None + + cluster = demo.new_cluster(inventory) + apiserver_extra_args = cluster.inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual(feature_gates_expected, apiserver_extra_args.get('feature-gates')) + self.assertEqual('/etc/kubernetes/pki/admission.yaml', apiserver_extra_args['admission-control-config-file']) + + test_utils.stub_associations_packages(cluster, {}) + finalized_inventory = test_utils.make_finalized_inventory(cluster) + apiserver_extra_args = finalized_inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual(feature_gates_expected, apiserver_extra_args.get('feature-gates')) + self.assertEqual('/etc/kubernetes/pki/admission.yaml', apiserver_extra_args['admission-control-config-file']) + self.assertNotIn('psp', finalized_inventory['rbac']) + + def test_enrich_pss_extra_args_feature_gates_custom(self): + inventory = demo.generate_inventory(**demo.ALLINONE) + inventory['services']['kubeadm'] = { + 'kubernetesVersion': 'v1.27.8', + 'apiServer': {'extraArgs': {'feature-gates': 'ServiceAccountIssuerDiscovery=true'}}, + } + inventory['rbac'] = { + 'admission': 'pss', + 'pss': {'pod-security': 'enabled'} + } + + cluster = demo.new_cluster(inventory) + apiserver_extra_args = cluster.inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual('ServiceAccountIssuerDiscovery=true,PodSecurity=true', apiserver_extra_args.get('feature-gates')) + + test_utils.stub_associations_packages(cluster, {}) + finalized_inventory = test_utils.make_finalized_inventory(cluster) + apiserver_extra_args = finalized_inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual('ServiceAccountIssuerDiscovery=true,PodSecurity=true', apiserver_extra_args.get('feature-gates')) + self.assertNotIn('psp', finalized_inventory['rbac']) + if __name__ == '__main__': unittest.main() diff --git a/test/unit/test_kubernetes_components.py b/test/unit/test_kubernetes_components.py new file mode 100644 index 000000000..9ba9be139 --- /dev/null +++ b/test/unit/test_kubernetes_components.py @@ -0,0 +1,483 @@ +import json +import random +import re +import unittest +from contextlib import contextmanager +from copy import deepcopy +from typing import List + +import yaml +from ordered_set import OrderedSet + +from kubemarine import demo, plugins, system +from kubemarine.kubernetes import components +from test.unit import utils as test_utils + + +class KubeadmConfigTest(unittest.TestCase): + def test_get_init_config_control_plane(self): + inventory = demo.generate_inventory(master=1, worker=1, balancer=0) + cluster = demo.new_cluster(inventory) + control_plane = cluster.nodes['control-plane'].get_first_member() + init_config = components.get_init_config(cluster, control_plane, init=True) + + self.assertEqual({'advertiseAddress': inventory['nodes'][0]['internal_address']}, + init_config.get('localAPIEndpoint')) + + self.assertEqual(None, init_config.get('nodeRegistration', {}).get('taints')) + + self.assertNotIn('discovery', init_config) + + def test_get_init_config_combined(self): + inventory = demo.generate_inventory(**demo.ALLINONE) + cluster = demo.new_cluster(inventory) + control_plane = cluster.nodes['control-plane'].get_first_member() + init_config = components.get_init_config(cluster, control_plane, init=True) + + self.assertEqual({'advertiseAddress': inventory['nodes'][0]['internal_address']}, + init_config.get('localAPIEndpoint')) + + self.assertEqual([], init_config.get('nodeRegistration', {}).get('taints')) + + self.assertNotIn('discovery', init_config) + + def test_get_join_config_control_plane(self): + inventory = demo.generate_inventory(master=1, worker=1, balancer=0) + cluster = demo.new_cluster(inventory) + control_plane = cluster.nodes['control-plane'].get_first_member() + join_config = components.get_init_config(cluster, control_plane, init=False, join_dict={ + 'certificate-key': '01233456789abcdef', + 'token': 'abc.xyz', + 'discovery-token-ca-cert-hash': 'sha256:01233456789abcdef', + }) + + self.assertEqual({ + 'localAPIEndpoint': {'advertiseAddress': inventory['nodes'][0]['internal_address']}, + 'certificateKey': '01233456789abcdef' + }, join_config.get('controlPlane')) + self.assertEqual(None, join_config.get('nodeRegistration', {}).get('taints')) + + self.assertIn('bootstrapToken', join_config.get('discovery', {})) + + def test_get_init_config_worker_group(self): + inventory = demo.generate_inventory(master=1, worker=2, balancer=0) + cluster = demo.new_cluster(inventory) + workers = cluster.nodes['worker'] + init_config = components.get_init_config(cluster, workers, init=True) + + self.assertEqual(None, init_config.get('localAPIEndpoint')) + self.assertEqual(None, init_config.get('nodeRegistration', {}).get('taints')) + + def test_merge_with_inventory(self): + inventory = demo.generate_inventory(**demo.ALLINONE) + inventory['services']['kubeadm_kube-proxy'] = { + 'nested': {'property': 'new'}, + 'array': [2] + } + cluster = demo.new_cluster(inventory) + + control_plane_host = inventory['nodes'][0]['address'] + data = {'data': {'config.conf': yaml.dump({ + 'kind': 'KubeProxyConfiguration', + 'nested': {'untouched': True, 'property': 'old'}, + 'array': [1] + })}} + results = demo.create_hosts_result([control_plane_host], stdout=json.dumps(data)) + cmd = f'kubectl get configmap -n kube-system kube-proxy -o json' + cluster.fake_shell.add(results, 'sudo', [cmd]) + + control_plane = cluster.make_group([control_plane_host]) + + kubeadm_config = components.KubeadmConfig(cluster) + kubeadm_config.load('kube-proxy', control_plane, kubeadm_config.merge_with_inventory('kube-proxy')) + + self._test_merge_with_inventory(kubeadm_config.maps['kube-proxy']) + self._test_merge_with_inventory( + yaml.safe_load(kubeadm_config.loaded_maps['kube-proxy'].obj['data']['config.conf'])) + + kubeadm_config = components.KubeadmConfig(cluster) + loaded_config = kubeadm_config.load('kube-proxy', control_plane) + self.assertEqual('old', loaded_config.get('nested', {}).get('property')) + self.assertEqual(True, loaded_config.get('nested', {}).get('untouched')) + self.assertEqual([1], loaded_config.get('array')) + + merged_config = kubeadm_config.merge_with_inventory('kube-proxy')(deepcopy(loaded_config)) + self._test_merge_with_inventory(merged_config) + + def _test_merge_with_inventory(self, config: dict): + self.assertEqual('new', config.get('nested', {}).get('property')) + self.assertEqual(True, config.get('nested', {}).get('untouched')) + self.assertEqual([2], config.get('array')) + + +class WaitForPodsTest(unittest.TestCase): + def setUp(self): + self.inventory = demo.generate_inventory(**demo.FULLHA) + random.shuffle(self.inventory['nodes']) + + self.inventory.setdefault('globals', {}).setdefault('expect', {}).setdefault('pods', {})['kubernetes'] = { + 'timeout': 0, 'retries': 3 + } + + def _new_cluster(self) -> demo.FakeKubernetesCluster: + return demo.new_cluster(self.inventory) + + def _stub_get_pods(self, cluster: demo.FakeKubernetesCluster, hosts: List[str], pods: List[str], node_name: str, + *, ready: bool = True): + internal_address = cluster.get_node_by_name(node_name)['internal_address'] + ready_string = '1/1' if ready else '0/1' + output = '\n'.join(( + f'{pod} {ready_string} Running 0 1s {internal_address} {node_name} ' + for pod in pods + )) + results = demo.create_hosts_result(hosts, stdout=output) + cmd = f'kubectl get pods -n kube-system -o=wide | grep {node_name}' + cluster.fake_shell.add(results, 'sudo', [cmd]) + + def test_wait_empty(self): + cluster = self._new_cluster() + components.wait_for_pods(cluster.nodes['all'], []) + + def test_wait_not_supported(self): + cluster = self._new_cluster() + with self.assertRaisesRegex(Exception, re.escape(components.ERROR_WAIT_FOR_PODS_NOT_SUPPORTED.format( + components=['kube-apiserver/cert-sans', 'kubelet', 'unexpected-component']))): + components.wait_for_pods(cluster.nodes['all'].get_any_member(), components.ALL_COMPONENTS + ['unexpected-component']) + + def test_wait_workers_successful(self): + cluster = self._new_cluster() + first_control_plane = next(node for node in self.inventory['nodes'] if 'master' in node['roles'])['address'] + for node in self.inventory['nodes']: + if 'worker' in node['roles']: + self._stub_get_pods(cluster, [first_control_plane], ['calico-node-abc12', 'kube-proxy-34xyz'], node['name']) + + components.wait_for_pods(cluster.nodes['worker']) + + def test_wait_worker_failed(self): + cluster = self._new_cluster() + first_control_plane = next(node for node in self.inventory['nodes'] if 'master' in node['roles'])['address'] + for node in self.inventory['nodes']: + if 'worker' in node['roles']: + self._stub_get_pods(cluster, [first_control_plane], ['calico-node-abc12', 'kube-proxy-34xyz'], + node['name'], ready=False) + + with self.assertRaisesRegex(Exception, re.escape(plugins.ERROR_PODS_NOT_READY)): + components.wait_for_pods(cluster.nodes['worker'].get_any_member()) + + def test_wait_control_planes_successful(self): + cluster = self._new_cluster() + for node in self.inventory['nodes']: + if 'master' in node['roles']: + self._stub_get_pods(cluster, [node['address']], [ + "calico-node-abc12", f"etcd-{node['name']}", + f"kube-apiserver-{node['name']}", f"kube-controller-manager-{node['name']}", + "kube-proxy-34xyz", f"kube-scheduler-{node['name']}", + ], node['name']) + + components.wait_for_pods(cluster.nodes['control-plane']) + + def test_wait_control_plane_failed(self): + cluster = self._new_cluster() + for node in self.inventory['nodes']: + if 'master' in node['roles']: + self._stub_get_pods(cluster, [node['address']], [ + "calico-node-abc12", # f"etcd-{node['name']}", + f"kube-apiserver-{node['name']}", f"kube-controller-manager-{node['name']}", + "kube-proxy-34xyz", f"kube-scheduler-{node['name']}", + ], node['name']) + + with self.assertRaisesRegex(Exception, re.escape(plugins.ERROR_PODS_NOT_READY)): + components.wait_for_pods(cluster.nodes['control-plane'].get_any_member()) + + def test_wait_specific(self): + cluster = self._new_cluster() + for node in self.inventory['nodes']: + if 'master' in node['roles']: + self._stub_get_pods(cluster, [node['address']], [ + "calico-node-abc12", f"etcd-{node['name']}", + f"kube-apiserver-{node['name']}", f"kube-controller-manager-{node['name']}", + "kube-proxy-34xyz", f"kube-scheduler-{node['name']}", + ], node['name']) + + with test_utils.mock_call(plugins.expect_pods) as run: + components.wait_for_pods(cluster.nodes['all'], ['kube-apiserver']) + node_names = {call[1]['node_name'] for call in run.call_args_list} + self.assertEqual({'master-1', 'master-2', 'master-3'}, node_names) + + with test_utils.mock_call(plugins.expect_pods) as run: + components.wait_for_pods(cluster.nodes['all'], ['kube-proxy']) + node_names = {call[1]['node_name'] for call in run.call_args_list} + self.assertEqual({'master-1', 'master-2', 'master-3', 'worker-1', 'worker-2', 'worker-3'}, node_names) + + +class RestartComponentsTest(unittest.TestCase): + def setUp(self): + self.inventory = demo.generate_inventory(**demo.FULLHA) + random.shuffle(self.inventory['nodes']) + + def _new_cluster(self) -> demo.FakeKubernetesCluster: + return demo.new_cluster(self.inventory) + + def test_restart_empty(self): + cluster = self._new_cluster() + components.restart_components(cluster.nodes['all'], []) + + def test_restart_not_supported(self): + cluster = self._new_cluster() + with self.assertRaisesRegex(Exception, re.escape(components.ERROR_RESTART_NOT_SUPPORTED.format( + components=['kube-apiserver/cert-sans', 'kubelet', 'kube-proxy', 'unexpected-component']))): + components.restart_components(cluster.nodes['all'].get_any_member(), components.ALL_COMPONENTS + ['unexpected-component']) + + def test_restart_all_supported(self): + cluster = self._new_cluster() + with test_utils.mock_call(components._restart_containers) as restart_containers, \ + test_utils.mock_call(plugins.expect_pods) as expect_pods: + + all_components = ['kube-apiserver', 'kube-scheduler', 'kube-controller-manager', 'etcd'] + components.restart_components(cluster.nodes['all'], all_components) + + control_plane_components = ['kube-apiserver', 'kube-scheduler', 'kube-controller-manager', 'etcd'] + expected_control_planes = [node['name'] for node in self.inventory['nodes'] if 'master' in node['roles']] + + restart_containers_expected_calls = [(node, control_plane_components) for node in expected_control_planes] + restart_containers_actual_calls = [(call[0][1].get_node_name(), list(call[0][2])) + for call in restart_containers.call_args_list + if call[0][2]] + self.assertEqual(restart_containers_expected_calls, restart_containers_actual_calls) + + actual_called_nodes = [call[1]['node_name'] for call in expect_pods.call_args_list] + self.assertEqual(expected_control_planes, actual_called_nodes) + + for call in expect_pods.call_args_list: + self.assertEqual(control_plane_components, call[0][1]) + + def test_restart_specific(self): + cluster = self._new_cluster() + with test_utils.mock_call(components._restart_containers) as restart_containers, \ + test_utils.mock_call(plugins.expect_pods) as expect_pods: + + components.restart_components(cluster.nodes['control-plane'].get_first_member(), [ + 'kube-apiserver' + ]) + + first_control_plane = next(node for node in self.inventory['nodes'] if 'master' in node['roles']) + + self.assertEqual(1, restart_containers.call_count) + self.assertEqual(first_control_plane['name'], restart_containers.call_args[0][1].get_node_name()) + self.assertEqual(['kube-apiserver'], list(restart_containers.call_args[0][2])) + + self.assertEqual(1, expect_pods.call_count) + self.assertEqual(first_control_plane['name'], expect_pods.call_args[1]['node_name']) + self.assertEqual(['kube-apiserver'], expect_pods.call_args[0][1]) + + +class ReconfigureComponentsTest(unittest.TestCase): + def setUp(self): + self.inventory = demo.generate_inventory(**demo.FULLHA) + random.shuffle(self.inventory['nodes']) + self.control_planes = [node['name'] for node in self.inventory['nodes'] if 'master' in node['roles']] + self.workers = [node['name'] for node in self.inventory['nodes'] if 'worker' in node['roles']] + + self.control_plane_components = ['kube-apiserver', 'kube-scheduler', 'kube-controller-manager', 'etcd'] + + def _new_cluster(self) -> demo.FakeKubernetesCluster: + return demo.new_cluster(self.inventory) + + def test_reconfigure_empty(self): + cluster = self._new_cluster() + components.reconfigure_components(cluster.nodes['all'], []) + + def test_reconfigure_not_supported(self): + cluster = self._new_cluster() + with self.assertRaisesRegex(Exception, re.escape(components.ERROR_RECONFIGURE_NOT_SUPPORTED.format( + components=['unexpected-component']))): + components.reconfigure_components(cluster.nodes['all'].get_any_member(), components.ALL_COMPONENTS + ['unexpected-component']) + + def test_reconfigure_all_supported(self): + for changes_detected, force_restart in ( + ([], False), + (['control-planes'], False), + (['kubelet'], False), + (['kube-proxy'], False), + (['control-planes', 'kubelet', 'kube-proxy'], False), + ([], True) + ): + with self.subTest(f"Changes detected: {changes_detected}, force restart: {force_restart}"): + self._test_reconfigure_all_supported(changes_detected, force_restart) + + def _test_reconfigure_all_supported(self, changes_detected: List[str], force_restart: bool): + cluster = self._new_cluster() + with test_utils.mock_call(components._prepare_nodes_to_reconfigure_components), \ + self._test_reconfigure_apiserver_certsans(), \ + self._test_reconfigure_control_plane('control-planes' in changes_detected, self.control_plane_components), \ + self._test_reconfigure_kubelet('kubelet' in changes_detected), \ + test_utils.mock_call(components._update_configmap, + return_value='kube-proxy' in changes_detected), \ + self._test_restart_kubelet('kubelet' in changes_detected or force_restart), \ + self._test_delete_kube_proxy_pods(force_restart or set(changes_detected) & {'kube-proxy', 'kubelet'}), \ + self._test_restart_containers(self.control_plane_components, True, + 'control-planes' in changes_detected or force_restart, + 'kubelet' in changes_detected or force_restart), \ + self._test_wait_for_pods(self.control_plane_components, True, True): + + components.reconfigure_components(cluster.nodes['all'], components.ALL_COMPONENTS, + force_restart=force_restart) + + def test_reconfigure_apiserver_certsans(self): + cluster = self._new_cluster() + with test_utils.mock_call(components._prepare_nodes_to_reconfigure_components), \ + self._test_reconfigure_apiserver_certsans(), \ + test_utils.mock_call(components._update_configmap, return_value=False), \ + self._test_restart_containers([], True, False, False), \ + self._test_wait_for_pods(['kube-apiserver'], False, False): + + components.reconfigure_components(cluster.nodes['all'], ['kube-apiserver/cert-sans']) + + def test_reconfigure_control_planes_specific(self): + for changes_detected, force_restart in ( + (True, False), + (False, False), + (False, True) + ): + with self.subTest(f"Changes detected: {changes_detected}, force restart: {force_restart}"), \ + test_utils.mock_call(components._prepare_nodes_to_reconfigure_components), \ + self._test_reconfigure_control_plane(changes_detected, ['etcd']), \ + test_utils.mock_call(components._update_configmap, return_value=changes_detected), \ + self._test_restart_containers(['etcd'], False, + changes_detected or force_restart, False), \ + self._test_wait_for_pods(['etcd'], False, False): + + cluster = self._new_cluster() + components.reconfigure_components(cluster.nodes['all'], ['etcd'], + force_restart=force_restart) + + def test_reconfigure_kubelet(self): + for changes_detected, force_restart in ( + (True, False), + (False, False), + (False, True) + ): + with self.subTest(f"Changes detected: {changes_detected}, force restart: {force_restart}"), \ + test_utils.mock_call(components._prepare_nodes_to_reconfigure_components), \ + self._test_reconfigure_kubelet(changes_detected), \ + test_utils.mock_call(components._update_configmap, return_value=changes_detected), \ + self._test_restart_kubelet(changes_detected or force_restart), \ + self._test_delete_kube_proxy_pods(force_restart or changes_detected), \ + self._test_restart_containers([], False, + False, changes_detected or force_restart), \ + self._test_wait_for_pods([], True, False): + + cluster = self._new_cluster() + components.reconfigure_components(cluster.nodes['all'], ['kubelet'], + force_restart=force_restart) + + def test_reconfigure_kube_proxy(self): + for changes_detected, force_restart in ( + (True, False), + (False, False), + (False, True) + ): + with self.subTest(f"Changes detected: {changes_detected}, force restart: {force_restart}"), \ + test_utils.mock_call(components._prepare_nodes_to_reconfigure_components), \ + test_utils.mock_call(components._update_configmap, return_value=changes_detected), \ + self._test_delete_kube_proxy_pods(force_restart or changes_detected), \ + self._test_wait_for_pods([], False, True): + + cluster = self._new_cluster() + components.reconfigure_components(cluster.nodes['all'], ['kube-proxy'], + force_restart=force_restart) + + @contextmanager + def _test_reconfigure_apiserver_certsans(self): + with test_utils.mock_call(components._reconfigure_apiserver_certsans) as mock: + yield + actual_calls = [call[0][0].get_node_name() for call in mock.call_args_list] + self.assertEqual(self.control_planes, actual_calls) + + @contextmanager + def _test_reconfigure_control_plane(self, changes_detected: bool, components_: List[str]): + with test_utils.mock_call(components._reconfigure_control_plane_component, return_value=changes_detected) as mock: + yield + + expected_calls = [(node, component) for node in self.control_planes for component in components_] + actual_calls = [(call[0][1].get_node_name(), call[0][2]) for call in mock.call_args_list] + self.assertEqual(expected_calls, actual_calls) + + @contextmanager + def _test_reconfigure_kubelet(self, changes_detected: bool): + with test_utils.mock_call(components._reconfigure_kubelet, return_value=changes_detected) as mock: + yield + + actual_calls = [call[0][1].get_node_name() for call in mock.call_args_list] + self.assertEqual(self.control_planes + self.workers, actual_calls) + + @contextmanager + def _test_restart_kubelet(self, should_restart: bool): + with test_utils.mock_call(system.restart_service) as mock: + yield + + expected_calls = ((self.control_planes + self.workers) if should_restart else []) + actual_calls = [call[0][0].get_node_name() for call in mock.call_args_list] + self.assertEqual(expected_calls, actual_calls) + + @contextmanager + def _test_delete_kube_proxy_pods(self, should_delete: bool): + with test_utils.mock_call(components._delete_pods) as mock: + yield + + expected_calls = [(node, 'kube-proxy') for node in (self.control_planes + self.workers) + if should_delete] + actual_calls = [(call[0][1].get_node_name(), component) + for call in mock.call_args_list + for component in call[0][3]] + self.assertEqual(expected_calls, actual_calls) + + @contextmanager + def _test_restart_containers(self, control_plane_components: List[str], + configure_certsans: bool, components_restart: bool, kubelet_restart: bool): + with test_utils.mock_call(components._restart_containers) as mock: + yield + + expected_calls = [] + for node in self.control_planes: + expected_components = [] + if configure_certsans: + # It is currently not possible to detect changes in cert SANs, so kube-apiserver is restarted anyway. + expected_components = ['kube-apiserver'] + if components_restart: + expected_components = list(OrderedSet(expected_components + control_plane_components)) + + if expected_components: + expected_calls.append((node, expected_components)) + + if kubelet_restart: + expected_calls.append((node, self.control_plane_components)) + + actual_calls = [(call[0][1].get_node_name(), list(call[0][2])) + for call in mock.call_args_list + if call[0][2]] + + self.assertEqual(expected_calls, actual_calls) + + @contextmanager + def _test_wait_for_pods(self, control_plane_components: List[str], + reconfigure_kubelet: bool, reconfigure_kube_proxy): + with test_utils.mock_call(plugins.expect_pods) as mock: + yield + + expected_calls = [] + for node in self.control_planes: + if control_plane_components: + expected_calls.append((node, control_plane_components)) + if reconfigure_kubelet: + expected_calls.append((node, ['kube-proxy'] + self.control_plane_components)) + elif reconfigure_kube_proxy: + expected_calls.append((node, ['kube-proxy'])) + + expected_calls.extend((node, ['kube-proxy']) for node in self.workers if reconfigure_kubelet or reconfigure_kube_proxy) + actual_calls = [(call[1]['node_name'], call[0][1]) for call in mock.call_args_list] + self.assertEqual(expected_calls, actual_calls) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/unit/test_manage_psp.py b/test/unit/test_manage_psp.py index 018651c4c..0473fa2fe 100644 --- a/test/unit/test_manage_psp.py +++ b/test/unit/test_manage_psp.py @@ -11,12 +11,15 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - +import re import unittest from copy import deepcopy -from kubemarine import demo -from kubemarine.core import errors +from kubemarine import demo, plugins, admission +from kubemarine.core import errors, flow +from kubemarine.kubernetes import components +from kubemarine.procedures import manage_psp +from test.unit import utils as test_utils class EnrichmentValidation(unittest.TestCase): @@ -83,6 +86,111 @@ def _stub_resource(self, kind): } } + def test_inconsistent_config(self): + self.inventory['rbac']['admission'] = 'pss' + with self.assertRaisesRegex(Exception, re.escape(admission.ERROR_INCONSISTENT_INVENTORIES)): + self._create_cluster() + + +class EnrichmentAndFinalization(unittest.TestCase): + def setUp(self): + self.inventory = demo.generate_inventory(**demo.ALLINONE) + self.inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = 'v1.24.11' + self.inventory['rbac'] = { + 'admission': 'psp', + 'psp': { + 'pod-security': 'enabled', + } + } + self.context = demo.create_silent_context(['fake.yaml'], procedure='manage_psp') + self.manage_psp = demo.generate_procedure_inventory('manage_psp') + + def _create_cluster(self): + return demo.new_cluster(self.inventory, procedure_inventory=self.manage_psp, + context=self.context) + + def test_change_psp_state(self): + for target_state_enabled in (False, True): + target_state = 'enabled' if target_state_enabled else 'disabled' + previous_state = 'disabled' if target_state_enabled else 'enabled' + with self.subTest(f"Target state: {target_state}"): + self.inventory['rbac']['psp']['pod-security'] = previous_state + self.manage_psp['psp']['pod-security'] = target_state + + admission_plugins_expected = 'NodeRestriction' + if target_state_enabled: + admission_plugins_expected += ',PodSecurityPolicy' + + cluster = self._create_cluster() + apiserver_extra_args = cluster.inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual(target_state, cluster.inventory['rbac']['psp']['pod-security']) + self.assertEqual(admission_plugins_expected, apiserver_extra_args.get('enable-admission-plugins')) + + test_utils.stub_associations_packages(cluster, {}) + finalized_inventory = test_utils.make_finalized_inventory(cluster) + apiserver_extra_args = finalized_inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual(target_state, finalized_inventory['rbac']['psp']['pod-security']) + self.assertEqual(admission_plugins_expected, apiserver_extra_args.get('enable-admission-plugins')) + + final_inventory = test_utils.get_final_inventory(cluster, self.inventory) + apiserver_extra_args = final_inventory['services'].get('kubeadm', {}).get('apiServer', {}).get('extraArgs', {}) + + self.assertEqual(target_state, final_inventory['rbac']['psp']['pod-security']) + self.assertEqual(None, apiserver_extra_args.get('enable-admission-plugins')) + + +class RunTasks(unittest.TestCase): + def setUp(self): + self.inventory = demo.generate_inventory(**demo.ALLINONE) + self.inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = 'v1.24.11' + self.inventory['rbac'] = { + 'admission': 'psp', + 'psp': { + 'pod-security': 'enabled', + } + } + self.manage_psp = demo.generate_procedure_inventory('manage_psp') + + def _run_tasks(self, tasks_filter: str) -> demo.FakeResources: + context = demo.create_silent_context( + ['fake.yaml', '--tasks', tasks_filter], procedure='manage_psp') + + nodes_context = demo.generate_nodes_context(self.inventory) + resources = demo.FakeResources(context, self.inventory, + procedure_inventory=self.manage_psp, nodes_context=nodes_context) + flow.run_actions(resources, [manage_psp.PSPAction()]) + return resources + + def test_reconfigure_plugin_disable_psp(self): + self.inventory['rbac']['psp']['pod-security'] = 'enabled' + self.manage_psp['psp']['pod-security'] = 'disabled' + with test_utils.mock_call(components._prepare_nodes_to_reconfigure_components), \ + test_utils.mock_call(admission.delete_privileged_policy), \ + test_utils.mock_call(admission.manage_policies), \ + test_utils.mock_call(components._reconfigure_control_plane_component, return_value=True) as reconfigure_control_plane, \ + test_utils.mock_call(components._update_configmap, return_value=True), \ + test_utils.mock_call(components._restart_containers) as restart_containers, \ + test_utils.mock_call(plugins.expect_pods) as expect_pods: + res = self._run_tasks('reconfigure_psp') + + self.assertTrue(reconfigure_control_plane.called, + "There should be a successful attempt to reconfigure kube-apiserver") + + self.assertTrue(restart_containers.called) + self.assertEqual(['kube-apiserver'], restart_containers.call_args[0][2], + "kube-apiserver should be restarted") + + self.assertTrue(expect_pods.called) + self.assertEqual(['kube-apiserver'], expect_pods.call_args[0][1], + "kube-apiserver pods should be waited for") + + admission_plugins_expected = 'NodeRestriction' + apiserver_extra_args = res.last_cluster.inventory['services']['kubeadm']['apiServer']['extraArgs'] + self.assertEqual(admission_plugins_expected, apiserver_extra_args.get('enable-admission-plugins'), + "Unexpected apiserver extra args") + if __name__ == '__main__': unittest.main() diff --git a/test/unit/test_manage_pss.py b/test/unit/test_manage_pss.py index 9bce45e3e..86875d23a 100644 --- a/test/unit/test_manage_pss.py +++ b/test/unit/test_manage_pss.py @@ -11,12 +11,14 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - +import re import unittest from copy import deepcopy -from kubemarine import demo -from kubemarine.core import errors +from kubemarine import demo, admission, plugins +from kubemarine.core import errors, flow +from kubemarine.kubernetes import components +from kubemarine.procedures import manage_pss from test.unit import utils @@ -73,6 +75,12 @@ def test_invalid_namespaces_defaults_profile(self): with self.assertRaisesRegex(errors.FailException, r"Value should be one of \['privileged', 'baseline', 'restricted']"): self._create_cluster() + def test_inconsistent_config(self): + self.inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = 'v1.24.11' + self.inventory['rbac']['admission'] = 'psp' + with self.assertRaisesRegex(Exception, re.escape(admission.ERROR_INCONSISTENT_INVENTORIES)): + self._create_cluster() + class EnrichmentAndFinalization(unittest.TestCase): def setUp(self): @@ -110,6 +118,130 @@ def test_merge_exemptions(self): final_inventory = utils.get_final_inventory(cluster, self.inventory) self.assertEqual(['a', 'b', 'c'], final_inventory['rbac']['pss']['exemptions']['namespaces']) + def test_disable_pss_dont_enrich_feature_gates(self): + self.inventory['rbac']['pss']['pod-security'] = 'enabled' + self.manage_pss['pss']['pod-security'] = 'disabled' + + cluster = self._create_cluster() + apiserver_extra_args = cluster.inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual('disabled', cluster.inventory['rbac']['pss']['pod-security']) + self.assertEqual(None, apiserver_extra_args.get('feature-gates')) + self.assertEqual(None, apiserver_extra_args.get('admission-control-config-file')) + + utils.stub_associations_packages(cluster, {}) + finalized_inventory = utils.make_finalized_inventory(cluster) + apiserver_extra_args = finalized_inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual('disabled', finalized_inventory['rbac']['pss']['pod-security']) + self.assertEqual(None, apiserver_extra_args.get('feature-gates')) + self.assertEqual(None, apiserver_extra_args.get('admission-control-config-file')) + + final_inventory = utils.get_final_inventory(cluster, self.inventory) + apiserver_extra_args = final_inventory['services'].get('kubeadm', {}).get('apiServer', {}).get('extraArgs', {}) + + self.assertEqual('disabled', final_inventory['rbac']['pss']['pod-security']) + self.assertEqual(None, apiserver_extra_args.get('feature-gates')) + self.assertEqual(None, apiserver_extra_args.get('admission-control-config-file')) + + def test_enable_pss_conditional_enrich_feature_gates(self): + for k8s_version, feature_gates_enriched in (('v1.27.8', True), ('v1.28.4', False)): + with self.subTest(f"Kubernetes: {k8s_version}"): + self.inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = k8s_version + self.inventory['rbac']['pss']['pod-security'] = 'disabled' + self.manage_pss['pss']['pod-security'] = 'enabled' + + feature_gates_expected = 'PodSecurity=true' if feature_gates_enriched else None + + cluster = self._create_cluster() + apiserver_extra_args = cluster.inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual('enabled', cluster.inventory['rbac']['pss']['pod-security']) + self.assertEqual(feature_gates_expected, apiserver_extra_args.get('feature-gates')) + self.assertEqual('/etc/kubernetes/pki/admission.yaml', apiserver_extra_args.get('admission-control-config-file')) + + utils.stub_associations_packages(cluster, {}) + finalized_inventory = utils.make_finalized_inventory(cluster) + apiserver_extra_args = finalized_inventory["services"]["kubeadm"]['apiServer']['extraArgs'] + + self.assertEqual('enabled', finalized_inventory['rbac']['pss']['pod-security']) + self.assertEqual(feature_gates_expected, apiserver_extra_args.get('feature-gates')) + self.assertEqual('/etc/kubernetes/pki/admission.yaml', apiserver_extra_args.get('admission-control-config-file')) + self.assertNotIn('psp', finalized_inventory['rbac']) + + final_inventory = utils.get_final_inventory(cluster, self.inventory) + apiserver_extra_args = final_inventory['services'].get('kubeadm', {}).get('apiServer', {}).get('extraArgs', {}) + + self.assertEqual('enabled', final_inventory['rbac']['pss']['pod-security']) + self.assertEqual(None, apiserver_extra_args.get('feature-gates')) + self.assertEqual(None, apiserver_extra_args.get('admission-control-config-file')) + + +class RunTasks(unittest.TestCase): + def setUp(self): + self.inventory = demo.generate_inventory(**demo.ALLINONE) + self.inventory['rbac'] = { + 'admission': 'pss', + 'pss': { + 'pod-security': 'enabled', + 'defaults': {}, + } + } + self.manage_pss = demo.generate_procedure_inventory('manage_pss') + + def _run_tasks(self, tasks_filter: str) -> demo.FakeResources: + context = demo.create_silent_context( + ['fake.yaml', '--tasks', tasks_filter], procedure='manage_pss') + + nodes_context = demo.generate_nodes_context(self.inventory) + resources = demo.FakeResources(context, self.inventory, + procedure_inventory=self.manage_pss, nodes_context=nodes_context) + flow.run_actions(resources, [manage_pss.PSSAction()]) + return resources + + def test_manage_pss_enable_pss(self): + self.inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = 'v1.27.8' + self.inventory['rbac']['pss']['pod-security'] = 'disabled' + self.manage_pss['pss']['pod-security'] = 'enabled' + with utils.mock_call(admission.label_namespace_pss), \ + utils.mock_call(admission.copy_pss), \ + utils.mock_call(components.reconfigure_components) as run: + res = self._run_tasks('manage_pss') + + self.assertTrue(run.called) + self.assertEqual(['kube-apiserver'], run.call_args[1]['components'], + "kube-apiserver was not reconfigured") + + apiserver_extra_args = res.last_cluster.inventory['services']['kubeadm']['apiServer']['extraArgs'] + self.assertEqual('PodSecurity=true', apiserver_extra_args.get('feature-gates'), + "Unexpected apiserver extra args") + + def test_manage_pss_change_configuration_restart(self): + self.inventory['rbac']['pss']['pod-security'] = 'enabled' + self.inventory['rbac']['pss']['defaults']['enforce'] = 'baseline' + self.manage_pss['pss']['pod-security'] = 'enabled' + self.manage_pss['pss']['defaults'] = {'enforce': 'restricted'} + + with utils.mock_call(admission.label_namespace_pss), \ + utils.mock_call(admission.copy_pss), \ + utils.mock_call(components._prepare_nodes_to_reconfigure_components), \ + utils.mock_call(components._reconfigure_control_plane_component, return_value=False) as reconfigure_control_plane, \ + utils.mock_call(components._update_configmap, return_value=True), \ + utils.mock_call(components._restart_containers) as restart_containers, \ + utils.mock_call(plugins.expect_pods) as expect_pods: + self._run_tasks('manage_pss') + + self.assertTrue(reconfigure_control_plane.called, + "There should be an attempt to reconfigure kube-apiserver, but nothing is changed") + + self.assertTrue(restart_containers.called) + self.assertEqual(['kube-apiserver'], restart_containers.call_args[0][2], + "kube-apiserver should be restarted") + + self.assertTrue(expect_pods.called) + self.assertEqual(['kube-apiserver'], expect_pods.call_args[0][1], + "kube-apiserver pods should be waited for") + if __name__ == '__main__': unittest.main() diff --git a/test/unit/test_reconfigure.py b/test/unit/test_reconfigure.py new file mode 100644 index 000000000..0f37532bf --- /dev/null +++ b/test/unit/test_reconfigure.py @@ -0,0 +1,281 @@ +import re +import unittest + +from kubemarine import demo, kubernetes +from kubemarine.core import flow +from kubemarine.procedures import reconfigure +from test.unit import utils as test_utils + + +class ReconfigureKubeadmEnrichment(unittest.TestCase): + def setUp(self): + self.setUpScheme(demo.ALLINONE) + + def setUpScheme(self, scheme: dict): + self.inventory = demo.generate_inventory(**scheme) + self.context = demo.create_silent_context(['fake.yaml'], procedure='reconfigure') + + self.reconfigure = demo.generate_procedure_inventory('reconfigure') + self.reconfigure['services'] = {} + + def new_cluster(self) -> demo.FakeKubernetesCluster: + return demo.new_cluster(self.inventory, procedure_inventory=self.reconfigure, context=self.context) + + def test_enrich_and_finalize_inventory(self): + self.inventory['services']['kubeadm'] = { + 'kubernetesVersion': 'v1.25.7', + 'apiServer': { + 'extraArgs': {'api_k1': 'api_v1'}, + 'extraVolumes': [{'name': 'api_name1', 'hostPath': '/home/path', 'mountPath': '/mount/path'}], + 'certSANs': ['san1'], + 'timeoutForControlPlane': '4m0s', + }, + 'scheduler': { + 'extraArgs': {'sched_key1': 'sched_v1'}, + }, + 'controllerManager': { + 'extraVolumes': [{'name': 'ctrl_name1', 'hostPath': '/home/path', 'mountPath': '/mount/path'}], + }, + 'etcd': {'local': { + 'extraArgs': {'etcd_k1': 'etcd_v1'}, + 'imageTag': '1.2.3' + }}, + } + self.reconfigure['services']['kubeadm'] = { + 'apiServer': { + 'extraArgs': {'api_k1': 'api_v1_new', 'api_k2': 'api_v2_new'}, + 'extraVolumes': [ + {'<<': 'merge'}, + {'name': 'api_name2_new', 'hostPath': '/home/path', 'mountPath': '/mount/path'} + ], + 'certSANs': ['san2_new'], + 'timeoutForControlPlane': '5m0s', + }, + 'scheduler': { + 'extraVolumes': [{'name': 'sched_name1_new', 'hostPath': '/home/path', 'mountPath': '/mount/path'}], + }, + 'controllerManager': { + 'extraArgs': {'ctrl_k1': 'ctrl_k1_new'}, + }, + 'etcd': {'local': { + 'extraArgs': {'etcd_k1': 'etcd_v1_new'}, + }}, + } + + self.inventory['services']['kubeadm_kubelet'] = { + 'enableDebuggingHandlers': False, + 'serializeImagePulls': True + } + self.reconfigure['services']['kubeadm_kubelet'] = { + 'serializeImagePulls': False + } + + self.inventory['services']['kubeadm_kube-proxy'] = { + 'logging': {'format': 'test-format'} + } + self.reconfigure['services']['kubeadm_kube-proxy'] = { + 'logging': {'verbosity': 5} + } + + self.inventory['services']['kubeadm_patches'] = { + 'apiServer': [{'groups': ['control-plane'], 'patch': {'api_kp1': 'api_vp1'}}] + } + self.reconfigure['services']['kubeadm_patches'] = { + 'apiServer': [ + {'nodes': ['master-1'], 'patch': {'api_kp2': 'api_vp2_new'}}, + {'<<': 'merge'} + ], + 'kubelet': [ + {'groups': ['worker'], 'patch': {'kubelet_kp1': 'kubelet_vp1_new'}} + ] + } + + cluster = self.new_cluster() + services = cluster.inventory['services'] + self._test_enrich_and_finalize_inventory_check(services, True) + + test_utils.stub_associations_packages(cluster, {}) + services = test_utils.make_finalized_inventory(cluster)['services'] + self._test_enrich_and_finalize_inventory_check(services, True) + + services = test_utils.get_final_inventory(cluster, self.inventory)['services'] + self._test_enrich_and_finalize_inventory_check(services, False) + + def _test_enrich_and_finalize_inventory_check(self, services: dict, enriched: bool): + apiserver_args = services['kubeadm']['apiServer']['extraArgs'].items() + self.assertIn(('api_k1', 'api_v1_new'), apiserver_args) + self.assertIn(('api_k2', 'api_v2_new'), apiserver_args) + self.assertEqual(enriched, ('profiling', 'false') in apiserver_args) + + apiserver_volumes = services['kubeadm']['apiServer']['extraVolumes'] + self.assertIn({'name': 'api_name1', 'hostPath': '/home/path', 'mountPath': '/mount/path'}, apiserver_volumes) + self.assertIn({'name': 'api_name2_new', 'hostPath': '/home/path', 'mountPath': '/mount/path'}, apiserver_volumes) + + apiserver_certsans = services['kubeadm']['apiServer']['certSANs'] + self.assertNotIn('san1', apiserver_certsans) + self.assertIn('san2_new', apiserver_certsans) + self.assertEqual(enriched, 'master-1' in apiserver_certsans) + + self.assertEqual('5m0s', services['kubeadm']['apiServer']['timeoutForControlPlane']) + + scheduler_args = services['kubeadm']['scheduler']['extraArgs'].items() + self.assertIn(('sched_key1', 'sched_v1'), scheduler_args) + self.assertEqual(enriched, ('profiling', 'false') in scheduler_args) + + scheduler_volumes = services['kubeadm']['scheduler']['extraVolumes'] + self.assertIn({'name': 'sched_name1_new', 'hostPath': '/home/path', 'mountPath': '/mount/path'}, scheduler_volumes) + + ctrl_args = services['kubeadm']['controllerManager']['extraArgs'].items() + self.assertIn(('ctrl_k1', 'ctrl_k1_new'), ctrl_args) + self.assertEqual(enriched, ('profiling', 'false') in ctrl_args) + + ctrl_volumes = services['kubeadm']['controllerManager']['extraVolumes'] + self.assertIn({'name': 'ctrl_name1', 'hostPath': '/home/path', 'mountPath': '/mount/path'}, ctrl_volumes) + + etcd_args = services['kubeadm']['etcd']['local']['extraArgs'].items() + self.assertIn(('etcd_k1', 'etcd_v1_new'), etcd_args) + + self.assertEqual('1.2.3', services['kubeadm']['etcd']['local']['imageTag']) + + kubelet = services['kubeadm_kubelet'] + self.assertEqual(False, kubelet['enableDebuggingHandlers']) + self.assertEqual(False, kubelet['serializeImagePulls']) + self.assertEqual(enriched, kubelet.get('cgroupDriver') == 'systemd') + + kube_proxy = services['kubeadm_kube-proxy'] + self.assertEqual({'format': 'test-format', 'verbosity': 5}, kube_proxy['logging']) + + kubeadm_patches = services['kubeadm_patches'] + self.assertEqual([ + {'nodes': ['master-1'], 'patch': {'api_kp2': 'api_vp2_new'}}, + {'groups': ['control-plane'], 'patch': {'api_kp1': 'api_vp1'}} + ], kubeadm_patches['apiServer']) + + self.assertEqual([{'groups': ['worker'], 'patch': {'kubelet_kp1': 'kubelet_vp1_new'}}], kubeadm_patches['kubelet']) + + def test_pss_managed_arg_not_redefined(self): + self.inventory.setdefault('rbac', {})['admission'] = 'pss' + self.inventory['rbac']['pss'] = {'pod-security': 'enabled'} + self.reconfigure['services']['kubeadm'] = { + 'apiServer': { + 'extraArgs': {'admission-control-config-file': '/some/redefined/path'}, + }, + } + + inventory = self.new_cluster().inventory + # This is a potential subject for change. + # The behaviour just follows historical behaviour if installation procedure. + self.assertEqual('/etc/kubernetes/pki/admission.yaml', + inventory['services']['kubeadm']['apiServer']['extraArgs']['admission-control-config-file']) + + def test_error_control_plane_patch_refers_worker(self): + self.setUpScheme(demo.FULLHA) + self.reconfigure['services']['kubeadm_patches'] = { + 'apiServer': [ + {'nodes': ['worker-1'], 'patch': {'key': 'value'}}, + ] + } + with self.assertRaisesRegex( + Exception, re.escape(kubernetes.ERROR_CONTROL_PLANE_PATCH_NOT_CONTROL_PLANE_NODE % 'apiServer')): + self.new_cluster() + + def test_error_kubelet_patch_refers_balancer(self): + self.setUpScheme(demo.FULLHA) + self.inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = 'v1.25.7' + self.reconfigure['services']['kubeadm_patches'] = { + 'kubelet': [ + {'nodes': ['balancer-1'], 'patch': {'key': 'value'}}, + ] + } + with self.assertRaisesRegex( + Exception, re.escape(kubernetes.ERROR_KUBELET_PATCH_NOT_KUBERNETES_NODE % 'kubelet')): + self.new_cluster() + + def test_kubeadm_before_v1_25x_supports_patches(self): + kubernetes_version = 'v1.24.11' + self.inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = kubernetes_version + self.reconfigure['services']['kubeadm_patches'] = { + 'apiServer': [ + {'nodes': ['master-1'], 'patch': {'api_key': 'api_value'}}, + ], + 'etcd': [ + {'groups': ['control-plane'], 'patch': {'etcd_key': 'api_value'}}, + ] + } + # No error should be raised + self.new_cluster() + + def test_error_kubeadm_before_v1_25x_dont_support_patches_kubelet(self): + kubernetes_version = 'v1.24.11' + self.inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = kubernetes_version + self.reconfigure['services']['kubeadm_patches'] = { + 'kubelet': [ + {'nodes': ['master-1'], 'patch': {'key': 'value'}}, + ] + } + with self.assertRaisesRegex( + Exception, re.escape(kubernetes.ERROR_KUBEADM_DOES_NOT_SUPPORT_PATCHES_KUBELET.format(version=kubernetes_version))): + self.new_cluster() + + +class RunTasks(unittest.TestCase): + def setUp(self): + self.inventory = demo.generate_inventory(**demo.ALLINONE) + self.reconfigure = demo.generate_procedure_inventory('reconfigure') + self.reconfigure.setdefault('services', {}) + + def _run_tasks(self, tasks_filter: str) -> demo.FakeResources: + context = demo.create_silent_context( + ['fake.yaml', '--tasks', tasks_filter], procedure='reconfigure') + + nodes_context = demo.generate_nodes_context(self.inventory) + resources = demo.FakeResources(context, self.inventory, + procedure_inventory=self.reconfigure, nodes_context=nodes_context) + flow.run_actions(resources, [reconfigure.ReconfigureAction()]) + return resources + + def test_kubernetes_reconfigure_empty_procedure_inventory(self): + self._run_tasks('deploy.kubernetes.reconfigure') + + def test_kubernetes_reconfigure_empty_sections(self): + self.reconfigure['services'] = { + 'kubeadm': {'apiServer': {}, 'scheduler': {}, 'controllerManager': {}, 'etcd': {}}, + 'kubeadm_kubelet': {}, + 'kubeadm_kube-proxy': {}, + } + with test_utils.mock_call(kubernetes.components.reconfigure_components) as run: + self._run_tasks('deploy.kubernetes.reconfigure') + + actual_called = run.call_args[1]['components'] if run.called else [] + expected_called = ['kube-apiserver', 'kube-scheduler', 'kube-controller-manager', 'etcd', 'kubelet', 'kube-proxy'] + self.assertEqual(expected_called, actual_called, + "Unexpected list of components to reconfigure") + + def test_kubernetes_reconfigure_empty_patch_sections(self): + self.inventory['services'].setdefault('kubeadm', {})['kubernetesVersion'] = 'v1.25.7' + self.reconfigure.setdefault('services', {})['kubeadm_patches'] = { + 'apiServer': [], 'scheduler': [], 'controllerManager': [], 'etcd': [], 'kubelet': [], + } + with test_utils.mock_call(kubernetes.components.reconfigure_components) as run: + self._run_tasks('deploy.kubernetes.reconfigure') + + actual_called = run.call_args[1]['components'] if run.called else [] + expected_called = ['kube-apiserver', 'kube-scheduler', 'kube-controller-manager', 'etcd', 'kubelet'] + self.assertEqual(expected_called, actual_called, + "Unexpected list of components to reconfigure") + + def test_kubernetes_reconfigure_empty_apiserver_certsans(self): + self.reconfigure['services'] = { + 'kubeadm': {'apiServer': {'certSANs': []}} + } + with test_utils.mock_call(kubernetes.components.reconfigure_components) as run: + self._run_tasks('deploy.kubernetes.reconfigure') + + actual_called = run.call_args[1]['components'] if run.called else [] + expected_called = ['kube-apiserver/cert-sans', 'kube-apiserver'] + self.assertEqual(expected_called, actual_called, + "Unexpected list of components to reconfigure") + + +if __name__ == '__main__': + unittest.main() diff --git a/test/unit/test_upgrade.py b/test/unit/test_upgrade.py index 4d7115cba..91abafbd9 100755 --- a/test/unit/test_upgrade.py +++ b/test/unit/test_upgrade.py @@ -13,15 +13,18 @@ # See the License for the specific language governing permissions and # limitations under the License. import itertools +import json import random import re import unittest from copy import deepcopy -from typing import List, Optional, Tuple +from typing import List, Optional, Tuple, Set + +import yaml from kubemarine import kubernetes from kubemarine.core import errors, utils as kutils, static, flow -from kubemarine.procedures import upgrade +from kubemarine.procedures import upgrade, install from kubemarine import demo from test.unit import utils @@ -663,5 +666,160 @@ def test_iterative_sandbox_image_redefinition(self): "Containerd config was not redefined in recreated inventory.") +class RunTasks(unittest.TestCase): + def setUpVersions(self, old: str, new: str): + self.old = old + self.new = new + self.inventory = demo.generate_inventory(**demo.ALLINONE) + self.inventory['services']['kubeadm'] = { + 'kubernetesVersion': old + } + self.nodes_context = demo.generate_nodes_context(self.inventory) + self.upgrade = demo.generate_procedure_inventory('upgrade') + self.upgrade['upgrade_plan'] = [new] + + self.fake_shell = demo.FakeShell() + self.fake_fs = demo.FakeFS() + + def _run_tasks(self, tasks_filter: str) -> demo.FakeResources: + context = demo.create_silent_context(['fake_path.yaml', '--tasks', tasks_filter], procedure='upgrade') + + resources = demo.FakeResources(context, self.inventory, + procedure_inventory=self.upgrade, nodes_context=self.nodes_context, + fake_shell=self.fake_shell, fake_fs=self.fake_fs) + + kubernetes_nodes = [node['name'] for node in self._get_nodes({'worker', 'master', 'control-plane'})] + with utils.mock_call(kubernetes.autodetect_non_upgraded_nodes, return_value=kubernetes_nodes): + flow.run_actions(resources, [upgrade.UpgradeAction(self.new, True)]) + + return resources + + def _run_kubernetes_task(self) -> demo.FakeResources: + with utils.mock_call(kubernetes.upgrade_first_control_plane), \ + utils.mock_call(install.deploy_coredns), \ + utils.mock_call(kubernetes.upgrade_other_control_planes), \ + utils.mock_call(kubernetes.upgrade_workers), \ + utils.mock_call(upgrade.kubernetes_cleanup_nodes_versions): + return self._run_tasks('kubernetes') + + def _stub_load_configmap(self, configmap: str, data: dict) -> None: + first_control_plane = self._first_control_plane()['address'] + results = demo.create_hosts_result([first_control_plane], stdout=json.dumps(data)) + cmd = f'kubectl get configmap -n kube-system {configmap} -o json' + self.fake_shell.add(results, 'sudo', [cmd]) + + def _get_nodes(self, roles: Set[str]) -> List[dict]: + return [node for node in self.inventory['nodes'] if set(node['roles']) & roles] + + def _first_control_plane(self) -> dict: + return self._get_nodes({'master', 'control-plane'})[0] + + def test_kubernetes_preconfigure_apiserver_feature_gates_if_necessary(self): + for old, new, expected_called in ( + ('v1.26.11', 'v1.27.8', False), + ('v1.27.8', 'v1.28.4', True), + ('v1.28.3', 'v1.28.4', False), + ): + with self.subTest(f"old: {old}, new: {new}"), \ + utils.mock_call(kubernetes.components.reconfigure_components) as run: + self.setUpVersions(old, new) + self.inventory.setdefault('rbac', {}).update({ + 'admission': 'pss', 'pss': {'pod-security': 'enabled'} + }) + + res = self._run_kubernetes_task() + + actual_called = run.called and 'kube-apiserver' in run.call_args[1]['components'] + + self.assertEqual(expected_called, actual_called, + f"kube-apiserver was {'not' if expected_called else 'unexpectedly'} preconfigured") + + apiserver_extra_args = res.last_cluster.inventory['services']['kubeadm']['apiServer']['extraArgs'] + feature_gates_expected = 'PodSecurity=true' if kutils.version_key(new)[:2] < (1, 28) else None + self.assertEqual(feature_gates_expected, apiserver_extra_args.get('feature-gates'), + "Unexpected apiserver extra args") + + def test_kubernetes_preconfigure_apiserver_feature_gates_edit_func(self): + for custom_feature_gates in ('ServiceAccountIssuerDiscovery=true', None): + with self.subTest(f"custom feature-gates: {bool(custom_feature_gates)}"), \ + utils.mock_call(kubernetes.components._prepare_nodes_to_reconfigure_components), \ + utils.mock_call(kubernetes.components._reconfigure_control_plane_components), \ + utils.mock_call(kubernetes.components._update_configmap, return_value=True): + self.setUpVersions('v1.27.8', 'v1.28.4') + self.inventory.setdefault('rbac', {}).update({ + 'admission': 'pss', 'pss': {'pod-security': 'enabled'} + }) + initial_feature_gates = 'PodSecurity=true' + if custom_feature_gates: + self.inventory['services']['kubeadm'].update({'apiServer': {'extraArgs': {'feature-gates': custom_feature_gates}}}) + initial_feature_gates = custom_feature_gates + ',' + initial_feature_gates + + self._stub_load_configmap('kubeadm-config', {'data': {'ClusterConfiguration': yaml.dump({ + 'kind': 'ClusterConfiguration', + 'kubernetesVersion': self.old, + 'apiServer': {'extraArgs': {'feature-gates': initial_feature_gates}} + })}}) + self._run_kubernetes_task() + + upload_config = self.fake_fs.read(self._first_control_plane()['address'], '/etc/kubernetes/upload-config.yaml') + cluster_config = next(filter(lambda cfg: cfg['kind'] == 'ClusterConfiguration', yaml.safe_load_all(upload_config))) + + actual_extra_args = cluster_config['apiServer']['extraArgs'] + self.assertEqual(custom_feature_gates, actual_extra_args.get('feature-gates'), + "Unexpected preconfigured kube-apiserver feature gates") + + self.assertEqual(self.old, cluster_config['kubernetesVersion'], + "Kubernetes version should not change during preconfiguring of kube-apiserver") + + def test_kubernetes_preconfigure_kube_proxy_conntrack_min_if_necessary(self): + for old, new, expected_called in ( + ('v1.27.8', 'v1.28.4', False), + ('v1.28.4', 'v1.29.1', True), + ): + with self.subTest(f"old: {old}, new: {new}"), \ + utils.mock_call(kubernetes.components.reconfigure_components) as run: + self.setUpVersions(old, new) + + res = self._run_kubernetes_task() + + actual_called = run.called and 'kube-proxy' in run.call_args[1]['components'] + + self.assertEqual(expected_called, actual_called, + f"kube-proxy was {'not' if expected_called else 'unexpectedly'} preconfigured") + + conntrack_min_actual = res.last_cluster.inventory['services']['kubeadm_kube-proxy'].get('conntrack', {}).get('min') + conntrack_min_expected = None if kutils.version_key(new)[:2] < (1, 29) else 1000000 + self.assertEqual(conntrack_min_expected, conntrack_min_actual, + "Unexpected kubeadm_kube-proxy.conntrack.min") + + def test_kubernetes_preconfigure_kube_proxy_conntrack_min_edit_func(self): + with utils.mock_call(kubernetes.components._prepare_nodes_to_reconfigure_components), \ + utils.mock_call(kubernetes.components._reconfigure_node_components), \ + utils.mock_call(kubernetes.components._update_configmap, return_value=True), \ + utils.mock_call(kubernetes.components._kube_proxy_configmap_uploader) as kube_proxy_uploader: + self.setUpVersions('v1.28.4', 'v1.29.1') + + self._stub_load_configmap('kube-proxy', {'data': {'config.conf': yaml.dump({ + 'kind': 'KubeProxyConfiguration', + 'conntrack': {'min': None} + })}}) + self._run_kubernetes_task() + + self.assertTrue(kube_proxy_uploader.called, "kube-proxy ConfigMap was not updated") + + kubeadm_config: kubernetes.components.KubeadmConfig = kube_proxy_uploader.call_args[0][1] + + self.assertTrue(kubeadm_config.is_loaded('kube-proxy'), "kube-proxy ConfigMap should already be loaded") + + conntrack_min_actual = kubeadm_config.maps['kube-proxy'].get('conntrack', {}).get('min') + self.assertEqual(1000000, conntrack_min_actual, + "Unexpected preconfigured kube-proxy conntrack.min") + + conntrack_min_actual = yaml.safe_load(kubeadm_config.loaded_maps['kube-proxy'].obj['data']['config.conf'])\ + .get('conntrack', {}).get('min') + self.assertEqual(1000000, conntrack_min_actual, + "Unexpected preconfigured kube-proxy conntrack.min") + + if __name__ == '__main__': unittest.main() diff --git a/test/unit/utils.py b/test/unit/utils.py index bddec301e..e20168a70 100644 --- a/test/unit/utils.py +++ b/test/unit/utils.py @@ -11,10 +11,13 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import inspect import unittest from contextlib import contextmanager from copy import deepcopy -from typing import Dict +from types import FunctionType +from typing import Dict, Iterator, Callable, cast +from unittest import mock from kubemarine import demo, packages from kubemarine.core import utils, errors, static @@ -83,3 +86,13 @@ def backup_globals(): yield finally: static.GLOBALS = backup + + +@contextmanager +def mock_call(call: Callable, return_value: object = None) -> Iterator[mock.MagicMock]: + func = cast(FunctionType, call) + name = func.__name__ + module = inspect.getmodule(func) + with mock.patch.object(module, name, return_value=return_value) as run: + run.__name__ = name + yield run