Skip to content

Commit

Permalink
Add documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@ilia1243
ilia1243 committed Feb 7, 2024
1 parent 2a87312 commit 495a513
Show file tree
Hide file tree
Showing 3 changed files with 195 additions and 24 deletions.
3 changes: 2 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,14 @@ Kubemarine is an open source, lightweight and powerful management tool built for
- [upgrade](documentation/Maintenance.md#upgrade-procedure)
- [backup](documentation/Maintenance.md#backup-procedure)
- [restore](documentation/Maintenance.md#restore-procedure)
- [reconfigure](documentation/Maintenance.md#reconfigure-procedure)
- [check_iaas](documentation/Kubecheck.md#iaas-procedure)
- [check_paas](documentation/Kubecheck.md#paas-procedure)
- [migrate_kubemarine](documentation/Maintenance.md#kubemarine-migration-procedure)
- [manage_psp](documentation/Maintenance.md#manage-psp-procedure)
- [manage_pss](documentation/Maintenance.md#manage-pss-procedure)
- [cert_renew](documentation/Maintenance.md#certificate-renew-procedure)
- [migrate_cri](documentation/Maintenance.md#migration-cri-procedure)
- [migrate_cri](documentation/Maintenance.md#cri-migration-procedure)
- [Single cluster inventory](documentation/Installation.md#configuration) for all operations, highly customizable
- Default values of all parameters in configurations with a minimum of required parameters
- [Control planes balancing](documentation/Installation.md#full-ha-scheme) with external balancers and VRRP
Expand Down
53 changes: 30 additions & 23 deletions documentation/Installation.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -498,7 +498,7 @@ For more information about the structure of the inventory and how to specify the
* [Minimal All-in-one Inventory Example](../examples/cluster.yaml/allinone-cluster.yaml) - It provides the minimum set of parameters for deploying All-in-one scheme.
* [Minimal Mini-HA Inventory Example](../examples/cluster.yaml/miniha-cluster.yaml) - It provides the minimum set of parameters for deploying Mini-HA scheme.
#### Inventory validation
#### Inventory validation
When configuring the inventory, you can use your favorite IDE supporting YAML validation by JSON schema.
JSON schema for inventory file can be used by [URL](../kubemarine/resources/schemas/cluster.json?raw=1).
Expand Down Expand Up @@ -1086,23 +1086,25 @@ In the `services.kubeadm` section, you can override the original settings for ku
For more information about these settings, refer to the official Kubernetes documentation at [https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file).
By default, the installer uses the following parameters:

|Parameter| Default Value |
|---|----------------------------------------------------------|
|kubernetesVersion| `v1.26.11` |
|controlPlaneEndpoint| `{{ cluster_name }}:6443` |
|networking.podSubnet| `10.128.0.0/14` for IPv4 or `fd02::/48` for IPv6 |
|networking.serviceSubnet| `172.30.0.0/16` for IPv4 or `fd03::/112` for IPv6 |
|apiServer.certSANs| List with all nodes internal IPs, external IPs and names |
|apiServer.extraArgs.enable-admission-plugins| `NodeRestriction` |
|apiServer.extraArgs.profiling| `false` |
|apiServer.extraArgs.audit-log-path| `/var/log/kubernetes/audit/audit.log` |
|apiServer.extraArgs.audit-policy-file| `/etc/kubernetes/audit-policy.yaml` |
|apiServer.extraArgs.audit-log-maxage| `30` |
|apiServer.extraArgs.audit-log-maxbackup| `10` |
|apiServer.extraArgs.audit-log-maxsize| `100` |
|scheduler.extraArgs.profiling| `false` |
|controllerManager.extraArgs.profiling| `false` |
|controllerManager.extraArgs.terminated-pod-gc-threshold| `1000` |
| Parameter | Default Value | Description |
|-------------------------------------------------------|----------------------------------------------------------|--------------------------------------------------------------------------------------------------|
|kubernetesVersion | `v1.26.11` | |
|controlPlaneEndpoint | `{{ cluster_name }}:6443` | |
|networking.podSubnet | `10.128.0.0/14` for IPv4 or `fd02::/48` for IPv6 | |
|networking.serviceSubnet | `172.30.0.0/16` for IPv4 or `fd03::/112` for IPv6 | |
|apiServer.certSANs | List with all nodes internal IPs, external IPs and names | Custom SANs are only appended to, but do not override the default list |
|apiServer.extraArgs.enable-admission-plugins | `NodeRestriction` | `PodSecurityPolicy` plugin is added if [Admission psp](#admission-psp) is enabled |
|apiServer.extraArgs.feature-gates | | `PodSecurity=true` is added for Kubernetes < v1.28 if [Admission pss](#admission-pss) is enabled |
|apiServer.extraArgs.admission-control-config-file | `/etc/kubernetes/pki/admission.yaml` | Provided default value **overrides** custom value if [Admission pss](#admission-pss) is enabled. |
|apiServer.extraArgs.profiling | `false` | |
|apiServer.extraArgs.audit-log-path | `/var/log/kubernetes/audit/audit.log` | |
|apiServer.extraArgs.audit-policy-file | `/etc/kubernetes/audit-policy.yaml` | |
|apiServer.extraArgs.audit-log-maxage | `30` | |
|apiServer.extraArgs.audit-log-maxbackup | `10` | |
|apiServer.extraArgs.audit-log-maxsize | `100` | |
|scheduler.extraArgs.profiling | `false` | |
|controllerManager.extraArgs.profiling | `false` | |
|controllerManager.extraArgs.terminated-pod-gc-threshold| `1000` | |

The following is an example of kubeadm defaults override:

Expand Down Expand Up @@ -1139,10 +1141,7 @@ services:

**Note**: Those parameters remain in manifests files after Kubernetes upgrade. That is the proper way to preserve custom settings for system services.

**Warning**: These kubeadm parameters are configurable only during installation, currently.
Kubemarine currently do not provide special procedure to change these parameters after installation.
To reconfigure the parameters manually, refer to the official documentation at [https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure).
For more information about generic approach of the cluster maintenance, refer to [Maintenance Basics](Maintenance.md#basics).
**Note**: These kubeadm parameters can be reconfigured after installation using [Reconfigure Procedure](Maintenance.md#reconfigure-procedure).

During init, join, upgrade procedures kubeadm runs `preflight` procedure to do some preliminary checks. In case of any error kubeadm stops working. Sometimes it is necessary to ignore some preflight errors to deploy or upgrade successfully.

Expand Down Expand Up @@ -1502,6 +1501,8 @@ By default, the installer uses the following parameters:

`serializeImagePulls` parameter defines whether the images will be pulled in parallel (false) or one at a time.

**Note**: Some of the parameters can be reconfigured after installation using [Reconfigure Procedure](Maintenance.md#reconfigure-procedure).

**Warning**: If you want to change the values of variables `podPidsLimit` and `maxPods`, you have to update the value of the `pid_max` (this value should not less than result of next expression: `maxPods * podPidsLimit + 2048`), which can be done using task `prepare.system.sysctl`. To get more info about `pid_max` you can go to [sysctl](#sysctl) section.

The following is an example of kubeadm defaults override:
Expand Down Expand Up @@ -1536,6 +1537,8 @@ By default, the installer uses the following parameters:

`conntrack.min` inherits the `services.sysctl.net.netfilter.nf_conntrack_max` value from [sysctl](#sysctl).

**Note**: These parameters can be reconfigured after installation using [Reconfigure Procedure](Maintenance.md#reconfigure-procedure).

#### kubeadm_patches

*Installation task*: `deploy.kubernetes`
Expand Down Expand Up @@ -1591,7 +1594,11 @@ services:

By default Kubemarine sets `bind-address` parameter of `kube-apiserver` to `node.internal_address` via patches at every control-plane node.

**Note**: If a parameter of control-plane pods is defined in `kubeadm.<service>.extraArgs` or is set by default by kubeadm and then redefined in `kubeadm.paches`, the pod manifest file will contain the same flag twice and the running pod will take into account the last mentioned value (taken from `kubeadm.patches`). This behaviour persists at the moment: https://github.com/kubernetes/kubeadm/issues/1601.
**Note**: These parameters can be reconfigured after installation using [Reconfigure Procedure](Maintenance.md#reconfigure-procedure).

**Note**: If a parameter of control-plane pods is defined in `kubeadm.<service>.extraArgs` or is set by default by kubeadm and then redefined in `services.kubeadm_patches`,
the pod manifest file will contain the same flag twice and the running pod will take into account the last mentioned value (taken from `services.kubeadm_patches`).
This behaviour persists at the moment: https://github.com/kubernetes/kubeadm/issues/1601.

#### kernel_security

Expand Down
163 changes: 163 additions & 0 deletions documentation/Maintenance.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ This section describes the features and steps for performing maintenance procedu
- [Add Node Procedure](#add-node-procedure)
- [Operating System Migration](#operating-system-migration)
- [Remove Node Procedure](#remove-node-procedure)
- [Reconfigure Procedure](#reconfigure-procedure)
- [Manage PSP Procedure](#manage-psp-procedure)
- [Manage PSS Procedure](#manage-pss-procedure)
- [Reboot Procedure](#reboot-procedure)
Expand Down Expand Up @@ -970,6 +971,168 @@ To change the operating system on an already running cluster:

**Warning**: In case when you use custom associations, you need to specify them simultaneously for all types of operating systems. For more information, refer to the [associations](Installation.md#associations) section in the _Kubemarine Installation Procedure_.

## Reconfigure Procedure

This procedure is aimed to reconfigure the cluster.

It is supposed to reconfigure the cluster as a generalized concept described by the inventory file.
Though, currently the procedure supports to reconfigure only Kubeadm-managed settings.
If you are looking for how to reconfigure other settings, consider the following:
* Probably some other [maintenance procedure](#provided-procedures) can do the task.
* Some [installation tasks](Installation.md#tasks-list-redefinition) can reconfigure some system settings without full redeploy of the cluster.

**Basic prerequisites**:
* Make sure to follow the [Basics](#basics).
* Before starting the procedure, consider making a backup. For more information, see the section [Backup Procedure](#backup-procedure).

### Reconfigure Procedure Parameters

The procedure accepts required positional argument with the path to the procedure inventory file.
You can find description and examples of the accepted parameters in the next sections.

The JSON schema for procedure inventory is available by [URL](../kubemarine/resources/schemas/reconfigure.json?raw=1).
For more information, see [Validation by JSON Schemas](Installation.md#inventory-validation).

#### Reconfigure Kubeadm

The following Kubeadm-managed sections can be reconfigured:
* `services.kubeadm.apiServer`
* `services.kubeadm.apiServer.certSANs`
* `services.kubeadm.scheduler`
* `services.kubeadm.controllerManager`
* `services.kubeadm.etcd.local.extraArgs`
* `services.kubeadm_kubelet`
* `services.kubeadm_kube-proxy`
* `services.kubeadm_patches`

For more information, refer to the description of these sections:
* [kubeadm](Installation.md#kubeadm)
* [kubeadm_kubelet](Installation.md#kubeadm_kubelet)
* [kubeadm_kube-proxy](Installation.md#kubeadm_kube-proxy)
* [kubeadm_patches](Installation.md#kubeadm_patches)

Example of procedure inventory that reconfigures all the supported sections:

<details>
<summary>Click to expand</summary>

```yaml
services:
kubeadm:
apiServer:
certSANs:
- k8s-lb
extraArgs:
enable-admission-plugins: NodeRestriction,PodNodeSelector
profiling: "false"
audit-log-path: /var/log/kubernetes/audit/audit.log
audit-policy-file: /etc/kubernetes/audit-policy.yaml
audit-log-maxage: "30"
audit-log-maxbackup: "10"
audit-log-maxsize: "100"
scheduler:
extraArgs:
profiling: "false"
controllerManager:
extraArgs:
profiling: "false"
terminated-pod-gc-threshold: "1000"
etcd:
local:
extraArgs:
heartbeat-interval: "1000"
election-timeout: "10000"
kubeadm_kubelet:
readOnlyPort: 0
protectKernelDefaults: true
podPidsLimit: 2048
maxPods: 100
cgroupDriver: systemd
kubeadm_kube-proxy:
conntrack:
min: 1000000
kubeadm_patches:
apiServer:
- groups: [control-plane]
patch:
max-requests-inflight: 500
- nodes: [master-3]
patch:
max-requests-inflight: 600
etcd:
- nodes: [master-1]
patch:
snapshot-count: 110001
- nodes: [master-2]
patch:
snapshot-count: 120001
- nodes: [master-3]
patch:
snapshot-count: 130001
controllerManager:
- groups: [control-plane]
patch:
authorization-webhook-cache-authorized-ttl: 30s
scheduler:
- nodes: [master-2,master-3]
patch:
profiling: true
kubelet:
- nodes: [worker5]
patch:
maxPods: 100
- nodes: [worker6]
patch:
maxPods: 200
```
</details>

The above configuration is merged with the corresponding sections in the main `cluster.yaml`,
and the related Kubernetes components are reconfigured based on the resulting inventory.

In this way it is not possible to delete some property,
allowing the corresponding Kubernetes component to fall back to the default behaviour.
This can be worked around by manual changing of the `cluster.yaml`
and running the `reconfigure` procedure with **empty** necessary section.
For example, you can delete `services.kubeadm.etcd.local.extraArgs.election-timeout` from `cluster.yaml`
and then run the procedure with the following procedure inventory:

```yaml
services:
kubeadm:
etcd: {}
```

**Note**: It is not possible to delete default parameters offered by Kubemarine.

**Note**: The mentioned hint to delete custom properties is not enough for `services.kubeadm_kube-proxy` due to existing restrictions of Kubeadm CLI tool.
One should additionally edit the `kube-proxy` ConfigMap and set the value that is considered the default.

**Note**: Passing of empty `services.kubeadm.apiServer` section reconfigures the `kube-apiserver`,
but does not write new certificate.
To **additionally** write new certificate, pass the desirable extra SANs in `services.kubeadm.apiServer.certSANs`.

**Restrictions**:

* Very few options of `services.kubeadm_kubelet` section can be reconfigured currently.
To learn exact set of options, refer to the JSON schema.
* Some properties cannot be fully redefined.
For example, this relates to some settings in `services.kubeadm.apiServer`.
For details, refer to the description of the corresponding sections in the installation guide.

**Basic flow**:

If the procedure affects the particular Kubernetes component, the component is reconfigured on each relevant node one by one.
The flow proceeds to the next nodes only after the component is considered up and ready on the reconfigured node.

### Reconfigure Procedure Tasks Tree

The `reconfigure` procedure executes the following sequence of tasks:

* deploy
* kubernetes
* reconfigure

## Manage PSP Procedure

The manage PSP procedure allows you to change PSP configuration on an already installed cluster. Using this procedure, you can:
Expand Down

0 comments on commit 495a513

Please sign in to comment.