Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add documentation for setting Kubernetes labels and annotations #3687

Merged
merged 8 commits into from
May 13, 2024
Merged

Add documentation for setting Kubernetes labels and annotations #3687

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
36606d3
Add documentation for setting kubernetes labels and annotations
dominic-p May 6, 2024
b3ff3e4
Add links to kubernetes docs for annotations and labels
dominic-p May 8, 2024
b3f7856
Collocate label and annotation env variable docs
dominic-p May 8, 2024
6a78863
Reference env variable names for label and annotation config
dominic-p May 8, 2024
bc00af7
Fix env variable link anchor
dominic-p May 8, 2024
7a2cbd0
Standardize capitalization in Kubernetes docs
dominic-p May 13, 2024
89a5808
Merge branch 'main' into patch-1
qwerty287 May 13, 2024
f861909
fix link
qwerty287 May 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 49 additions & 19 deletions docs/docs/30-administration/22-backends/40-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,19 @@ toc_max_heading_level: 2

# Kubernetes backend

The kubernetes backend executes steps inside standalone pods. A temporary PVC is created for the lifetime of the pipeline to transfer files between steps.
The Kubernetes backend executes steps inside standalone Pods. A temporary PVC is created for the lifetime of the pipeline to transfer files between steps.

## Images from private registries

In order to pull private container images defined in your pipeline YAML you must provide [registry credentials in Kubernetes secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
As the secret is Agent-wide, it has to be placed in namespace defined by `WOODPECKER_BACKEND_K8S_NAMESPACE`.
Besides, you need to provide the secret name to Agent via `WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMES`.
In order to pull private container images defined in your pipeline YAML you must provide [registry credentials in Kubernetes Secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
As the Secret is Agent-wide, it has to be placed in namespace defined by `WOODPECKER_BACKEND_K8S_NAMESPACE`.
Besides, you need to provide the Secret name to Agent via `WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMES`.

## Job specific configuration

### Resources

The kubernetes backend also allows for specifying requests and limits on a per-step basic, most commonly for CPU and memory.
The Kubernetes backend also allows for specifying requests and limits on a per-step basic, most commonly for CPU and memory.
We recommend to add a `resources` definition to all steps to ensure efficient scheduling.

Here is an example definition with an arbitrary `resources` definition below the `backend_options` section:
Expand All @@ -42,13 +42,13 @@ You can use [Limit Ranges](https://kubernetes.io/docs/concepts/policy/limit-rang

### Runtime class

`runtimeClassName` specifies the name of the RuntimeClass which will be used to run this pod. If no `runtimeClassName` is specified, the default RuntimeHandler will be used.
See the [kubernetes documentation](https://kubernetes.io/docs/concepts/containers/runtime-class/) for more information on specifying runtime classes.
`runtimeClassName` specifies the name of the RuntimeClass which will be used to run this Pod. If no `runtimeClassName` is specified, the default RuntimeHandler will be used.
See the [Kubernetes documentation](https://kubernetes.io/docs/concepts/containers/runtime-class/) for more information on specifying runtime classes.

### Service account

`serviceAccountName` specifies the name of the ServiceAccount which the pod will mount. This service account must be created externally.
See the [kubernetes documentation](https://kubernetes.io/docs/concepts/security/service-accounts/) for more information on using service accounts.
`serviceAccountName` specifies the name of the ServiceAccount which the Pod will mount. This service account must be created externally.
See the [Kubernetes documentation](https://kubernetes.io/docs/concepts/security/service-accounts/) for more information on using service accounts.

### Node selector

Expand Down Expand Up @@ -83,8 +83,8 @@ You can use [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-

### Tolerations

When you use `nodeSelector` and the node pool is configured with Taints, you need to specify the Tolerations. Tolerations allow the scheduler to schedule pods with matching taints.
See the [kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) for more information on using tolerations.
When you use `nodeSelector` and the node pool is configured with Taints, you need to specify the Tolerations. Tolerations allow the scheduler to schedule Pods with matching taints.
See the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) for more information on using tolerations.

Example pipeline configuration:

Expand Down Expand Up @@ -117,7 +117,7 @@ steps:

### Volumes

To mount volumes a persistent volume (PV) and persistent volume claim (PVC) are needed on the cluster which can be referenced in steps via the `volumes` option.
To mount volumes a PersistentVolume (PV) and PersistentVolumeClaim (PVC) are needed on the cluster which can be referenced in steps via the `volumes` option.
Assuming a PVC named `woodpecker-cache` exists, it can be referenced as follows in a step:

```yaml
Expand All @@ -134,7 +134,7 @@ steps:

### Security context

Use the following configuration to set the [Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for the pod/container running a given pipeline step:
Use the following configuration to set the [Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for the Pod/container running a given pipeline step:

```yaml
steps:
Expand All @@ -151,9 +151,9 @@ steps:
[...]
```

Note that the `backend_options.kubernetes.securityContext` object allows you to set both pod and container level security context options in one object.
By default, the properties will be set at the pod level. Properties that are only supported on the container level will be set there instead. So, the
configuration shown above will result in something like the following pod spec:
Note that the `backend_options.kubernetes.securityContext` object allows you to set both Pod and container level security context options in one object.
By default, the properties will be set at the Pod level. Properties that are only supported on the container level will be set there instead. So, the
configuration shown above will result in something like the following Pod spec:

```yaml
kind: Pod
Expand Down Expand Up @@ -195,6 +195,24 @@ backend_options:
AppArmor syntax follows [KEP-24](https://github.com/kubernetes/enhancements/blob/fddcbb9cbf3df39ded03bad71228265ac6e5215f/keps/sig-node/24-apparmor/README.md).
:::

### Annotations and labels

You can specify arbitrary [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) and [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) to be set on the Pod definition for a given workflow step using the following configuration:

```yaml
backend_options:
kubernetes:
annotations:
workflow-group: alpha
io.kubernetes.cri-o.Devices: /dev/fuse
labels:
environment: ci
app.kubernetes.io/name: builder
```

In order to enable this configuration you need to set the appropriate environment variables to `true` on the woodpecker agent:
[WOODPECKER_BACKEND_K8S_POD_ANNOTATIONS_ALLOW_FROM_STEP](#woodpecker_backend_k8s_pod_annotations_allow_from_step) and/or [WOODPECKER_BACKEND_K8S_POD_LABELS_ALLOW_FROM_STEP](#woodpecker_backend_k8s_pod_labels_allow_from_step).

## Tips and tricks

### CRI-O
Expand All @@ -217,7 +235,7 @@ These env vars can be set in the `env:` sections of the agent.

> Default: `woodpecker`

The namespace to create worker pods in.
The namespace to create worker Pods in.

### `WOODPECKER_BACKEND_K8S_VOLUME_SIZE`

Expand All @@ -241,13 +259,25 @@ Determines if `RWX` should be used for the pipeline volume's [access mode](https

> Default: empty

Additional labels to apply to worker pods. Must be a YAML object, e.g. `{"example.com/test-label":"test-value"}`.
Additional labels to apply to worker Pods. Must be a YAML object, e.g. `{"example.com/test-label":"test-value"}`.

### `WOODPECKER_BACKEND_K8S_POD_LABELS_ALLOW_FROM_STEP`

> Default: `false`

Determines if additional Pod labels can be defined from a step's backend options.
qwerty287 marked this conversation as resolved.
Show resolved Hide resolved

### `WOODPECKER_BACKEND_K8S_POD_ANNOTATIONS`

> Default: empty

Additional annotations to apply to worker pods. Must be a YAML object, e.g. `{"example.com/test-annotation":"test-value"}`.
Additional annotations to apply to worker Pods. Must be a YAML object, e.g. `{"example.com/test-annotation":"test-value"}`.

### `WOODPECKER_BACKEND_K8S_POD_ANNOTATIONS_ALLOW_FROM_STEP`

> Default: `false`

Determines if Pod annotations can be defined from a step's backend options.

### `WOODPECKER_BACKEND_K8S_SECCTX_NONROOT`

Expand Down