diff --git a/Makefile b/Makefile index 2a50437c..4842e86e 100644 --- a/Makefile +++ b/Makefile @@ -6,10 +6,7 @@ getdeps: .PHONY: docs docs: - @echo "Fetching external docs…" - @find ./content/docs -maxdepth 1 -type l -delete @python3 ./tools/fcl-fetch-version-data.py ./content/docs/_index.md.in > ./content/docs/_index.md - python3 ./tools/docs-fetcher.py ./config.yaml run: hugo server --theme=flatcar --buildFuture --watch --disableFastRender --config ./config.yaml\,./tmp_modules.yaml diff --git a/content/docs/.gitignore b/content/docs/.gitignore index 9a5812bb..f20385a1 100644 --- a/content/docs/.gitignore +++ b/content/docs/.gitignore @@ -1,2 +1 @@ -* -!_index.md.in +_index.md diff --git a/content/docs/latest/_index.md b/content/docs/latest/_index.md new file mode 100644 index 00000000..185caf92 --- /dev/null +++ b/content/docs/latest/_index.md @@ -0,0 +1,249 @@ +--- +content_type: flatcar +title: Flatcar Container Linux +main_menu: true +weight: 40 +--- + +Flatcar Container Linux is a container optimized OS that ships a minimal OS +image, which includes only the tools needed to run containers. The OS is +shipped through an immutable filesystem and includes automatic atomic +updates. + + +### Getting Started + +If you're new to Flatcar and if you're looking for a brief introduction on getting Flatcar up and running, please have a look at our [quickstart guide][quick-start]. + +### Installing Flatcar + +Flatcar Container Linux runs on most cloud providers, virtualization +platforms and bare metal servers. + +#### Cloud Providers + * [Amazon EC2][ec2] + * [Microsoft Azure][azure] + * [Google Compute Engine][gce] + * [Equinix Metal][equinix-metal] + * [VMware][vmware] + * [DigitalOcean][digital-ocean] + * [Hetzner][hetzner] + * [OpenStack][openstack] + +#### Virtualization options +It's easy to run a local Flatcar VM on your laptop for testing and debugging +purposes. You can use any of the following options. + + * [QEMU][qemu] + * [libVirt][libvirt] + * [VirtualBox][virtualbox] (not officially supported) + * [Vagrant][vagrant] (not officially supported) + +#### Bare Metal +You can install Flatcar on bare metal machines in different ways: using ISO +images, booting from PXE or iPXE, and even by running an installation +script on an existing Linux system. + + * [Installing from ISO images][boot-iso] + * [Booting with PXE][pxe] + * [Booting with iPXE][ipxe] + * [Installing with flatcar-install][install-to-disk] + +If you want to provide metadata to your baremetal machines, we recommend +using [Matchbox][matchbox]. + +#### Upgrading from CoreOS Container Linux + +Flatcar Container Linux is a drop-in replacement of CoreOS Container Linux. +If you are a CoreOS Container Linux user looking for a replacement, +checkout our guides to [migrate from CoreOS Container +Linux][migrate-from-container-linux], or you can [update from CoreOS +Container Linux][update-from-container-linux] directly. + +### Provisioning Tools + +[Ignition][ignition-what] is the recommended way to provision Flatcar +Container Linux at first boot. Ignition uses a JSON configuration file, +and it is recommended to generate it from the [Container Linux +Config][container-linux-config] YAML format, which has additional features. +The [Container Linux Config Transpiler][config-transpiler] converts a +Container Linux Config to an Ignition config. + + * [Understanding the Boot Process][ignition-boot] + * [Configuring the Network with Ignition][ignition-network] + * [Using metadata during provisioning][ignition-metadata] + * [Getting started with Butane][config-intro] + * [Examples of using Butane][config-examples] + * [Using Terraform to provision Flatcar Container Linux][terraform] + * [Extending the base OS with systemd-sysext images][sysext] + +### Setting Flatcar Up and Common Operations + +Follow these guides to connect your machines together as a cluster, +configure machine parameters, create users, inject multiple SSH keys, and +more. + +#### Customizing Flatcar + * [Using networkd to customize networking][networkd-customize] + * [Using systemd drop-in units][systemd-drop-in] + * [Using environment variables in systemd units][environment-variables-systemd] + * [Using systemd and udev rules][udev-rules] + * [Using NVIDIA GPUs on Flatcar][using-nvidia] + * [Scheduling tasks with systemd timers][tasks-with-systemd] + * [Configuring DNS][dns] + * [Configuring date & timezone][date-timezone] + * [Adding users][users] + * [Kernel modules / sysctl parameters][parameters] + * [Adding swap][swap] + * [Power management][power-management] + * [ACPI][acpi] + +#### Managing Releases and Updates + * [Switching release channels][release-channels] + * [Configuring the update strategy][update-strategies] + * [Flatcar update configuration specification][update-conf] + * [Verifying Flatcar Images with GPG][verify-container-linux] + +#### Creating Clusters + * [Cluster architectures][cluster-architectures] + * [Clustering machines][clustering-machines] + * [Using Amazon EC2 Container Service][ec2-container-service] + +#### Managing Storage + * [Using RAID for the root filesystem][filesystem-placement] + * [Adding disk space][disk-space] + * [Mounting storage][mounting-storage] + * [iSCSI configuration][iscsi] + +#### Additional security options + * [Customizing the SSH daemon][ssh-daemon] + * [Configuring SSSD on Flatcar Container Linux][sssd-container-linux] + * [Hardening a Flatcar Container Linux machine][hardening-container-linux] + * [Trusted Computing Hardware Requirements][hardware-requirements] + * [Adding Cert Authorities][cert-authorities] + * [Using SELinux][selinux] + * [Disabling SMT][disabling-smt] + * [Enabling FIPS][enabling-fips] + * [Using the audit subsystem][audit-system] + +#### Debugging Flatcar + * [Install debugging tools][debugging-tools] + * [Working with btrfs][btrfs] + * [Reading the system log][system-log] + * [Collecting crash logs][crash-log] + * [Manual Flatcar Container Linux rollbacks][container-linux-rollbacks] + +### Container Runtimes +Flatcar Container Linux supports all of the popular methods for running +containers, and you can choose to interact with the containers at a +low-level, or use a higher level orchestration framework. Listed below are +some guides to help you choose and make use of the different runtimes. + + * [Getting started with Docker][docker] + * [Customizing Docker][customizing-docker] + * [Using systemd to manage Docker containers][manage-docker-containers] + * [Use a custom Docker or containerd version][use-a-custom-docker-or-containerd-version] + * [Authenticating to Container registries][registry-authentication] + * [Getting started with Kubernetes][kubernetes] + +### Developer guides and Reference +APIs and troubleshooting guides for working with Flatcar Container Linux. + +* [Developer guides][developer-guides]: Comprehensive guides on developing for Flatcar, working with the SDK, and on building and extending OS images. +* [Integrations][integrations] +* [Migrating from cloud-config to Container Linux Config][migrating-from-cloud-config] +* [Flatcar Supply Chain Security (SLSA and SPDX SBOM)][supply-chain-security] detailing security mechanisms employed at build / release time as well as at run-time to ensure validity of inputs processed and outputs shipped. + +### Tutorial +Flatcar tutorial to deep dive into some Flatcar fundamental concepts. +* [Introduction][tutorial-introduction] +* [Hands-on 1: Discovering][tutorial-hands-on-1] +* [Hands-on 2: Provisioning][tutorial-hands-on-2] +* [Hands-on 3: Deploying][tutorial-hands-on-3] +* [Hands-on 4: Updating][tutorial-hands-on-4] + +[quick-start]: installing +[supply-chain-security]: reference/supply-chain +[ignition-what]: provisioning/ignition/ +[ignition-boot]: provisioning/ignition/boot-process +[ignition-network]: provisioning/ignition/network-configuration +[ignition-metadata]: provisioning/ignition/metadata +[container-linux-config]: provisioning/cl-config/ +[config-transpiler]: provisioning/config-transpiler/ +[config-intro]: provisioning/config-transpiler/getting-started +[config-dynamic-data]: provisioning/config-transpiler/dynamic-data +[config-examples]: provisioning/config-transpiler/examples +[matchbox]: https://matchbox.psdn.io/ +[ipxe]: installing/bare-metal/booting-with-ipxe +[pxe]: installing/bare-metal/booting-with-pxe +[install-to-disk]: installing/bare-metal/installing-to-disk +[boot-iso]: installing/bare-metal/booting-with-iso +[filesystem-placement]: setup/storage/raid +[migrate-from-container-linux]: migrating-from-coreos/ +[update-from-container-linux]: migrating-from-coreos/update-from-container-linux +[ec2]: installing/cloud/aws-ec2 +[digital-ocean]: installing/cloud/digitalocean +[gce]: installing/cloud/gcp +[azure]: installing/cloud/azure +[qemu]: installing/vms/qemu +[equinix-metal]: installing/cloud/equinix-metal +[libvirt]: installing/vms/libvirt +[virtualbox]: installing/vms/virtualbox +[vagrant]: installing/vms/vagrant +[vmware]: installing/cloud/vmware +[cluster-architectures]: setup/clusters/architectures +[update-strategies]: setup/releases/update-strategies +[clustering-machines]: setup/clusters/discovery +[verify-container-linux]: setup/releases/verify-images +[networkd-customize]: setup/customization/network-config-with-networkd +[systemd-drop-in]: setup/systemd/drop-in-units +[environment-variables-systemd]: setup/systemd/environment-variables +[dns]: setup/customization/configuring-dns +[date-timezone]: setup/customization/configuring-date-and-timezone +[users]: setup/customization/adding-users +[parameters]: setup/customization/other-settings +[disk-space]: setup/storage/adding-disk-space +[mounting-storage]: setup/storage/mounting-storage +[power-management]: setup/customization/power-management +[registry-authentication]: container-runtimes/registry-authentication +[iscsi]: setup/storage/iscsi +[swap]: setup/storage/adding-swap +[ec2-container-service]: setup/clusters/booting-on-ecs/ +[manage-docker-containers]: setup/systemd/getting-started +[udev-rules]: setup/systemd/udev-rules +[update-conf]: setup/releases/update-conf +[release-channels]: setup/releases/switching-channels +[tasks-with-systemd]: setup/systemd/timers +[ssh-daemon]: setup/security/customizing-sshd +[sssd-container-linux]: setup/security/sssd +[hardening-container-linux]: setup/security/hardening-guide +[hardware-requirements]: setup/security/trusted-computing-hardware-requirements +[cert-authorities]: setup/security/adding-certificate-authorities +[selinux]: setup/security/selinux +[disabling-smt]: setup/security/disabling-smt +[enabling-fips]: setup/security/fips +[audit-system]: setup/security/audit +[debugging-tools]: setup/debug/install-debugging-tools +[btrfs]: setup/debug/btrfs-troubleshooting +[system-log]: setup/debug/reading-the-system-log +[crash-log]: setup/debug/collecting-crash-logs +[container-linux-rollbacks]: setup/debug/manual-rollbacks +[docker]: container-runtimes/getting-started-with-docker +[customizing-docker]: container-runtimes/customizing-docker +[use-a-custom-docker-or-containerd-version]: container-runtimes/use-a-custom-docker-or-containerd-version +[developer-guides]: reference/developer-guides/ +[integrations]: reference/integrations/ +[migrating-from-cloud-config]: provisioning/cl-config/from-cloud-config +[containerd-for-kubernetes]: container-runtimes/switching-from-docker-to-containerd-for-kubernetes +[terraform]: provisioning/terraform/ +[hetzner]: installing/cloud/hetzner +[sysext]: provisioning/sysext/ +[acpi]: setup/customization/ACPI +[openstack]: installing/cloud/openstack +[kubernetes]: container-runtimes/getting-started-with-kubernetes +[using-nvidia]: setup/customization/using-nvidia +[tutorial-introduction]: tutorial/ +[tutorial-hands-on-1]: tutorial/hands-on-1 +[tutorial-hands-on-2]: tutorial/hands-on-2 +[tutorial-hands-on-3]: tutorial/hands-on-3 +[tutorial-hands-on-4]: tutorial/hands-on-4 diff --git a/content/docs/latest/container-runtimes/_index.md b/content/docs/latest/container-runtimes/_index.md new file mode 100644 index 00000000..09837b7e --- /dev/null +++ b/content/docs/latest/container-runtimes/_index.md @@ -0,0 +1,10 @@ +--- +title: Container Runtimes +description: > + Flatcar Container Linux supports all of the popular methods for running + containers, and you can choose to interact with the containers at a + low-level, or use a higher level orchestration framework. These guides + can help you choose and use the different container runtimes supported. +weight: 60 +--- + diff --git a/content/docs/latest/container-runtimes/customizing-docker.md b/content/docs/latest/container-runtimes/customizing-docker.md new file mode 100644 index 00000000..ea418528 --- /dev/null +++ b/content/docs/latest/container-runtimes/customizing-docker.md @@ -0,0 +1,383 @@ +--- +title: Customizing Docker +description: > + How to select which runtime to use, make docker available on a + TCP socket, enable TLS, and other customizations. +weight: 30 +aliases: + - ../os/customizing-docker +--- + +The Docker systemd unit can be customized by overriding the unit that ships with the default Flatcar Container Linux settings or through a drop-in unit. Common use-cases for doing this are covered below. + +For switching to using containerd with Kubernetes, there is an [extra guide](../switching-from-docker-to-containerd-for-kubernetes/). + +## Use a custom containerd configuration + +The default configuration under `/run/torcx/unpack/docker/usr/share/containerd/config.toml` can't be changed but you can copy it to `/etc/containerd/config.toml` and modify it. +**NOTE** that newer Flatcar major releases (above major release version 3760) ship the default configuration under `/usr/share/containerd/config.toml`. + +Create a `/etc/systemd/system/containerd.service.d/10-use-custom-config.conf` unit drop-in file to select the new configuration: + +```ini +[Service] +ExecStart= +ExecStart=/usr/bin/containerd +``` + +On a running system, execute `systemctl daemon-reload ; systemctl restart containerd` for it to take effect. + +## Enable the remote API on a new socket + +Create a file called `/etc/systemd/system/docker-tcp.socket` to make Docker available on a TCP socket on port 2375. + +```ini +[Unit] +Description=Docker Socket for the API + +[Socket] +ListenStream=2375 +BindIPv6Only=both +Service=docker.service + +[Install] +WantedBy=sockets.target +``` + +Then enable this new socket: + +```shell +systemctl enable docker-tcp.socket +systemctl stop docker +systemctl start docker-tcp.socket +systemctl start docker +``` + +Test that it's working: + +```shell +docker -H tcp://127.0.0.1:2375 ps +``` + +### Butane Config + +To enable the remote API on every Flatcar Container Linux machine in a cluster, use a [Butane Config][butane-configs]. We need to provide the new socket file and Docker's socket activation support will automatically start using the socket: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: docker-tcp.socket + enabled: true + contents: | + [Unit] + Description=Docker Socket for the API + + [Socket] + ListenStream=2375 + BindIPv6Only=both + Service=docker.service + + [Install] + WantedBy=sockets.target +``` + +To keep access to the port local, replace the `ListenStream` configuration above with: + +```yaml + [Socket] + ListenStream=127.0.0.1:2375 +``` + +## Enable the remote API with TLS authentication + +Docker TLS configuration consists of three parts: keys creation, configuring new [systemd socket][systemd-socket] unit and systemd [drop-in][drop-in] configuration. + +### TLS keys creation + +Please follow the [instruction][self-signed-certs] to know how to create self-signed certificates and private keys. Then copy the following files into `/etc/docker` Flatcar Container Linux's directory and fix their permissions: + +```shell +scp ~/cfssl/{server.pem,server-key.pem,ca.pem} flatcar.example.com: +ssh core@flatcar.example.com +sudo mv {server.pem,server-key.pem,ca.pem} /etc/docker/ +sudo chown root:root /etc/docker/{server-key.pem,server.pem,ca.pem} +sudo chmod 0600 /etc/docker/server-key.pem +``` + +On your local host copy certificates into `~/.docker`: + +```shell +mkdir ~/.docker +chmod 700 ~/.docker +cd ~/.docker +cp -p ~/cfssl/ca.pem ca.pem +cp -p ~/cfssl/client.pem cert.pem +cp -p ~/cfssl/client-key.pem key.pem +``` + +### Enable the secure remote API on a new socket + +Create a file called `/etc/systemd/system/docker-tls-tcp.socket` to make Docker available on a secured TCP socket on port 2376. + +```ini +[Unit] +Description=Docker Secured Socket for the API + +[Socket] +ListenStream=2376 +BindIPv6Only=both +Service=docker.service + +[Install] +WantedBy=sockets.target +``` + +Then enable this new socket: + +```shell +systemctl enable docker-tls-tcp.socket +systemctl stop docker +systemctl start docker-tls-tcp.socket +``` + +### Drop-in configuration + +Create `/etc/systemd/system/docker.service.d/10-tls-verify.conf` [drop-in][drop-in] for systemd Docker service: + +```ini +[Service] +Environment="DOCKER_OPTS=--tlsverify --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server.pem --tlskey=/etc/docker/server-key.pem" +``` + +Reload systemd config files and restart docker service: + +```shell +sudo systemctl daemon-reload +sudo systemctl restart docker.service +``` + +Now you can access your Docker's API through TLS secured connection: + +```shell +docker --tlsverify -H tcp://server:2376 images +# or +docker --tlsverify -H tcp://server.example.com:2376 images +``` + +If you've experienceed problems connection to remote Docker API using TLS connection, you can debug it with `curl`: + +```shell +curl -v --cacert ~/.docker/ca.pem --cert ~/.docker/cert.pem --key ~/.docker/key.pem https://server:2376 +``` + +Or on your Flatcar Container Linux host: + +```shell +journalctl -f -u docker.service +``` + +In addition you can export environment variables and use docker client without additional options: + +```shell +export DOCKER_HOST=tcp://server.example.com:2376 DOCKER_TLS_VERIFY=1 +docker images +``` + +### Butane Config (TLS) + +A Butane Config for Docker TLS authentication will look like: + +```yaml +variant: flatcar +version: 1.0.0 +storage: + files: + - path: /etc/docker/ca.pem + mode: 0644 + contents: + inline: | + -----BEGIN CERTIFICATE----- + MIIFNDCCAx6gAwIBAgIBATALBgkqhkiG9w0BAQswLTEMMAoGA1UEBhMDVVNBMRAw + DgYDVQQKEwdldGNkLWNhMQswCQYDVQQLEwJDQTAeFw0xNTA5MDIxMDExMDhaFw0y + NTA5MDIxMDExMThaMC0xDDAKBgNVBAYTA1VTQTEQMA4GA1UEChMHZXRjZC1jYTEL + ... ... ... + - path: /etc/docker/server.pem + mode: 0644 + contents: + inline: | + -----BEGIN CERTIFICATE----- + MIIFajCCA1SgAwIBAgIBBTALBgkqhkiG9w0BAQswLTEMMAoGA1UEBhMDVVNBMRAw + DgYDVQQKEwdldGNkLWNhMQswCQYDVQQLEwJDQTAeFw0xNTA5MDIxMDM3MDFaFw0y + NTA5MDIxMDM3MDNaMEQxDDAKBgNVBAYTA1VTQTEQMA4GA1UEChMHZXRjZC1jYTEQ + ... ... ... + - path: /etc/docker/server-key.pem + mode: 0644 + contents: + inline: | + -----BEGIN RSA PRIVATE KEY----- + MIIJKAIBAAKCAgEA23Q4yELhNEywScrHl6+MUtbonCu59LIjpxDMAGxAHvWhWpEY + P5vfas8KgxxNyR+U8VpIjEXvwnhwCx/CSCJc3/VtU9v011Ir0WtTrNDocb90fIr3 + YeRWq744UJpBeDHPV9opf8xFE7F74zWeTVMwtiMPKcQDzZ7XoNyJMxg1wmiMbdCj + ... ... ... +systemd: + units: + - name: docker-tls-tcp.socket + enabled: true + contents: | + [Unit] + Description=Docker Secured Socket for the API + + [Socket] + ListenStream=2376 + BindIPv6Only=both + Service=docker.service + + [Install] + WantedBy=sockets.target + - name: docker.service + dropins: + - name: flags.conf + contents: | + [Service] + Environment="DOCKER_OPTS=--tlsverify --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server.pem --tlskey=/etc/docker/server-key.pem" +``` + +## Use attached storage for Docker images + +Docker containers can be very large and debugging a build process makes it easy to accumulate hundreds of containers. It's advantageous to use attached storage to expand your capacity for container images. Check out the guide to [mounting storage to your Flatcar Container Linux machine][mounting-storage] for an example of how to bind mount storage into `/var/lib/docker`. + +## Enabling the Docker debug flag + +Set the `--debug` (`-D`) flag in the `DOCKER_OPTS` environment variable by using a drop-in file. For example, the following could be written to `/etc/systemd/system/docker.service.d/10-debug.conf`: + +```ini +[Service] +Environment=DOCKER_OPTS=--debug +``` + +Now tell systemd about the new configuration and restart Docker: + +```shell +systemctl daemon-reload +systemctl restart docker +``` + +To test our debugging stream, run a Docker command and then read the systemd journal, which should contain the output: + +```shell +docker ps +journalctl -u docker +``` + +### Butane Config (flags) + +If you need to modify a flag across many machines, you can add the flag with a Butane Config: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: docker.service + dropins: + - name: flags.conf + contents: | + [Service] + Environment="DOCKER_OPTS=--debug" +``` + +## Use an HTTP proxy + +If you're operating in a locked down networking environment, you can specify an HTTP proxy for Docker to use via an environment variable. First, create a directory for drop-in configuration for Docker: + +```shell +mkdir /etc/systemd/system/docker.service.d +``` + +Now, create a file called `/etc/systemd/system/docker.service.d/http-proxy.conf` that adds the environment variable: + +```ini +[Service] +Environment="HTTP_PROXY=http://proxy.example.com:8080" +``` + +To apply the change, reload the unit and restart Docker: + +```shell +systemctl daemon-reload +systemctl restart docker +``` + +Proxy environment variables can also be set [system-wide][systemd-env-vars]. + +### Butane Config (proxy) + +The easiest way to use this proxy on all of your machines is via a Butane Config: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: docker.service + enabled: true + dropins: + - name: 20-http-proxy.conf + contents: | + [Service] + Environment="HTTP_PROXY=http://proxy.example.com:8080" +``` + +## Increase ulimits + +If you need to increase certain ulimits that are too low for your application by default, like memlock, you will need to modify the Docker service to increase the limit. First, create a directory for drop-in configuration for Docker: + +```shell +mkdir /etc/systemd/system/docker.service.d +``` + +Now, create a file called `/etc/systemd/system/docker.service.d/increase-ulimit.conf` that adds increased limit: + +```ini +[Service] +LimitMEMLOCK=infinity +``` + +To apply the change, reload the unit and restart Docker: + +```shell +systemctl daemon-reload +systemctl restart docker +``` + +### Butane Config (ulimits) + +The easiest way to use these new ulimits on all of your machines is via a Butane Config: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: docker.service + enabled: true + dropins: + - name: 30-increase-ulimit.conf + contents: | + [Service] + LimitMEMLOCK=infinity +``` + +## Using a dockercfg file for authentication + +A json file `.dockercfg` can be created in your home directory that holds authentication information for a public or private Docker registry. + +[docker-socket-systemd]: https://github.com/docker/docker/pull/17211 +[drop-in]: ../setup/systemd/drop-in-units +[mounting-storage]: ../setup/storage/mounting-storage +[self-signed-certs]: ../setup/security/generate-self-signed-certificates +[systemd-socket]: https://www.freedesktop.org/software/systemd/man/systemd.socket.html +[systemd-env-vars]: ../setup/systemd/environment-variables/#system-wide-environment-variables +[butane-configs]: ../../provisioning/config-transpiler diff --git a/content/docs/latest/container-runtimes/getting-started-with-docker.md b/content/docs/latest/container-runtimes/getting-started-with-docker.md new file mode 100644 index 00000000..275dd1d7 --- /dev/null +++ b/content/docs/latest/container-runtimes/getting-started-with-docker.md @@ -0,0 +1,175 @@ +--- +title: Getting started with Docker +description: Basic Docker operations on Flatcar +weight: 10 +aliases: + - ../os/getting-started-with-docker +--- + +Docker is an open-source project that makes creating and managing Linux containers really easy. Containers are like extremely lightweight VMs – they allow code to run in isolation from other containers but safely share the machine’s resources, all without the overhead of a hypervisor. + +Docker containers can boot extremely fast (in milliseconds!) which gives you unprecedented flexibility in managing load across your cluster. For example, instead of running chef on each of your VMs, it’s faster and more reliable to have your build system create a container and launch it on the appropriate number of Flatcar Container Linux hosts. This guide will show you how to launch a container, install some software on it, commit that container, and optionally launch it on another Flatcar Container Linux machine. Before starting, make sure you've got at least one Flatcar Container Linux machine up and running — try it on [Amazon EC2][aws-ec2] or locally with [Vagrant][vagrant]. + +## Docker CLI basics + +Docker has a [straightforward CLI][docker-cli] that allows you to do almost everything you could want to a container. All of these commands use the image id (ex. be29975e0098), the image name (ex. myusername/webapp) and the container id (ex. 72d468f455ea) interchangeably depending on the operation you are trying to do. This is confusing at first, so pay special attention to what you're using. + +## Launching a container + +Launching a container is simple as `docker run` + the image name you would like to run + the command to run within the container. If the image doesn't exist on your local machine, Docker will attempt to fetch it from the public image registry. Later we'll explore how to use Docker with a private registry. It's important to note that containers are designed to stop once the command executed within them has exited. For example, if you ran `/bin/echo hello world` as your command, the container will start, print hello world and then stop: + +```shell +docker run ubuntu /bin/echo hello world +``` + +Let's launch an Ubuntu container and install Apache inside of it using the bash prompt: + +```shell +docker run -t -i ubuntu /bin/bash +``` + +The `-t` and `-i` flags allocate a pseudo-tty and keep stdin open even if not attached. This will allow you to use the container like a traditional VM as long as the bash prompt is running. Install Apache with `apt-get update && apt-get install apache2`. You're probably wondering what address you can connect to in order to test that Apache was correctly installed...we'll get to that after we commit the container. + +## Committing a container + +After that completes, we need to `commit` these changes to our container with the container ID and the image name. + +To find the container ID, open another shell (so the container is still running) and read the ID using `docker ps`. + +The image name is in the format of `username/name`. We're going to use `flatcar` as our username in this example but you should [sign up for a Docker.IO user account][docker-signup] and use that instead. + +It's important to note that you can commit using any username and image name locally, but to push an image to the public registry, the username must be a valid [Docker.IO user account][docker-signup]. + +Commit the container with the container ID, your username, and the name `apache`: + +```shell +docker commit 72d468f455ea myname/myapache +``` + +The overlay filesystem works similar to git: our image now builds off of the `ubuntu` base and adds another layer with Apache on top. These layers get cached separately so that you won't have to pull down the ubuntu base more than once. + +## Keeping the Apache container running + +Now we have our Ubuntu container with Apache running in one shell and an image of that container sitting on disk. Let's launch a new container based on that image but set it up to keep running indefinitely. The basic syntax looks like this, but we need to configure a few additional options that we'll fill in as we go: + +```shell +docker run [options] [image] [process] +``` + +The first step is to tell Docker that we want to run our `myname/myapache` image: + +```shell +docker run [options] myname/myapache [process] +``` + +### Run container detached + +When running Docker containers manually, the most important option is to run the container in detached mode with the `-d` flag. This will output the container ID to show that the command was successful, but nothing else. At any time you can run `docker ps` in the other shell to view a list of the running containers. Our command now looks like: + +```shell +docker run -d myname/myapache [process] +``` + +After you are comfortable with the mechanics of running containers by hand, it's recommended to use [systemd units][systemd-getting-started] to run your containers on a cluster of Flatcar Container Linux machines. + +Do not run containers with detached mode inside of systemd unit files. Detached mode prevents your init system, in our case systemd, from monitoring the process that owns the container because detached mode forks it into the background. To prevent this issue, just omit the `-d` flag if you aren't running something manually. + +### Run Apache in foreground + +We need to run the apache process in the foreground, since our container will stop when the process specified in the `docker run` command stops. We can do this with a flag `-D` when starting the apache2 process: + +```shell +/usr/sbin/apache2ctl -D FOREGROUND +``` + +Let's add that to our command: + +```shell +docker run -d myname/myapache /usr/sbin/apache2ctl -D FOREGROUND +``` + +### Permanently running a container + +While the sections above explained how to run a container when configuring it, for a production setup, you should not manually start and babysit containers. + +Instead, create a systemd unit file to make systemd keep that container running. See [Getting Started with systemd][systemd-getting-started] for details. + +Alternatively, Docker also has a feature to start existing containers on boot, when the container has the `restart` attribute set to `always`. +This requires the Docker service to get started on boot instead of using the default socket activation that starts on-demand. + +Here is a Butane Config to enable the Docker service while disabling socket activation: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + # Ensure docker starts automatically instead of being only socket-activated + - name: docker.service + enabled: true +storage: + links: + - path: /etc/systemd/system/multi-user.target.wants/docker.service + target: /usr/lib/systemd/system/docker.service + hard: false + overwrite: true +``` + +**NOTE** for Flatcar versions prior to (older than) the 3761 major release the soft link is unnecessary. The following configuration suffices: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + # Ensure docker starts automatically instead of being only socket-activated + - name: docker.service + enabled: true +``` + +### Network access to 80 + +The default apache install will be running on port 80. To give our container access to traffic over port 80, we use the `-p` flag and specify the port on the host that maps to the port inside the container. In our case we want 80 for each, so we include `-p 80:80` in our command: + +```shell +docker run -d -p 80:80 myname/myapache /usr/sbin/apache2ctl -D FOREGROUND +``` + +You can now run this command on your Flatcar Container Linux host to create the container. You should see the default apache webpage when you load either `localhost:80` or the IP of your remote server. Be sure that any firewall or EC2 Security Group allows traffic to port 80. + +## Using the Docker registry + +Earlier we downloaded the ubuntu image remotely from the Docker public registry because it didn't exist on our local machine. We can also push local images to the public registry (or a private registry) very easily with the `push` command: + +```shell +docker push myname/myapache +``` + +To push to a private repository the syntax is very similar. First, we must prefix our image with the host running our private registry instead of our username. List images by running `docker images` and insert the correct ID into the `tag` command: + +```shell +docker tag f455ea72d468 registry.example.com:5000/myname/myapache +``` + +After tagging, the image needs to be pushed to the registry: + +```shell +docker push registry.example.com:5000/myname/myapache +``` + +Once the image is done uploading, you should be able to start the exact same container on a different Flatcar Container Linux host by running: + +```shell +docker run -d -p 80:80 registry.example.com:5000/myname/myapache /usr/sbin/apache2ctl -D FOREGROUND +``` + +## More information + + * [Docker Website](http://www.docker.com/) + * [docker's Getting Started Guide](https://docs.docker.com/mac/started/) + +[aws-ec2]: ../installing/cloud/aws-ec2 +[vagrant]: ../installing/vms/vagrant +[docker-cli]: https://docs.docker.com/engine/reference/commandline/cli/ +[docker-signup]: https://hub.docker.com/account/signup/ +[systemd-getting-started]: ../setup/systemd/getting-started diff --git a/content/docs/latest/container-runtimes/getting-started-with-kubernetes.md b/content/docs/latest/container-runtimes/getting-started-with-kubernetes.md new file mode 100644 index 00000000..d0399562 --- /dev/null +++ b/content/docs/latest/container-runtimes/getting-started-with-kubernetes.md @@ -0,0 +1,381 @@ +--- +title: Getting started with Kubernetes +description: Operate Kubernetes from Flatcar +aliases: + - ../os/switching-from-docker-to-containerd-for-kubernetes + - ./switching-from-docker-to-containerd-for-kubernetes +weight: 11 +--- + +One of the Flatcar purposes is to run container workloads, this term is quite generic: it goes from running a single Docker container to operate a Kubernetes cluster. + +This documentation will cover preliminary aspects of operating Kubernetes cluster based on Flatcar. + +# Supported Kubernetes version + +A Kubernetes basic scenario (deploy a simple Nginx) is being tested on Flatcar accross the channels and various CNIs, it mainly ensures that Kubernetes can be correctly installed and can operate in a simple way. + +One way to contribute to Flatcar would be to extend the covered CNIs (example: [kubenet][kubenet]) or to provide more complex scenarios (example: [cilium extension][cilium]). + +This is a compatibility matrix between Flatcar and Kubernetes deployed using vanilla components and Flatcar provided software: +| :arrow_down: Flatcar channel \ Kubernetes Version :arrow_right: | 1.23 | 1.24 | 1.25 | 1.26 | 1.27 | 1.28 | +|--------------------------------------|--------------------|--------------------|--------------------|--------------------|--------------------|---------------------------------| +| Alpha | :large_orange_diamond: | :large_orange_diamond: | :white_check_mark: | :white_check_mark: |:white_check_mark: | :white_check_mark: | +| Beta | :large_orange_diamond: | :large_orange_diamond: | :white_check_mark: | :white_check_mark: |:white_check_mark: | :white_check_mark: | +| Stable | :large_orange_diamond: | :large_orange_diamond: | :white_check_mark: | :white_check_mark: |:white_check_mark: | :white_check_mark: | +| LTS | :large_orange_diamond: | :large_orange_diamond: | :white_check_mark: | :x: |:x: | :x: | + +:large_orange_diamond:: The version is not tested anymore before a release but was known for working. + +Tested CNIs: +- Cilium +- Flannel +- Calico + +_Known issues_: +* Flannel > 0.17.0 does not work with enforced SELinux ([flatcar#779][flatcar-779]) +* Cilium needs to be patched regarding SELinux labels to work (even in permissive mode) ([flatcar#891][flatcar-891]) + +# Deploy a Kubernetes cluster with Flatcar + +## Using Kubeadm + +`kubeadm` remains one standard way to quickly deploy and operate a Kubernetes cluster. It's possible to install the tools (`kubeadm`, `kubelet`, etc.) using Ignition or directly with the Kubernetes sysext image distributed from the [flatcar/sysext-bakery][sysext-bakery] release page. + +### Setup the control plane + +Here are two examples to setup a control plane with [Butane][butane]. The first example is using the systemd-sysext approach to bring in the binaries and update them through systemd-sysupdate. The second approach fetches the binaries but has no way of updating them in-place. + +
+ +
+
+
+ This is an example using systemd-sysext and systemd-sysupdate. NOTE: We are using Kured to coordinate nodes reboot when there is a new Kubernetes sysext image available (or if Flatcar has been updated), hence the /run/reboot-required file. + 
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  links:
+    - target: /opt/extensions/kubernetes/kubernetes-v1.27.4-x86-64.raw
+      path: /etc/extensions/kubernetes.raw
+      hard: false
+  files:
+    - path: /etc/sysupdate.kubernetes.d/kubernetes.conf
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/kubernetes.conf
+    - path: /etc/sysupdate.d/noop.conf
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/noop.conf
+    - path: /opt/extensions/kubernetes/kubernetes-v1.27.4-x86-64.raw
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/kubernetes-v1.27.4-x86-64.raw
+systemd:
+  units:
+    - name: systemd-sysupdate.timer
+      enabled: true
+    - name: systemd-sysupdate.service
+      dropins:
+        - name: kubernetes.conf
+          contents: |
+            [Service]
+            ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
+            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
+            ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
+            ExecStartPost=/usr/bin/sh -c "[[ $(cat /tmp/kubernetes) != $(cat /tmp/kubernetes-new) ]] && touch /run/reboot-required"
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        ConditionPathExists=!/etc/kubernetes/kubelet.conf
+        [Service]
+        ExecStartPre=/usr/bin/kubeadm init
+        ExecStartPre=/usr/bin/mkdir /home/core/.kube
+        ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/core/.kube/config
+        ExecStart=/usr/bin/chown -R core:core /home/core/.kube
+        [Install]
+        WantedBy=multi-user.target
+
+
+
+
+
+ :warning: To ease the reading, we voluntarily omitted the checksums of the downloaded artifacts. + 
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  files:
+    - path: /opt/bin/kubectl
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.26.0/bin/linux/amd64/kubectl
+    - path: /opt/bin/kubeadm
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.26.0/bin/linux/amd64/kubeadm
+    - path: /opt/bin/kubelet
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.26.0/bin/linux/amd64/kubelet
+    - path: /etc/systemd/system/kubelet.service
+      contents:
+        source: https://raw.githubusercontent.com/kubernetes/release/v0.14.0/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service
+    - path: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
+      contents:
+        source: https://raw.githubusercontent.com/kubernetes/release/v0.14.0/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf
+    - path: /etc/kubeadm.yml
+      contents:
+        inline: |
+          apiVersion: kubeadm.k8s.io/v1beta2
+          kind: InitConfiguration
+          nodeRegistration:
+            kubeletExtraArgs:
+              volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
+          ---
+          apiVersion: kubeadm.k8s.io/v1beta2
+          kind: ClusterConfiguration
+          controllerManager:
+            extraArgs:
+              flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
+systemd:
+  units:
+    - name: kubelet.service
+      enabled: true
+      dropins:
+        - name: 20-kubelet.conf
+          contents: |
+            [Service]
+            ExecStart=
+            ExecStart=/opt/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        ConditionPathExists=!/etc/kubernetes/kubelet.conf
+        [Service]
+        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
+        ExecStartPre=/opt/bin/kubeadm config images pull
+        ExecStartPre=/opt/bin/kubeadm init --config /etc/kubeadm.yml
+        ExecStartPre=/usr/bin/mkdir /home/core/.kube
+        ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/core/.kube/config
+        ExecStart=/usr/bin/chown -R core:core /home/core/.kube
+        [Install]
+        WantedBy=multi-user.target
+
+
+
+
+
+ + +This minimal configuration can be used with Flatcar on QEMU (:warning: be sure that the instance has enough memory: 4096mb is good). + +```bash +butane < config.yaml > config.json +./flatcar_production_qemu.sh -i config.json -- -display curses +kubectl get nodes +NAME STATUS ROLES AGE VERSION +localhost NotReady control-plane 6m5s v1.26.0 +``` + +The control plane will appear has non-ready until a CNI is deployed, here's an example with calico: +```bash +kubectl \ + apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml +kubectl get nodes +NAME STATUS ROLES AGE VERSION +localhost Ready control-plane 8m30s v1.26.0 +``` + +If you want to coordinate the nodes reboot when there is a new Kubernetes sysext image or a Flatcar update, you can deploy [`Kured`][kured]: +```bash +latest=$(curl -s https://api.github.com/repos/kubereboot/kured/releases | jq -r '.[0].tag_name') +kubectl apply -f "https://github.com/kubereboot/kured/releases/download/$latest/kured-$latest-dockerhub.yaml" +``` + +We can now prepare the nodes to join the cluster. + +### Setup the nodes + +Here's are two examples for a [butane][butane] configuration to setup the nodes. The first example is using the systemd-sysext approach to bring in the binaries and update them through systemd-sysupdate. The second approach fetches the binaries but has no way of updating them in-place. + +
+ +
+
+
+ This is an example using systemd-sysext and systemd-sysupdate. NOTE: We are using Kured to coordinate nodes reboot when there is a new Kubernetes sysext image available (or if Flatcar has been updated), hence the /run/reboot-required file. + 
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  links:
+    - target: /opt/extensions/kubernetes/kubernetes-v1.27.4-x86-64.raw
+      path: /etc/extensions/kubernetes.raw
+      hard: false
+  files:
+    - path: /etc/sysupdate.kubernetes.d/kubernetes.conf
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/kubernetes.conf
+    - path: /etc/sysupdate.d/noop.conf
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/noop.conf
+    - path: /opt/extensions/kubernetes/kubernetes-v1.27.4-x86-64.raw
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/kubernetes-v1.27.4-x86-64.raw
+systemd:
+  units:
+    - name: systemd-sysupdate.timer
+      enabled: true
+    - name: systemd-sysupdate.service
+      dropins:
+        - name: kubernetes.conf
+          contents: |
+            [Service]
+            ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
+            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
+            ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
+            ExecStartPost=/usr/bin/sh -c "[[ $(cat /tmp/kubernetes) != $(cat /tmp/kubernetes-new) ]] && touch /run/reboot-required"
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        [Service]
+        ExecStart=/usr/bin/kubeadm join $(output from 'kubeadm token create --print-join-command')
+        [Install]
+        WantedBy=multi-user.target
+
+
+
+
+
+ :warning: To ease the reading, we voluntarily omitted the checksums of the downloaded artifacts. + 
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  files:
+    - path: /opt/bin/kubeadm
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.26.0/bin/linux/amd64/kubeadm
+    - path: /opt/bin/kubelet
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.26.0/bin/linux/amd64/kubelet
+    - path: /etc/systemd/system/kubelet.service
+      contents:
+        source: https://raw.githubusercontent.com/kubernetes/release/v0.14.0/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service
+    - path: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
+      contents:
+        source: https://raw.githubusercontent.com/kubernetes/release/v0.14.0/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf
+systemd:
+  units:
+    - name: kubelet.service
+      enabled: true
+      dropins:
+        - name: 20-kubelet.conf
+          contents: |
+            [Service]
+            ExecStart=
+            ExecStart=/opt/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        [Service]
+        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
+        ExecStart=/opt/bin/kubeadm join $(output from 'kubeadm token create --print-join-command')
+        [Install]
+        WantedBy=multi-user.target
+
+
+
+
+
+ +This method is far from being ideal in terms of infrastructure as code as it requires a two steps manipulation: create the control plane to generate the join configuration then pass that configuration to the nodes. Other solutions exist to make things easier, like Cluster API or [Typhoon][typhoon]. + +### Switching from Docker to containerd for Kubernetes + +In Kubernetes v1.20, `dockershim` was deprecated and it has been [officially](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#dockershim-removed-from-kubelet) removed in Kubernetes v1.24. + +The `containerd` CRI plugin is enabled by default and you can use containerd for Kubernetes while still allowing Docker to function. +Recent Kubernetes versions will prefer containerd over Docker automatically on recent Flatcar versions. + +If you run `kubelet` in a Docker container, make sure it has access +to the following directories on the host file system: +- `/run/docker/libcontainerd/` +- `/run/containerd/` +- `/var/lib/containerd/` + +And that it has access to the following binaries on the host file system and that they are included in `PATH`: +- For Flatcar releases until major version 3760: + - `/run/torcx/unpack/docker/bin/containerd-shim-runc-v1` + - `/run/torcx/unpack/docker/bin/containerd-shim-runc-v2` +- For Flatcar releases above major version 3760: + - `/usr/bin/containerd-shim-runc-v1` + - `/usr/bin/containerd-shim-runc-v2` + +Finally, tell `kubelet` to use containerd by adding to it the following flags: +- `--container-runtime=remote` +- `--container-runtime-endpoint=unix:///run/containerd/containerd.sock` + +## Cluster API + +From the official [documentation][capi-documentation]: +> Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. + +As it requires to have some tools already installed on the OS to work correcly with CAPI, Flatcar images can be built using the [image-builder][image-builder] project. + +While CAPI is an evolving project and Flatcar support is in-progress regarding the various providers, here's the current list of supported providers: +* [AWS][capi-aws] +* [Azure][capi-azure] +* [OpenStack][openstack] +* [vSphere][capi-vsphere] + +## Kubespray + +Kubespray is an open-source project used to deploy production ready Kubernetes cluster, learn more about it on the [documentation][kubespray-documentation]. + +Based on users feedback, Flatcar is known to work with Kubespray - you can read more about it in this section: [https://kubespray.io/#/docs/flatcar][kubespray-documentation-flatcar]. + +[butane]: https://coreos.github.io/butane/ +[capi-documentation]: https://cluster-api.sigs.k8s.io/ +[capi-aws]: https://cluster-api-aws.sigs.k8s.io/ +[capi-azure]: https://capz.sigs.k8s.io/ +[capi-vsphere]: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/docs/ignition.md +[cilium]: https://github.com/flatcar/mantle/pull/292 +[flatcar-779]: https://github.com/flatcar/Flatcar/issues/779 +[flatcar-891]: https://github.com/flatcar/Flatcar/issues/891 +[image-builder]: https://github.com/kubernetes-sigs/image-builder +[kubenet]: https://github.com/flatcar/Flatcar/issues/579 +[kubespray-documentation]: https://kubespray.io +[kubespray-documentation-flatcar]: https://kubespray.io/#/docs/flatcar +[kured]: https://kured.dev/docs/ +[openstack]: https://cluster-api-openstack.sigs.k8s.io/clusteropenstack/configuration.html#ignition-based-images +[sysext-bakery]: https://github.com/flatcar/sysext-bakery +[typhoon]: https://typhoon.psdn.io/ diff --git a/content/docs/latest/container-runtimes/registry-authentication.md b/content/docs/latest/container-runtimes/registry-authentication.md new file mode 100644 index 00000000..bbd9b714 --- /dev/null +++ b/content/docs/latest/container-runtimes/registry-authentication.md @@ -0,0 +1,242 @@ +--- +title: Authenticating to Container Registries +description: Configuration examples for authenticating to different container registries. +weight: 50 +aliases: + - ../os/registry-authentication + - ../clusters/management/registry-authentication +--- + +Many container image registries require authentication. This document explains how to configure container management software like Docker, Kubernetes, rkt, and Mesos to authenticate with and pull containers from registries like [Quay][quay-site] and [Docker Hub][docker-hub-site]. + +## Using a Quay robot for registry auth + +The recommended way to authenticate container manager software with [quay.io][quay-site] is via a [Quay Robot][quay-robot]. The robot account acts as an authentication token with some nice features, including: + +* Readymade repository authentication configuration files +* Credentials are limited to specific repositories +* Choose from read, write, or admin privileges +* Token regeneration + +![Quay Robot settings][quay-bot-img] + +Quay robots provide config files for Kubernetes, Docker, Mesos, and rkt, along with instructions for using each. Find this information in the **Robot Accounts** tab under your Quay user settings. For more information, see the [Quay robot documentation][quay-robot]. + +## Manual registry auth setup + +If you are using a registry other than Quay (e.g., Docker Hub, Docker Store, etc) you will need to manually configure your credentials with your container-runtime or orchestration tool. + +### Docker + +The Docker client uses an interactive command to authenticate with a centralized service. + +```shell +docker login -u -p https://registry.example.io +``` + +This command creates the file `$HOME/.docker/config.json`, formatted like the following example: + +**/home/core/.docker/config.json:** + +```json +{ + "auths": { + "https://index.docker.io/v1/": { + "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=" + }, + "quay.io": { + "xxxx": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" + }, + "https://registry.example.io/v0/": { + "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=" + } + } +} +``` + +On Flatcar Container Linux, this process can be automated by writing out the config file during system provisioning [with a Butane Config][butane-configs]. Since the config is written to the `core` user's home directory, ensure that your systemd units run as that user, by adding, e.g., `User=core`. + +Docker also offers the ability to configure a credentials store, such as your operating system's keychain. This is outlined in the [Docker login documentation][docker-login]. + +### Kubernetes + +Kubernetes uses [*Secrets*][k8s-secrets] to store registry credentials. + +When manually configuring authentication with *any* registry in Kubernetes (including Quay and Docker Hub) the following command is used to generate the Kubernetes registry-auth secret: + +```shell +$ kubectl create secret docker-registry my-favorite-registry-secret --docker-username=giffee_lover_93 --docker-password='passphrases are great!' --docker-email='giffee.lover.93@example.com' --docker-server=registry.example.io +secret "my-favorite-registry-secret" created +``` + +If you prefer you can store this in a YAML file by adding the `--dry-run` and `-o yaml` flag to the end of your command and copying or redirecting the output to a file: + +```shell +kubectl create secret docker-registry my-favorite-registry [...] --dry-run -o yaml | tee credentials.yaml +``` + +```yaml +apiVersion: v1 +data: + .dockercfg: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx== +kind: Secret +metadata: + creationTimestamp: null + name: my-favorite-registry-secret +type: kubernetes.io/dockercfg +``` + +```shell +$ kubectl create -f credentials.yaml +secret "my-favorite-registry-secret" created +``` + +You can check that this secret is loaded with with the `kubectl get` command: + +```shell +$ kubectl get my-favorite-registry-secret +NAME TYPE DATA AGE +my-favorite-registry-secret kubernetes.io/dockercfg 1 30m +``` + +The secret can be used in a Pod spec with the `imagePullSecrets` variable: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: somepod + namespace: all +spec: + containers: + - name: web + image: registry.example.io/v0/giffee_lover_93/somerepo + + imagePullSecrets: + - name: my-favorite-registry-secret +``` + +For more information, check the [docker-registry Kubernetes secret][k8s-docker-registry] and [Kubernetes imagePullSecrets][k8s-image-pull] documentation. + +### rkt + +rkt stores registry-authentication in a JSON file stored in the directory `/etc/rkt/auth.d/`. + +`/etc/rkt/auth.d/registry.example.io.json`: + +```json +{ + "rktKind": "auth", + "rktVersion": "v1", + "domains": [ + "https://registry.example.io/v0/" + ], + "type": "basic", + "credentials": { + "user": "giffeeLover93", + "password": "passphrases are great!" + } +} +``` + +While you *can* embed your password in plaintext in this file, you should try using a disposable token instead. Check your registry documentation to see if it offers token-based authentication. + +Now rkt will authenticate with `https://registry.example.io/v0/` using the provided credentials to fetch images. + +For more information about rkt credentials, see the [rkt configuration docs][rkt-config]. + +Just like with the Docker config, this file can be copied to `/etc/rkt/auth.d/registry.example.io.json` on a Flatcar Container Linux node during system provisioning with [a Butane Config][butane-configs]. + +### Mesos + +Mesos uses a gzip-compressed archive of a `.docker/config.json` (directory and file) to access private repositories. + +Once you have followed the above steps to [create the docker registry auth config file][docker-instructions] create your Mesos configuration using `tar`: + +```shell +tar cxf ~/.docker/config.json +``` + +The archive secret is referenced via the `uris` field in a container specification file: + +```json +{ + "id": "/some/name/or/id", + "cpus": 1, + "mem": 1024, + "instances": 1, + "container": { + "type": "DOCKER", + "docker": { + "image": "https://registry.example.io/v0/giffee_lover_93/some-image", + "network": "HOST" + } + }, + + "uris": [ + "file:///path/to/registry.example.io.tar.gz" + ] +} +``` + +More thorough information about configuring Mesos registry authentication can be found on the ['Using a Private Docker Registry'][mesos-registry] documentation. + +## Copying the config file with a Butane Config + +[Butane Configs][butane-configs] can be used to provision a Flatcar Container Linux node on first boot. Here we will use it to copy registry authentication config files to their appropriate destination on disk. This provides immediate access to your private Docker Hub and Quay image repositories without the need for manual intervention. The same Butane Config file can be used to copy registry auth configs onto an entire cluster of Flatcar Container Linux nodes. + +Here is an example of using a Butane Config to write the `.docker/config.json` registry auth configuration file mentioned above to the appropriate path on the Flatcar Container Linux node: + +```yaml +variant: flatcar +version: 1.0.0 +storage: + files: + - path: /home/core/.docker/config.json + mode: 0644 + contents: + inline: | + { + "auths": { + "quay.io": { + "auth": "AbCdEfGhIj", + "email": "your.email@example.com" + } + } + } +``` + +Butane Configs can also download a file from a remote location and verify its integrity with a SHA512 hash: + +```yaml +variant: flatcar +version: 1.0.0 +storage: + files: + - path: /home/core/.docker/config.json + mode: 0644 + contents: + source: http://internal.infra.example.com/cluster-docker-config.json + verification: + hash: sha512-0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef +``` + +For details, check out the [Butane Config examples][butane-examples]. + +[config-valid]: https://coreos.com/validate/ +[docker-hub-site]: https://hub.docker.com/ +[docker-instructions]: #docker +[docker-login]: https://docs.docker.com/engine/reference/commandline/login/ +[docker-reg-v2]: https://docs.docker.com/registry/spec/auth/jwt/ +[k8s-docker-registry]: https://kubernetes.io/docs/user-guide/kubectl/kubectl_create_secret_docker-registry/ +[k8s-docker-registry]: https://kubernetes.io/docs/user-guide/kubectl/kubectl_create_secret_docker-registry/ +[k8s-image-pull]: https://kubernetes.io/docs/user-guide/images/ +[k8s-secrets]: https://kubernetes.io/docs/user-guide/secrets/ +[mesos-registry]: https://mesosphere.github.io/marathon/docs/native-docker-private-registry.html +[quay-bot-img]: ../img/quay-robot-screen.png +[quay-robot]: https://docs.quay.io/glossary/robot-accounts.html +[quay-site]: https://quay.io/ +[rfc-2397]: https://tools.ietf.org/html/rfc2397 +[rkt-config]: registry-authentication/#rkt +[butane-configs]: ../provisioning/config-transpiler +[butane-examples]: ../provisioning/config-transpiler/examples diff --git a/content/docs/latest/container-runtimes/switching-to-unified-cgroups.md b/content/docs/latest/container-runtimes/switching-to-unified-cgroups.md new file mode 100644 index 00000000..c68a7c98 --- /dev/null +++ b/content/docs/latest/container-runtimes/switching-to-unified-cgroups.md @@ -0,0 +1,153 @@ +--- +title: Switching to Unified Cgroups +linktitle: Switching to unified cgroups +description: Overview of changes necessary to use unified cgroups with Kubernetes +weight: 20 +aliases: +--- + +Beginning with Flatcar version 2969.0.0, Flatcar Linux has migrated to the unified +cgroup hierarchy (aka cgroup v2). Much of the container ecosystem has already +moved to default to cgroup v2. Cgroup v2 brings exciting new features in +areas such as eBPF and rootless containers. + +Flatcar nodes deployed prior to this change will be kept on cgroups v1 (legacy +hierarchy) and will require manual migration. During an update from an older +Flatcar version, a post update script does two things: + +* adds the kernel command line parameters `systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller` + to `/usr/share/oem/grub.cfg` +* creates a systemd drop-in unit at `/etc/systemd/system/containerd.service.d/10-use-cgroupfs.conf` that + configures `containerd` to keep using cgroupfs for cgroups. + +# Migrating old nodes to unified cgroups + +To undo the changes performed by the post update script, execute the following commands as root (or using `sudo`): + +```bash +rm /etc/systemd/system/containerd.service.d/10-use-cgroupfs.conf +sed -i -e '/systemd.unified_cgroup_hierarchy=0/d' /usr/share/oem/grub.cfg +sed -i -e '/systemd.legacy_systemd_cgroup_controller/d' /usr/share/oem/grub.cfg +reboot +``` + +# Starting new nodes with legacy cgroups + +Nodes deployed with the release incorporating the described changes use cgroups v2 by default. To revert to cgroups v1 on new +nodes during provisioning, use the following Ignition snippet (here as Butane YAML to be transpiled to Ignition JSON): + +```yaml +variant: flatcar +version: 1.0.0 +storage: + filesystems: + - name: "OEM" + mount: + device: "/dev/disk/by-label/OEM" + format: "btrfs" +kernel_arguments: + should_exist: + - systemd.unified_cgroup_hierarchy=0 + - systemd.legacy_systemd_cgroup_controller +systemd: + units: + - name: containerd.service + dropins: + - name: 10-use-cgroupfs.conf + contents: | + [Service] + Environment=CONTAINERD_CONFIG=/usr/share/containerd/config-cgroupfs.toml +``` + +However, the kernel commandline setting doesn't take effect on the first boot, and a reboot is required before the snippet becomes active. + +If your deployment can't tolerate the required reboot, consider using the following snippet to switch to legacy cgroups without a reboot. This is supported by Flatcar 3033.2.4 or newer: + +```yaml +variant: flatcar +version: 1.0.0 +storage: + filesystems: + - device: "/dev/disk/by-label/OEM" + format: "btrfs" + files: + - path: /etc/flatcar-cgroupv1 + mode: 0444 +kernel_arguments: + should_exist: + - systemd.unified_cgroup_hierarchy=0 + - systemd.legacy_systemd_cgroup_controller +systemd: + units: + - name: containerd.service + dropins: + - name: 10-use-cgroupfs.conf + contents: | + [Service] + Environment=CONTAINERD_CONFIG=/usr/share/containerd/config-cgroupfs.toml +``` + +Beware that over time it is expected that upstream projects will drop support for cgroups v1. + +**Known issues:** Unprivileged containers with user namespaces may lack permissions to access the bind mount ([Flatcar#722](https://github.com/flatcar/Flatcar/issues/722)) and unmounting the bind mount before starting the container is needed. + +## Generate AWS EC2 cgroups v1 AMIs + +The [`create_cgroupv1_ami.sh` script](https://raw.githubusercontent.com/flatcar/flatcar-docs/main/create_cgroupv1_ami.sh) performs the image modification as outlined above for you to upload your own Flatcar AMI that directly boots into cgroup v1 without needing an additonal reboot. + +# Kubernetes + +The unified cgroup hierarchy is supported starting with Docker v20.10 and +Kubernetes v1.19. Users that need to run older version will need to revert to +cgroups v1, but are urged to find a migration path. Flatcar now ships with Docker +v20.10, older versions can be deployed following the instructions on [running custom docker versions](use-a-custom-docker-or-containerd-version). + +Flatcar nodes that had Kubernetes deployed on them before the introduction of +cgroups v2 should be careful when migrating. Depending on the deployment method, +the `cgroupfs` cgroup driver may be hardcoded in the `kubelet` configuration. +Cgroups v2 are only supported with the `systemd` cgroup driver. See [configuring a cgroup driver][kube-cgroup-docs] in the Kubernetes documentation for a discussion of cgroup drivers and how to migrate nodes. We recommend redeploying Kubernetes on fresh nodes instead of migrating inplace. + +The cgroup driver used by `kubelet` should be the same as the one used by `docker` daemon. `docker` defaults to `systemd` cgroup driver when started on a system running cgroup v2 and `cgroupfs` when running on a system with cgroup v1. The cgroup driver can be explicitly configured for `docker` by extending `/etc/docker/daemon.json`: +```json +{ + "exec-opts": ["native.cgroupdriver=systemd"] +} +``` +or adding a `docker.service` drop-in at `/etc/systemd/system/docker.service.d/10-cgroup-v2.conf`: +```ini +[Service] +Environment="DOCKER_CGROUPS=--exec-opt native.cgroupdriver=systemd" +``` + +## Container Runtimes + +When deploying Kubernetes through `kubeadm`, the default container runtime on Flatcar is currently `dockershim`. In this setup, `kubelet` talks to `dockershim`, which talks to `docker`, which interfaces with `containerd`. The `SystemdCgroup` setting in `containerd`'s `config.toml` is ignored. `docker`'s cgroup driver and `kubelet` cgroup driver settings must match. Starting with Kubernetes v1.22, `kubeadm` will default to the `systemd` `cgroupDriver` setting if no setting is provided explicitly. Out of the box, Flatcar defaults are compatible with Docker and Kubernetes defaults - everything will use `systemd` cgroup driver. + +When using `kubeadm`, add the snippet to your `kubeadm-config.yaml` to configure the `kubelet` cgroup driver: + +```yaml +--- +kind: KubeletConfiguration +apiVersion: kubelet.config.k8s.io/v1beta1 +cgroupDriver: systemd +``` + +## Containerd + +If users choose the `containerd` runtime, they must ensure that `containerd`'s setting for `SystemdCgroup` is consistent with `kubelet` and `docker` settings. Flatcar enables `SystemdCgroup` by default for `containerd`. Users may change the setting to suit their deployment. +If you maintain your own containerd configuration or did follow the instructions on +[how to customize containerd configuration](customizing-docker), you should add the relevant lines to your `config.toml`: +```toml +version = 2 + +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] + # setting runc.options unsets parent settings + runtime_type = "io.containerd.runc.v2" + [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] + SystemdCgroup = true + ``` + +For a more detailed discussion of container runtimes, see the [Kubernetes documentation][kube-runtime-docs]. + +[kube-cgroup-docs]: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#migrating-to-the-systemd-driver +[kube-runtime-docs]: https://kubernetes.io/docs/setup/production-environment/container-runtimes/ diff --git a/content/docs/latest/container-runtimes/use-a-custom-docker-or-containerd-version.md b/content/docs/latest/container-runtimes/use-a-custom-docker-or-containerd-version.md new file mode 100644 index 00000000..9a2a1f06 --- /dev/null +++ b/content/docs/latest/container-runtimes/use-a-custom-docker-or-containerd-version.md @@ -0,0 +1,155 @@ +--- +title: Using a custom Docker or containerd version (LEGACY) +linktitle: Using custom versions +description: How to download and run a different version of docker or containerd than the one shipped by Flatcar. +weight: 30 +aliases: + - ../os/use-a-custom-docker-or-containerd-version +--- + +Some system tooling can't be run on Container Linux via containers and this is especially true for the container runtime itself. +As with other special binaries you want to bring to the system you can use an Ignition config that downloads the binaries. +Starting from Flatcar version ≥ 3185.0.0 a [systemd-sysext images](../provisioning/sysext/) should be used instead of the below. + +For custom Docker/containerd binaries sysext images are the recommended way. +However, the Flatcar versions below 3185.0.0 don't support it yet, and even in case support is there you may find it too complicated to build a sysext image and host it elsewhere. +In this case you can directly place the custom binaries to `/opt/bin/` as done by the following Butane Config which you can transpile to an Ignition config with [`butane`](../provisioning/config-transpiler/). + +This replicates the Docker setup as of Flatcar Container Linux 3033.2.3 but under `/etc` and `/opt/bin/`, and with additional support for the upstream Containerd socket location. +You can modify it to use different socket paths or plugins, or even only ship `containerd` if you don't need Docker. + +``` +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: prepare-docker.service + enabled: true + contents: | + [Unit] + Description=Unpack docker binaries to /opt/bin + ConditionPathExists=!/opt/bin/docker + [Service] + Type=oneshot + RemainAfterExit=true + Restart=on-failure + ExecStartPre=/usr/bin/mkdir -p /opt/bin + ExecStartPre=/usr/bin/tar -v --extract --file /opt/docker.tgz --directory /opt/ --no-same-owner + ExecStartPre=/usr/bin/rm /opt/docker.tgz + ExecStartPre=/usr/bin/sh -c "mv /opt/docker/* /opt/bin/" + ExecStart=/usr/bin/rmdir /opt/docker + [Install] + WantedBy=multi-user.target + - name: docker.socket + enabled: true + contents: | + [Unit] + PartOf=docker.service + Description=Docker Socket for the API + [Socket] + ListenStream=/var/run/docker.sock + SocketMode=0660 + SocketUser=root + SocketGroup=docker + [Install] + WantedBy=sockets.target + - name: docker.service + enabled: false + contents: | + [Unit] + Description=Docker Application Container Engine + After=containerd.service docker.socket network-online.target prepare-docker.service + Wants=network-online.target + Requires=containerd.service docker.socket prepare-docker.service + [Service] + Type=notify + EnvironmentFile=-/run/flannel/flannel_docker_opts.env + Environment=DOCKER_SELINUX=--selinux-enabled=true + # the default is not to use systemd for cgroups because the delegate issues still + # exists and systemd currently does not support the cgroup feature set required + # for containers run by docker + Environment=PATH=/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin + ExecStart=/opt/bin/dockerd --host=fd:// --containerd=/run/docker/libcontainerd/docker-containerd.sock $DOCKER_SELINUX $DOCKER_OPTS $DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ + ExecReload=/bin/kill -s HUP $MAINPID + LimitNOFILE=1048576 + # Having non-zero Limit*s causes performance problems due to accounting overhead + # in the kernel. We recommend using cgroups to do container-local accounting. + LimitNPROC=infinity + LimitCORE=infinity + # Uncomment TasksMax if your systemd version supports it. + # Only systemd 226 and above support this version. + TasksMax=infinity + TimeoutStartSec=0 + # set delegate yes so that systemd does not reset the cgroups of docker containers + Delegate=yes + # kill only the docker process, not all processes in the cgroup + KillMode=process + # restart the docker process if it exits prematurely + Restart=on-failure + StartLimitBurst=3 + StartLimitInterval=60s + [Install] + WantedBy=multi-user.target + - name: containerd.service + enabled: false + contents: | + [Unit] + Description=containerd container runtime + After=network.target prepare-docker.service + Requires=prepare-docker.service + [Service] + Delegate=yes + Environment=PATH=/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin + ExecStartPre=mkdir -p /run/docker/libcontainerd + ExecStartPre=ln -fs /run/containerd/containerd.sock /run/docker/libcontainerd/docker-containerd.sock + ExecStart=/opt/bin/containerd --config /etc/containerd/config.toml + KillMode=process + Restart=always + # (lack of) limits from the upstream docker service unit + LimitNOFILE=1048576 + LimitNPROC=infinity + LimitCORE=infinity + TasksMax=infinity + [Install] + WantedBy=multi-user.target +storage: + files: + - path: /etc/systemd/system-generators/torcx-generator + - path: /opt/docker.tgz + mode: 0644 + contents: + source: https://download.docker.com/linux/static/stable/x86_64/docker-20.10.12.tgz + verification: + hash: sha512-90c3ab8c465bfa6fa51e9e77cf5257ff4bf139723eeb4878afbf294e71a2f2f13558840708e392ff24f8b8853c519938013d4dff8d50b17d66ca0eeb6a1b3c1a + - path: /etc/containerd/config.toml + mode: 0644 + contents: + inline: | + version = 2 + # set containerd's OOM score + oom_score = -999 + [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] + # setting runc.options unsets parent settings + runtime_type = "io.containerd.runc.v2" + [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] + SystemdCgroup = true + links: + - path: /etc/extensions/docker-flatcar.raw + target: /dev/null + overwrite: true + - path: /etc/extensions/containerd-flatcar.raw + target: /dev/null + overwrite: true +``` + +While the system services have a `PATH` variable that prefers `/opt/bin/` by placing it first, you have to run the following command on every interactive login shell (also after `sudo` or `su`) to make sure you use the correct binaries. + +```sh +export PATH="/opt/bin:$PATH" +``` + +The empty file `/etc/systemd/system-generators/torcx-generator` serves the purpose of disabling Torcx to make sure it is not used accidentally in case `/opt/bin` was missing from the `PATH` variable. +Flatcar releases newer than major release 3760 do not ship torcx so that line can as well be removed from the above config. +However, leaving it in does not have any side effects. + +The `/etc/extensions/` symlinks make sure that the future built-in Docker/containerd sysext images won't be enabled. diff --git a/content/docs/latest/contribute/_index.md b/content/docs/latest/contribute/_index.md new file mode 100644 index 00000000..a4902e12 --- /dev/null +++ b/content/docs/latest/contribute/_index.md @@ -0,0 +1,58 @@ +--- +title: How to Contribute +content_type: contribute +weight: 130 +--- + +Flatcar documentation is released under the [Apache 2.0 License][asl], and we welcome contributions. Check out the [help-wanted tag][help-wanted] in this project's Issues list for good places to start participating. + +Submit fixes and additions in the form of [GitHub *Pull Requests* (PRs)][pull-requests]. The general process is the typical git fork-branch-PR-review-merge cycle: + +1. Fork this repository into your GitHub account +2. Make changes in a topic branch or your fork's `master` +3. Send a Pull Request from that topic branch to flatcar-linux/docs +4. Maintainers will review the PR and either merge it or make comments + +Cognizance of the tribal customs described and linked to below will help get your contributions incorporated with the greatest of ease. + +## Clear commit messages + +Commit messages follow a format that makes clear **what** changed and **why** it changed. The first line of each commit message should clearly state what module or file changed, summarize the change very briefly, and should end, without a period, somewhere short of 70 characters. After a blank line, the body of the commit message should then explain why the change was needed, with lines wrapped at 72 characters wide and sentences normally punctuated. Cite related issues or previous revisions as appropriate. For example: + +``` +ignition: Update etcd example to use %m + +Make the etcd configuration example use ignition's %m instead of the +ETCD_NAME environment variable. Fixes #123. +``` + +This format can be described somewhat more formally as: + +``` +: + + + +[