Skip to content

Latest commit

 

History

History
314 lines (233 loc) · 11.1 KB

working.md

File metadata and controls

314 lines (233 loc) · 11.1 KB
Raw
nav_order
4

Working with CoreOS Assembler

{: .no_toc }

  1. TOC {:toc}

Understanding "config git"

Conceptually, coreos-assembler ties together generating OSTree commits with disk images into a single "build schema". The build target is defined by the "config git" which should be in src/config (relative to the build directory).

We can hack on some local input configs by exporting them in the COREOS_ASSEMBLER_CONFIG_GIT env variable. For example:

$ export COREOS_ASSEMBLER_CONFIG_GIT=/path/to/github.com/coreos/fedora-coreos-config/
$ cosa init --force /dev/null
$ cosa fetch && cosa build

Components of "config git"

manifest.yaml

For generating OSTree commits, cosa uses manifest.yaml: An rpm-ostree "manifest" or "treefile", which mostly boils down to a list of RPMs and a set of rpm-md repositories they come from. It also supports postprocess to make arbitrary changes. See the rpm-ostree documentation for the Treefile format reference.

overlay.d/

coreos-assembler also supports a generic way to embed architecture-independent configuration and scripts by creating subdirectories in overlay.d/. Each subdirectory of the overlay.d. directory is added to the OSTree commit, in lexicographic order. It's recommended to name directories with a numeric prefix - e.g. 05core, 10extras. Non-directories are ignored. For example, a good practice is to add a README.md file into your overlay directories describing their structure.

image.yaml

This YAML file configures the output disk images. Supported keys are:

  • size: Size in GB for cloud images (OpenStack, AWS, etc.) Required.
  • extra-kargs: List of kernel arguments.
  • include: path to another YAML file to include. List values are appended. For other type, the values in the sourcing config win.

It's likely in the future we will extend this to support e.g. a separate /var partition or configuring the filesystem types. If you want to do anything like that today it requires forking the assembler and rebuilding it. See the fedora-coreos-config for an example.

Hacking on "config git"

First you can expand the size of the image; edit src/config/image.yaml and e.g. change 8 to 9. Rerun cosa build, and notice that the OSTree commit didn't change, but a new image is generated in builds. When you cosa run, you'll get it.

Another thing to try is editing src/config/manifest.yaml - add or remove entries from packages. You can also add local rpm-md file:/// repositories.

Using overrides

Development speed is closely tied to the "edit-compile-debug" cycle. coreos-assembler supports an overrides/ sub-directory of the coreos-assembler working directory, which allows easily overlaying locally-generated content on top of the base OS content.

There are two subdirectories of overrides/:

  • overrides/rootfs
  • overrides/rpm

Let's say you want to hack on both ostree and ignition-dracut. See for example this PR which added support for make install DESTDIR= to the latter. In general most upstream build systems support something like this; if they don't it's a good idea to add.

Concretely, if /path/to/cosa-workdir is where you ran cosa init, then after doing edits in a project, run a command like this from the source repository for the component:

$ make install DESTDIR=/path/to/cosa-workdir/overrides/rootfs

This would then install files like /path/to/cosa-buildroot/overrides/rootfs/usr/bin/ostree etc.

If you then run cosa build from the cosa workdir, those overrides will be automatically incorporated.

You can also choose to use overrides/rpm - this accepts pre-built binary RPMs. This can be convenient when you want to quickly test a binary RPM built elsewhere, or if you want to go through a more "official" build process. If any RPMs are present here, then coreos-assembler will automatically run createrepo_c and ensure that they are used in the build.

In the future, it's likely coreos-assembler will also support something like overrides/src which could be a directory of symlinks to local git repositories.

Using cosa build-fast

If you're working on e.g. the kernel or Ignition (things that go into the initramfs), then you probably need a cosa build workflow (or cosa buildinitramfs-fast, see below). However, let's say you want to test a change to something that runs purely in the real root, such as e.g. rpm-ostree, podman, or console-login-helper-messages. Rather than doing a full image build each time, a fast way to test out changes is to use cosa build-fast.

This command assumes you have a previous local coreos-assembler build. From the git checkout of the project you want to add:

$ export COSA_DIR=/srv/builds/fcos
$ cosa build-fast
$ cosa run

The cosa build-fast command will run make and inject the resulting binaries on a qcow2 overlay file, which will appear in your project working directory. The cosa run command similarly knows to look for these qcow2 files.

This will not affect the "real" cosa build in /srv/builds/fcos, but will use it as a data source.

Using cosa run --bind-ro for even faster iteration

This workflow can be used alongside cosa build-fast, or separate from it. If you invoke e.g.

$ cosa run --bind-ro ~/src/github/containers/podman,/run/workdir

The target VM will have the source directory bound in /run/workdir; then you can directly copy binaries from your development environment into the VM.

If you are running cosa in a container, you will have to change your current working directory to a parent directory common to both project directories and use relative paths:

$ cd ~
$ cosa run \
      --qemu-image src/fcos/build/latest/x86_64/fedora-coreos-*.x86_64.qcow2 \
      --bind-ro src/github/containers/podman,/run/workdir

Then in the booted VM, /run/workdir will point to the libpod directory on your host, allowing you to directly execute binaries from there. You can also use e.g. rpm-ostree usroverlay and then copy binaries from your host /run/workdir into the VM's rootfs.

(This currently only works on Fedora CoreOS which ships 9p, not RHCOS. A future version will use https://virtio-fs.gitlab.io/ )

Using host binaries

Another related trick is:

$ cosa run --bind-ro /usr/bin,/run/hostbin

Then in the VM you have e.g. /run/hostbin/strace. (This may fail in some scenarios where your dev container is different than the target).

If you are running cosa in a container, you will only have access to the binary installed in this container. You can install binaries before launching the VM with:

$ cosa shell
$ sudo dnf install ...
$ cosa run --bind-ro /usr/bin,/run/hostbin

Using cosa buildinitramfs-fast

If you're iterating on changes just to the initramfs, you can also use cosa buildinitramfs-fast. For example, suppose you are working on ignition. Follow these steps:

$ make
$ install -D -m 0755 bin/amd64/ignition /path/to/cosadir/overrides/initramfs/usr/bin/ignition
$ cd /path/to/cosadir
$ cosa buildinitramfs-fast
$ cosa run --qemu-image tmp/fastbuild/fastbuildinitrd-fedora-coreos-qemu.qcow2 -i config.ign

(Or instead of cosa run use e.g. cosa kola to run tests, etc.)

If you're adding new fields to the Ignition experimental spec, -i will silently remove those fields, since the copy of Ignition that's vendored into mantle won't know about them yet. Instead, you can specify that kola should pass the config to the machine unmodified:

$ kola qemuexec --qemu-image tmp/fastbuild/fastbuildinitrd-fedora-coreos-qemu.qcow2 -i config.ign --ignition-direct

You'll need to manually configure autologin in the Ignition config, since kola won't be able to do it for you.

Performing an in-place OS update manually

The output of coreos-assembler is conceptually two things:

  • an ostree container image
  • disk images (ISO, AWS AMI, qemu .qcow2, etc)

In many cases, rather than booting from a new disk image with the new OS, you will want to explicitly test in-place upgrades. This uses an ostree native container, which is in the form of an .ociarchive file generated by cosa build ostree (as well as the default cosa build, which also generates a qemu disk image).

You will need to make the container image available to your targeted system (VM or physical). One way to do this is to push the container to a public registry such as quay.io:

cosa push-container quay.io/exampleuser/fcos

Performing an in-place update from the cosa build output boils down to invoking a command of the form:

$ rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/exampleuser/fcos
$ systemctl reboot

Using different CA certificates

If you need access to CA certificates on your host (for example, when you need to access a git repo that is not on the public Internet), you can mount in the host certificates using the COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS variable.

NOTE Sharing the /etc/pki/ca-trust directory may be blocked by SELinux so you may have to use a directory with the system_u:object_r:container_file_t:s0 file context.

$ export COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS='-v /etc/pki/ca-trust:/etc/pki/ca-trust:ro'
$ cosa init https://github.com/coreos/fedora-coreos-config.git
$ cosa fetch && cosa build

See this Stack Overflow question for additional discussion.

Running CoreOS Assembler in OpenShift on Google Compute Platform

This is a guide to run a COSA pod in an OpenShift 4.6+ cluster in Google Compute Platform (GCP).

Install the KVM device plugin.

Up to this point, you needed to be kubeadmin. From this point on though, best practice is to switch to an "unprivileged" user.

(In fact the steps until this point could be run by a separate team that manages the cluster; other developers could just use it as unprivileged users)

Personally, I added a httpasswd identity provider and logged in with a password.

I also did oc new-project coreos-virt etc.

Schedule a cosa pod:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: cosa
  name: cosa
spec:
  containers:
  - args:
    - shell
    - sleep
    - infinity
    image: quay.io/coreos-assembler/coreos-assembler:latest
    name: cosa
    resources:
      requests:
        # Today COSA hardcodes 2048 for launching VMs.  We could
        # probably shrink that in the future.
        memory: "3Gi"
        devices.kubevirt.io/kvm: "1"
      limits:
        memory: "3Gi"
        devices.kubevirt.io/kvm: "1"
    volumeMounts:
    - mountPath: /srv
      name: workdir
  volumes:
  - name: workdir
    emptyDir: {}
  restartPolicy: Never

Then oc rsh pods/cosa and you should be able to ls -al /dev/kvm - and cosa build etc!