[FEATURE] Make github actions build docker images and push to ghcr.io

[afq984]

Is your feature request related to a problem? Please describe.
docker.io/thesofproject/sof AFAIK is built manually and can become stale from time to time

Describe the solution you'd like
make GitHub Actions build docker images on: schedule and push GitHub Action built docker images to ghcr.io (GitHub container registry)

Describe alternatives you've considered

• Do nothing.
• Build on: push: branches: [main]. sof_builder takes too long to build and most build dependencies are actually not pinned by thesofproject/sof so I think it doesn't make too much sense to tie the action run to thesofproject/sof commits.

Additional context
I tried the setup on my account, highlights of steps below:

---

Allow the repository actions to push to ghcr.io containers

Using GITHUB_TOKEN is the preferred method of accessing ghcr.io in actions (instead of using a personal access token). However the docs are a bit stale. What I did was:

1. Get github personal access token here. This is used just to create the container package.

2. Create ghcr.io package

   docker login ghcr.io -u afq984  # enter token when prompted with password
   docker tag hello-world ghcr.io/afq984/sof
   docker push ghcr.io/afq984/sof
   

3. Go to the package settings page of the created container package. Add the sof repository to the package to allow it to push to the registry

   

4. Repeat the above for sofqemu

5. Maybe need to tweak https://github.com/thesofproject/sof/settings/actions. The defaults worked for me but YMMV.

Setup github actions

Demos:

• Workflow YAML
• Workflow Run
• Published Images sof, sofqemu

[afq984]

cc @cujomalainey @marc-hb

[marc-hb]

Build on: push: branches: [main]. sof_builder takes too long to build and most build dependencies are actually not pinned > by thesofproject/sof so I think it doesn't make too much sense to tie the action run to thesofproject/sof commits.

It is possible to trigger some actions only when certain files/directories are modified, the Zephyr project uses this:

https://github.com/trilom/file-changes-action

https://github.com/zephyrproject-rtos/test_results/blob/ef9cc5a9387649/.github/workflows/check_results.yml#L33
uses: trilom/[email protected]

EDIT: this project looks orphaned. Also, does it have 1000+ dependencies!?
https://github.com/trilom/file-changes-action/graphs/contributors
https://github.com/trilom/file-changes-action/network/dependencies

[marc-hb]

Automated builds are nice, but without testing they're not worth much. That's why Docker images are built manually right now: because images have to be manually tested anyway. So for now I don't see the value of storing in some cloud all these docker images (one for every PR?) if they're not tested at all.

By the way how does ghcr.io compare with Docker Hub that we use currently, why do you seem to recommend a switch to it? Is its pricing there: https://github.com/features/packages#pricing ?

[afq984]

It is possible to trigger some actions only when certain files/directories are modified

yes that's possible, but I think we should also build periodically since I don't think we can detect xtensa-overlay, crosstool-ng & friends' changes easily now.

Also nit about uses: trilom/[email protected]: we should use trilom/file-changes-action@96385226e7e8530961a9bfebcd319aa054e8012c instead to avoid supply chain attacks as tags are mutable, unless we treat github.com/trilom as a trusted source.

So for now I don't see the value of storing in some cloud all these docker images (one for every PR?) if they're not tested at all.

I once wanted to use new toolchains added in the overlays but the it was not in the docker.io image. Would be nice if there's a grab & go container to use, as building the toolchain takes a long time.

For the testing concern, I think we can instead only tag verified images as latest

Or alternatively still use docker as source of truth, and let github just host unstable images as a first step.

Is its pricing there: https://github.com/features/packages#pricing ?

Yes, the page says GitHub Packages is free for public repositories

One more thing I'd like to mention is last time (~1 yrs) I tried ghcr.io's network bandwidth is not very good, which needs to be re-evaluated.

[afq984]

One more thing I'd like to mention is last time (~1 yrs) I tried ghcr.io's network bandwidth is not very good, which needs to be re-evaluated.

A clean pull from

• docker.io/thesofproject/sof took 60.13 seconds
• ghcr.io/afq984/sof took 67.60 seconds

While the docker.io registry download is much faster (I don't have a good way to measure it though), the time is dominated by image extraction so the difference isn't that much.

[marc-hb]

yes that's possible, but I think we should also build periodically since I don't think we can detect xtensa-overlay, crosstool-ng & friends' changes easily now.
I once wanted to use new toolchains added in the overlays but the it was not in the docker.io image. Would be nice if there's a grab & go container to use, as building the toolchain takes a long time.

Thanks for asking very good questions and starting this discussion. There is one simple way to solve most of these problems: 1. Extract sof/scripts/docker_build/ into a new git repo / github project. There is no reason to have a very tight coupling of the versions of our Docker image (or the toolchains) with the sof git repo as we have now. The former are external and optional dependencies. I don't think this tight coupling was by design. 2. In this new docker-image repo, add a new manifest / simple config file with the git versions of the overlays cloned by default by CI. Right now the build scripts just fetch the latest versions which is not reproducible and not good. They should still be able to fetch the latest versions but not in CI by default.

For the testing concern, I think we can instead only tag verified images as latest

I'm not familiar with docker image versioning but I'm afraid that's not enough separation. I mean when I type docker image history I wouldn't want to see a mix of officially tested and releases images mixed up with images built for random PRs from random users and it would be very inconvenient when trying to "bisect" a Docker image regression (and there are security concerns here too). How about different image names? How do other projects solve this problem, what are the best practices here? Maybe https://github.com/zephyrproject-rtos/docker-image have solved this already (I haven't looked at it yet)

[marc-hb]

While the docker.io registry download is much faster

There is a good chance the Github registery is very fast when used from Github-hosted Github Actions.

(I don't have a good way to measure it though),

Maybe this if you really really want to test
https://devops.stackexchange.com/questions/2731/downloading-docker-images-from-docker-hub-without-using-docker

the time is dominated by image extraction so the difference isn't that much.

Right, a large optimization of a small part is actually a small optimization. As long as the performance is comparable I would focus on all the other aspects instead: features and integration, reliability, ease of use, rate limits, price...

[afq984]

Extract sof/scripts/docker_build/ into a new git repo / github project

Yeah I imagine that'd make a lot of things easier

add a new manifest / simple config file with the git versions of the overlays cloned by default by CI

maybe git submodule is a good candidate?

docker image history

The docs isn't very clear about what this command does, but it seems to list what each image layer adds on top of the previous layer, rather than the history of different builds:

% docker image history docker.io/thesofproject/sof
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
f16acc0f8076   3 months ago   /bin/sh -c #(nop) WORKDIR /home/sof/work/sof…   0B        
<missing>      3 months ago   /bin/sh -c #(nop)  USER sof                     0B        
<missing>      3 months ago   |9 CLONE_DEFAULTS=--depth 5 CT_NG_REPO=https…   0B        
<missing>      3 months ago   |9 CLONE_DEFAULTS=--depth 5 CT_NG_REPO=https…   410MB     
<missing>      3 months ago   /bin/sh -c #(nop)  ARG NEWLIB_REPO=https://g…   0B        
<missing>      3 months ago   /bin/sh -c #(nop)  ENV PATH=/home/sof/work/x…   0B        
<missing>      3 months ago   /bin/sh -c #(nop)  ENV PATH=/home/sof/work/x…   0B        
...

Given that I think having a lot of builds won't actually pollute its output.

"bisect" a Docker image regression

docker images similarly to git can be tagged. It looks like this in docker hub's UI and this in ghcr's UI.

Assuming we have a dedicated sof-docker repository and use submodules to track dependencies,
We can map git branches/tags to docker tags:

• Each git tag v0.1, v0.2 etc is mapped to a docker image with the same tag.
• Docker tag latest is the default tag. We can tag a commit as latest to produce the corresponding image or just use the main/master branch for that.
• Build nightly tagged images from main. The difference is dependencies are sync'd to HEAD before building.

This way, we'll have view of latest, nightly, v0.3, v0.2, v0.1 on dockerhub/ghcr's web UI.
zephyrproject-rtos/docker-image's publish workflow is very similar. I imagine other CI jobs will only ever want to use v-* tagged images or latest. When a regression happens, we just need to update the latest tag to point to a good version.

[marc-hb]

maybe git submodule is a good candidate?

Submodules have very few fans, I think we should move to west instead which is what Zephyr uses. Long story in #3517.

In this particular case though I think a "stable" config file with few revisions would be much simpler and do the job. Either submodules or west would be overkill and unnecessarily require people to learn a new tool (git submodules have been "new" to me for 15 years, I have to refresh my memories every single time)

The docs isn't very clear about what this command [docker image history] does, but it seems to list what each image layer adds on top of the previous layer, rather than the history of different builds:

My bad. I think I finally found why I was ignorant about versioning Docker images: because Docker provides no such feature at all, everyone must roll their own versioning. It's very time consuming to search documentation of a feature that does not exist; fortunately there is stackoverflow :-)

Considering Docker's "roll your own" versioning approach, using different image names for official versus PR images seems like a bare minimum to me.

Each git tag v0.1, v0.2 etc is mapped to a docker image with the same tag.

Sounds good. For PR images we could use the PR number + the git sha.

Build nightly tagged images from main.

Considering how infrequently our docker image has been moving so far, I don't see the point of rebuilding the same image every night over and over again, sounds like noise. Nightly builds are useful there are a significant number of relatively fast moving parts. In such fast-paced environments you cannot build every combination of every git push or merge in every repo, so nightly builds provide useful checkpoints. However in the very different and slow moving case of our docker image and toolchains we can build every incoming code change. Building for every docker image change will result in much fewer images in total and more images different from each other.

Unless there are plans for some new, fast-paced toolchain developments that I'm not aware of? Scary :-)

When a regression happens, we just need to update [move back] the latest tag to point to a good version

Right, and not call a tag that moves back latest but something else like stable for instance.

[greg-intel]

Thanks for the information and verification @afq984. After working with some docker images today, I can see how there'd be a benefit to potentially leveraging some of this.

[afq984]

un-assigning myself as not having cycles to work on this

[lgirdwood]

@greg-intel is this something you can own ?

[marc-hb]

After using the same Docker image for a long period we had a recent burst of upgrades because of topologies v2 and there will be a couple more coming (see #5323 + links, maybe #5386 some day) but I think this was unusual, I expect the tempo to slow down again. This means the return on investment of automating some testing will be low, more effort than testing manually.

tl;dr: I think this is a P3 right now.

[greg-intel]

Yeah, I agree with Marc. Also, automated means someone's not actively vetting the image, which means we'd maybe want more framework than running the docker build and upload commands.

TBH, it's entirely possible, but I haven't given it too much thought.

But if someone finds themselves with the bandwidth to work on this, I'd support their effort.

[marc-hb]

The Docker image with the Zephyr SDK is available on both docker.io and ghcr.io. So I submitted a switch of our Zephyr build to ghcr:

• [SKIP CI] .github/zephyr: download docker image from ghcr.io instead of docker.io #5738

It's not faster. I still think it's valuable, see why in #5738

[afq984]

Reading #5738, I do feel this FR is essentially two independent things combined together:

1. Use CI to build docker images
2. Host the docker image also on ghcr.io

While there's no great demand on (1), how about hosting the docker image on both docker.io & ghcr.io? The benefits would be the same as #5738, not sure if it's worth the maintenance burden though.

[marc-hb]

This could be useful https://github.com/marketplace/actions/docker-setup-buildx