app-containers: switch to upstream docker, containerd, runc, bump docker to v24 #1305

t-lo · 2023-10-25T12:18:46Z

This change removes Flatcar specific builds of docker[-cli], containerd, cri-tools, and runc and instead switches to upstream Gentoo ebuilds added to portage-stable.

The change updates docker to 24.0.6, and cri-tools to 1.27.0.

NOTE that there currently is no upstream ebuild for containerd-1.7.7, so this change adds that ebuild based on the upstream containerd-1.7.6 ebuild.

Flatcar customisations like systemd units etc. are now applied in the manglefs script of the respective sysexts, based on file system trees in coreos-overlay/coreos/sysext/(containerd|docker).

The build_sysext script has been extended by an option to strip all binaries in a sysext; the option is deactivated by default.

Lastly, the sysext command line syntax of build_image has been extended to allow specifying multiple packages for a sysext. This was necessary because docker-cli and docker do not have any runtime relationships and therefore must both be specified for installation to correctly mirror Flatcar's own docker packaging.

Closes flatcar/Flatcar#1091 .

Testing

Related PRs

Upstreaming

Upstream PR eclass/go-env.eclass: add compile env helper to enable cross compiling Go programs gentoo/gentoo#33539 has been opened to add cross-compile capabilities to Gentoo's Go related eclasses.

build_library/sysext_prod_builder

build_sysext

changelog/updates/2023-10-25-docker-24-critools-1.27.md

sdk_container/src/third_party/coreos-overlay/profiles/coreos/base/package.accept_keywords

sdk_container/src/third_party/portage-stable/app-containers/containerd/containerd-1.7.7.ebuild

changelog/changes/2023-10-25-docker-gentoo-upstream.md

sdk_container/src/third_party/portage-stable/eclass/go-env.eclass

sdk_container/src/third_party/portage-stable/eclass/go-module.eclass

sdk_container/src/third_party/portage-stable/eclass/golang-vcs-snapshot.eclass

sdk_container/src/third_party/portage-stable/eclass/golang-vcs.eclass

github-actions · 2023-10-26T18:42:57Z

Build action triggered: https://github.com/flatcar/scripts/actions/runs/6717733064

t-lo · 2023-10-27T16:10:42Z

Created flatcar/Flatcar#1223 to track the btrfs deprecation in docker 20 -> 24 and to run tests for gauging the fallout.

Created flatcar/Flatcar#1222 and flatcar/mantle#474 to ensure we continue shipping the devicemapper docker storage driver (which almost was silently discarded while working on the portage stable docker migration PR).

t-lo · 2023-10-28T08:50:26Z

All tests succeeded. Good to merge?

(Please also have a look at flatcar/mantle#474 ; it's not a requirement for this PR but a nice addition)

pothos · 2023-10-30T03:09:03Z

Some thoughts without looking at the details:

We should check for differences in the systemd units and the default config files (containerd toml)
Check if the cgroupv1 containerd config still works which was created for old nodes on update
Use this change here as opportunity to support drop-ins: [RFE] Define containerd drop-in config directory Flatcar#1127 [RFE] Include certs.d registry plugin in containerd config.toml Flatcar#583

t-lo · 2023-10-30T08:44:11Z

Some thoughts without looking at the details:

We should check for differences in the systemd units and the default config files (containerd toml)

I'm actually using the systemd unit from "our" ebuild (in https://github.com/flatcar/scripts/tree/t-lo/gentoo-upstream-containerd-docker/sdk_container/src/third_party/coreos-overlay/coreos/sysext/docker/usr/lib/systemd/system); it's added to the sysext via docker's manglefs script.

Same for the containerd toml files - as far as I can tell, upstream Gentoo's containerd ebuild does not even ship any tomls.

Check if the cgroupv1 containerd config still works which was created for old nodes on update

Is there a test for that?

Use this change here as opportunity to support drop-ins: [RFE] Define containerd drop-in config directory Flatcar#1127 [RFE] Include certs.d registry plugin in containerd config.toml Flatcar#583

Absolutely, but I'd put that into a follow-up PR. Should be straightforward after this one gets merged, and could be part of the same release.

This adds plain Gentoo upstream containerd ebuilds to coreos-overlay and copies containerd-1.7.6.ebuild to containerd-1.7.7.ebuild since upstream does not support 1.7.7 yet. Signed-off-by: Thilo Fromm <[email protected]>

Signed-off-by: Thilo Fromm <[email protected]>

Co-authored-by: Krzesimir Nowak <[email protected]> Signed-off-by: Thilo Fromm <[email protected]>

Signed-off-by: Thilo Fromm <[email protected]>

These were for coreos-overlay. The packages have been moved to portage-stable and are now handled by the weekly package updates automation. Signed-off-by: Thilo Fromm <[email protected]>

Signed-off-by: Thilo Fromm <[email protected]>

This change adds exporting CGO_* flags to go-env.eclass; the upstream pr gentoo/gentoo#33539 has been updated accordingly. Also, CGO_ENABLED=1 has been added to coreos/../make.conf to enable gco by default. This fixes a build issue for arm64 with Docker's device-mapper storage driver: daemon/graphdriver/devmapper/deviceset.go:306:25: undefined: devicemapper.SetTransactionID ... daemon/graphdriver/devmapper/deviceset.go:867:28: undefined: devicemapper.ErrEnxio daemon/graphdriver/devmapper/deviceset.go:867:28: too many errors gco is enabled on AMD64 by default, and cgo was always enabled in the coreos docker ebuilds. This way we retain that setting for the Gentoo ebuilds.

Signed-off-by: Thilo Fromm <[email protected]>

Temporary commit to use flatcar/mantle#475 for testing which ships the btrfs backwards compatibility test. Signed-off-by: Thilo Fromm <[email protected]>

t-lo · 2023-10-30T16:27:39Z

Rebased on latest main, restarted GH CI and Jenkins tests (and updated the test links in the summary).

t-lo · 2023-10-31T12:52:01Z

Both Github Actions and Jenkins CI are green once again. This now includes the docker brtfs storage update test: flatcar/mantle#475 ensuring that we won't break instances using btrfs storage when these update.

krnowak

So the only thing left are mantle tests, right?

t-lo · 2023-11-01T08:12:38Z

So the only thing left are mantle tests, right?

Yes - the most important one is the docker btrfs update test, the other one (devicemapper) is merged. Will wait for the open PR to be reviewed and merged, then update the mantle-container version in this PR before merging it.

t-lo · 2023-11-01T09:03:56Z

Updated the mantle ref to flatcar-master head; merging.

t-lo had a problem deploying to development October 25, 2023 12:18 — with GitHub Actions Error

t-lo mentioned this pull request Oct 25, 2023

kola: support docker in portage stable, add support for docker 24 flatcar/mantle#471

Merged

t-lo force-pushed the t-lo/gentoo-upstream-containerd-docker branch from c0abe1a to 1b698eb Compare October 25, 2023 13:39

t-lo had a problem deploying to development October 25, 2023 13:39 — with GitHub Actions Error

t-lo force-pushed the t-lo/gentoo-upstream-containerd-docker branch from 1b698eb to 76ead5b Compare October 25, 2023 13:51

t-lo had a problem deploying to development October 25, 2023 13:51 — with GitHub Actions Error

t-lo had a problem deploying to development October 25, 2023 15:22 — with GitHub Actions Error

t-lo had a problem deploying to development October 25, 2023 15:23 — with GitHub Actions Error

t-lo temporarily deployed to development October 25, 2023 16:08 — with GitHub Actions Inactive

t-lo marked this pull request as ready for review October 25, 2023 16:09

t-lo requested a review from a team October 25, 2023 16:09

krnowak reviewed Oct 26, 2023

View reviewed changes

t-lo force-pushed the t-lo/gentoo-upstream-containerd-docker branch 2 times, most recently from 49455db to 13ab672 Compare October 26, 2023 14:35

t-lo had a problem deploying to development October 26, 2023 14:36 — with GitHub Actions Error

t-lo temporarily deployed to development October 26, 2023 14:41 — with GitHub Actions Inactive

t-lo requested a review from krnowak October 26, 2023 15:10

krnowak reviewed Oct 26, 2023

View reviewed changes

t-lo force-pushed the t-lo/gentoo-upstream-containerd-docker branch 2 times, most recently from 0db405f to a0322e0 Compare October 27, 2023 08:52

t-lo temporarily deployed to development October 27, 2023 08:52 — with GitHub Actions Inactive

t-lo temporarily deployed to development October 27, 2023 10:25 — with GitHub Actions Inactive

t-lo had a problem deploying to development October 27, 2023 11:03 — with GitHub Actions Error

t-lo had a problem deploying to development October 27, 2023 12:21 — with GitHub Actions Error

t-lo temporarily deployed to development October 27, 2023 15:00 — with GitHub Actions Inactive

t-lo and others added 13 commits October 30, 2023 17:21

coreos-overlay/containerd: add Gentoo ebuilds and version 1.7.7

e2173e0

This adds plain Gentoo upstream containerd ebuilds to coreos-overlay and copies containerd-1.7.6.ebuild to containerd-1.7.7.ebuild since upstream does not support 1.7.7 yet. Signed-off-by: Thilo Fromm <[email protected]>

changelog: move Docker 24 btrfs driver notice to "changes".

f71ccfa

Signed-off-by: Thilo Fromm <[email protected]>

build_library/sysext_prod_builder: improve error message

383e609

Signed-off-by: Thilo Fromm <[email protected]>

portage-stable-packages-list: remove containerd

10b27ce

Signed-off-by: Thilo Fromm <[email protected]>

coreos/base/package.accept_keywords: sort alphabetically

47a74ba

Signed-off-by: Thilo Fromm <[email protected]>

sysext_prod_builder: Add suggestions from PR review

456b368

Co-authored-by: Krzesimir Nowak <[email protected]> Signed-off-by: Thilo Fromm <[email protected]>

partage stable docker 24: addressed PR feedback

f2a4b4a

Signed-off-by: Thilo Fromm <[email protected]>

.github/: remove docker, runc auto-update workflows

ed12285

These were for coreos-overlay. The packages have been moved to portage-stable and are now handled by the weekly package updates automation. Signed-off-by: Thilo Fromm <[email protected]>

build_sysext: let strip_binaries default to false

c522b04

Signed-off-by: Thilo Fromm <[email protected]>

run_local_tests.sh: simplify update test handline

28f4360

test qemu_update.sh: Add docker btrfs backwards compat test

b33a908

Signed-off-by: Thilo Fromm <[email protected]>

mantle-container: use PR #475 version for btrfs test

0476269

Temporary commit to use flatcar/mantle#475 for testing which ships the btrfs backwards compatibility test. Signed-off-by: Thilo Fromm <[email protected]>

t-lo force-pushed the t-lo/gentoo-upstream-containerd-docker branch from bd4f137 to 0476269 Compare October 30, 2023 16:25

t-lo temporarily deployed to development October 30, 2023 16:25 — with GitHub Actions Inactive

t-lo requested review from krnowak, pothos and a team October 30, 2023 16:26

chagelog: update docker btrfs backwards compatibility

63aabb9

t-lo temporarily deployed to development October 30, 2023 17:07 — with GitHub Actions Inactive

krnowak approved these changes Nov 1, 2023

View reviewed changes

mantle-container: bump version to include btrfs update test

ce88737

t-lo had a problem deploying to development November 1, 2023 09:02 — with GitHub Actions Failure

t-lo merged commit fd33dfd into main Nov 1, 2023
1 check failed

t-lo deleted the t-lo/gentoo-upstream-containerd-docker branch November 1, 2023 09:04

github-actions bot mentioned this pull request Nov 22, 2023

Monthly contributions report 2023-10-22 - 2023-11-21 flatcar/Flatcar#1259

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

app-containers: switch to upstream docker, containerd, runc, bump docker to v24 #1305

app-containers: switch to upstream docker, containerd, runc, bump docker to v24 #1305

t-lo commented Oct 25, 2023 •

edited

Loading

github-actions bot commented Oct 26, 2023 •

edited

Loading

t-lo commented Oct 27, 2023

t-lo commented Oct 28, 2023

pothos commented Oct 30, 2023

t-lo commented Oct 30, 2023

t-lo commented Oct 30, 2023

t-lo commented Oct 31, 2023

krnowak left a comment

t-lo commented Nov 1, 2023

t-lo commented Nov 1, 2023

app-containers: switch to upstream docker, containerd, runc, bump docker to v24 #1305

app-containers: switch to upstream docker, containerd, runc, bump docker to v24 #1305

Conversation

t-lo commented Oct 25, 2023 • edited Loading

github-actions bot commented Oct 26, 2023 • edited Loading

t-lo commented Oct 27, 2023

t-lo commented Oct 28, 2023

pothos commented Oct 30, 2023

t-lo commented Oct 30, 2023

t-lo commented Oct 30, 2023

t-lo commented Oct 31, 2023

krnowak left a comment

Choose a reason for hiding this comment

t-lo commented Nov 1, 2023

t-lo commented Nov 1, 2023

t-lo commented Oct 25, 2023 •

edited

Loading

github-actions bot commented Oct 26, 2023 •

edited

Loading