Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contrib: Deprecate torcx, ship containerd / docker as sysexts #1216

Merged
merged 40 commits into from
Oct 24, 2023
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
851009c
Install docker as a system extension, use manglefs script ( convert t…
krishjainx Jul 7, 2023
c026d9e
Remove app-arch/torcx from coreos-overlay
krishjainx Jul 7, 2023
3d17f65
Create seperate docker and containerd sysext
krishjainx Jul 15, 2023
2be2e88
Add manglefs script for containerd
krishjainx Jul 17, 2023
e5f1ecb
Only keep docker stuff in manglefs_docker
krishjainx Jul 17, 2023
e6bb547
Add mangle script for containerd
krishjainx Jul 17, 2023
9709a64
incorporate suggestions from Jeremi
krishjainx Jul 18, 2023
4b00bc1
tail, not head
krishjainx Jul 18, 2023
abe8bf6
Use helper function query_available_package()
krishjainx Jul 18, 2023
4e8b276
Remove duplicate making directory root_fs_dir/etc/extensions
krishjainx Jul 20, 2023
fe949f5
Make sure binary packages are built in build_packages already
krishjainx Jul 20, 2023
f6f5c32
add information about the packages contained in the built sysext
krishjainx Jul 21, 2023
4633cc3
Fix syntax errors, quoting
krishjainx Jul 24, 2023
298e283
fix permissions issue
krishjainx Jul 25, 2023
3c4639b
use sudo_clobber function
krishjainx Jul 25, 2023
cc029bb
use DEVCONTAINER=1
krishjainx Jul 25, 2023
9cede02
dev_container_util.sh: remove spurious DEVCONTAINER=1
t-lo Oct 5, 2023
9837c35
build_sysext: add sysext dependency handling
t-lo Oct 5, 2023
aae3fd8
prod_image_util.sh: remove pkgdb tarball (replaced by sysext)
t-lo Oct 5, 2023
8f8f262
torcx: remove from scripts, use docker+containerd sysexts
t-lo Oct 5, 2023
8e01a2c
build_image: add sysext command line option
t-lo Oct 6, 2023
c589fb8
coreos-overlay: removed all remaining app-torcx artifacts
t-lo Oct 6, 2023
89555ed
base OS sysexts: separate build script, inventory generation
t-lo Oct 10, 2023
e85ca49
ci-automation: remove torcx artifact handling and tests
t-lo Oct 12, 2023
6f651a2
build scripts: separate txt inventory; official naming for sysexts
t-lo Oct 13, 2023
35779f1
run_local_tests.sh: script for running tests on local builds
t-lo Oct 13, 2023
bd54e3a
.github: removed torcx artifact handling from github CI
t-lo Oct 16, 2023
8ef820a
vendor-testing/qemu_update.sh: make update payload configurable
t-lo Oct 17, 2023
919d368
mantle-container: bump image ref to support torcx removal
t-lo Oct 17, 2023
4285030
run_sdk_container: Add -U flag to use the version file unchanged
t-lo Oct 17, 2023
6512ef0
overlay coreos-base/update_engine: remove torcx sanity check
t-lo Oct 17, 2023
22582dc
changelog: add torcx removal, sysext migration
t-lo Oct 19, 2023
57ed77e
ci-automation: fix local image detection for nightly SDKs
t-lo Oct 19, 2023
b775036
torcx removal: address PR comments
t-lo Oct 20, 2023
c2d1c13
Apply bash scripting suggestions from code review
t-lo Oct 20, 2023
37cc05c
coreos/base/package.use: squashfs-tools: enable all compression formats
t-lo Oct 20, 2023
d744d32
build_sysext: optimise squashfs for size
t-lo Oct 20, 2023
6dd2f20
run_local_tests.sh: un-break variable quoting
t-lo Oct 20, 2023
65bf2bf
Update run_local_tests.sh: improve quoting
t-lo Oct 23, 2023
f81bbeb
build_sysext: run package inventorisation before mangle script
t-lo Oct 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add manglefs script for containerd
  • Loading branch information
krishjainx authored and t-lo committed Oct 23, 2023
commit 2be2e884b3a24fb9ffb2cf91a3c64544b866876d
2 changes: 1 addition & 1 deletion build_library/build_image_util.sh
Original file line number Diff line number Diff line change
@@ -637,7 +637,7 @@ finish_image() {


emerge-"${FLAGS_board}" app-containers/containerd
sudo "$(dirname ${BASH_SOURCE[0]})/../build_sysext" --board="${BOARD}" --image_builddir=${BUILD_DIR} --squashfs_base="${BUILD_DIR}/${image_sysext_base}" containerd-flatcar app-containers/containerd
sudo "$(dirname ${BASH_SOURCE[0]})/../build_sysext" --board="${BOARD}" --image_builddir=${BUILD_DIR} --squashfs_base="${BUILD_DIR}/${image_sysext_base}" --manglefs_script="$(dirname ${BASH_SOURCE[0]})/../manglefs_containerd" containerd-flatcar app-containers/containerd
sudo install -m 0644 -D "${BUILD_DIR}/containerd-flatcar.raw" "${root_fs_dir}"/usr/share/flatcar/
sudo mkdir -p "${root_fs_dir}"/etc/extensions/
sudo ln -sf /usr/share/flatcar/containerd-flatcar.raw "${root_fs_dir}"/etc/extensions/containerd-flatcar.raw
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pothos does this work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also don't quite understand why we first create a docker sysext pretending containerd is installed via ${PORTAGE_CONFIGROOT}/etc/portage/profile/package.provided and then create a containerd sysext anyway.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was done to have a split containerd and Docker sysext setup. This way users can disable Docker but keep our containerd.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The pretending of containerd being installed is to build the Docker sysext without containerd in it.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean whether the symlinking to etc at this phase of the build will work

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jepio to answer your question this works because of

# Backup the /etc contents to /usr/share/flatcar/etc to serve as
# source for creating missing files. Make sure that the preexisting
# /usr/share/flatcar/etc does not have any meaningful (non-empty)
# files, so we remove nothing important. There shouldn't be any
# symlinks either. Add "! -type d" to exclude directories as "stat"
# usually returns a size of a directory being 4096 or so.
if [[ $(sudo find "${root_fs_dir}/usr/share/flatcar/etc" -size +0 ! -type d 2>/dev/null | wc -l) -gt 0 ]]; then
die "Unexpected non-empty files in ${root_fs_dir}/usr/share/flatcar/etc"
fi
sudo rm -rf "${root_fs_dir}/usr/share/flatcar/etc"
sudo cp -a "${root_fs_dir}/etc" "${root_fs_dir}/usr/share/flatcar/etc"
.

Copy link
Member Author

@t-lo t-lo Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was done to have a split containerd and Docker sysext setup. This way users can disable Docker but keep our containerd.

Oh I understand the intent, but I'd argue that if we want stackable / dependent sysexts then we should make the packages shipped with each layer explicit, and re-use these when we build the next "upper" layer.

I've extended build_sysext.sh to cover this, and updated build_image_util.sh to use it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks more complex than it needs to be. How about just mount the whole previous sysext when building the next one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sysexts do not contain package information since they only cover /usr (and/or /opt). Installed packages are registered in sub-directories below /var/db/pkg.
(I wish this was easier though!)