rootless: use fuse-overlayfs snapshotter by default #2278

AkihiroSuda · 2021-05-25T06:32:27Z

kind v0.11 uses overlayfs containerd snapshotter for rootless, but rootless overlayfs has very complex requirement:

(debian kernel && modprobe opton permit_mounts_in_userns==1) || (ubuntu kernel) || (kernel >= 5.11 && !selinux)

And yet Debian variant is known to be broken (while Ubuntu variant is not): moby/moby#42302

OTOH, fuse-overlayfs just requires kernel >= 4.18.

The snapshotter can be still rolled back to overlayfs by setting KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=overlayfs.

Close #2275

Node image: akihirosuda/tmp-kind-node:g42e6ce83-v1.21.1

The snapshotter can be overridden with `$KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER`. e.g., `KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=overlayfs`. Close 2275 Signed-off-by: Akihiro Suda <[email protected]>

k8s-ci-robot · 2021-05-25T06:32:35Z

Hi @AkihiroSuda. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

AkihiroSuda · 2021-05-25T07:20:41Z

CI failure is unrelated

URL: ["https://vagrantcloud.com/fedora/33-cloud-base"]
Error: The requested URL returned error: 403

aojea · 2021-05-25T10:03:51Z

CI failure is unrelated

URL: ["https://vagrantcloud.com/fedora/33-cloud-base"]
Error: The requested URL returned error: 403

oh,you already detected it

BenTheElder · 2021-05-26T08:20:14Z

images/base/Dockerfile

+# Configure containerd-fuse-overlayfs snapshotter binary from upstream
+ARG CONTAINERD_FUSE_OVERLAYFS_VERSION="1.0.2"
+ARG CONTAINERD_FUSE_OVERLAYFS_TARBALL="v${CONTAINERD_FUSE_OVERLAYFS_VERSION}/containerd-fuse-overlayfs-${CONTAINERD_FUSE_OVERLAYFS_VERSION}-linux-${TARGETARCH}.tar.gz"
+ARG CONTAINERD_FUSE_OVERLAYFS_URL="https://github.com/containerd/fuse-overlayfs-snapshotter/releases/download/${CONTAINERD_FUSE_OVERLAYFS_TARBALL}"


Will this eventually be merged in? It seems like this is crucial enough to bake in 🤔

No, the containerd/containerd repo does not accept a new built-in snapshotter.

If having a separate binary is not comfortable, maybe we can consider patching kind-ci/containerd-nightlies. (Just adding import _ "github.com/containerd/fuse-overlayfs-snapshotter/plugin" line)

No, the containerd/containerd repo does not accept a new built-in snapshotter.

That seems unfortunate, especially with it being essentially required for rootless. I hope they may reconsider for this case.

If having a separate binary is not comfortable, maybe we can consider patching kind-ci/containerd-nightlies. (Just adding import _ "github.com/containerd/fuse-overlayfs-snapshotter/plugin" line)

We probably don't want to patch containerd sources, but maybe worth considering if we can do it with our own stub source importing or something.

Users building images for compliance (e.g. EKS AIUI) will still need to update their builds to fix this though. It's less that we need to add a binary to the image and more that there's another unverified binary they may need to notice and start overriding + building. If we keep doing that it makes updating kind a riskier proposition.

It's also a little annoying to need to run more services though / extra entrypoint complexity though and I imagine the binary size is smaller when merged in. But it is not a deal breaker. It's still more important to make sure kind works.

images/base/files/etc/systemd/system/containerd-fuse-overlayfs.service

BenTheElder · 2021-05-26T08:24:26Z

pkg/cluster/internal/providers/docker/provision.go

@@ -243,6 +243,11 @@ func runArgsForNode(node *config.Node, clusterIPFamily config.ClusterIPFamily, n
 		"--volume", "/var",
 		// some k8s things want to read /lib/modules
 		"--volume", "/lib/modules:/lib/modules:ro",
+		// propagate KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER to the entrypoint script
+		"-e", "KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER",


Is there any reason to not just enable this when rootless and forget the env?

This env var is for users who want to opt-in to KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=overlayfs (kernel-mode real overlayfs) for better performance.
This env var is not needed for most users.

Ah, hopefully in the future there some way we can check if real overlayfs is viable and automate selection.

BenTheElder · 2021-05-26T08:25:25Z

pkg/cluster/internal/providers/podman/provision.go

+		// propagate KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER to the entrypoint script
+		"-e", "KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER",
+		// enable /dev/fuse explicitly for fuse-overlayfs
+		// (Rootless Podman does not automatically mount /dev/fuse with --privileged)


What's the reason for that? I'm only familiar with the rough concept of fuse 😅

Because fuse-overlayfs needs /dev/fuse to mount FUSE.

Because fuse-overlayfs needs /dev/fuse to mount FUSE.

Sorry, that part I get, what I meant was: What's the reason for (the comment):

Rootless Podman does not automatically mount /dev/fuse with --privileged

Why this deviation?

Kinda bug / TODO (of runc) due to complexity

it sounds like they don't because it may not be present see e.g. openshift/enhancements#362

BenTheElder · 2021-05-27T20:35:35Z

btw I'd be happy to sponsor your kubernetes organization membership, which would remove the needs-ok-to-test from all future PRs immediately, if nothing else 😅

BenTheElder · 2021-05-27T20:36:32Z

/ok-to-test

https://github.com/kubernetes/community/blob/master/community-membership.md#member

I'm sure @aojea would be happy to be a second sponsor

aojea · 2021-05-27T20:37:28Z

I'm sure @aojea would be happy to be a second sponsor

yep, absolutely

AkihiroSuda · 2021-05-27T20:43:11Z

Thanks, opened a request in kubernetes/org#2740

I should have requested this a few years ago... 😅

BenTheElder · 2021-05-27T22:46:43Z

#2287 we're going to cut a release with the runc update / CVE patch, but after that's done I intend to update the base image and move forward with this PR. thank you for your patience.

BenTheElder

/lgtm
/approve

k8s-ci-robot · 2021-05-28T09:11:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AkihiroSuda, BenTheElder

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [BenTheElder]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

BenTheElder · 2021-11-15T21:42:13Z

pkg/cluster/internal/providers/docker/provision.go

+		"-e", "KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER",
+		// enable /dev/fuse explicitly for fuse-overlayfs
+		// (Rootless Docker does not automatically mount /dev/fuse with --privileged)
+		"--device", "/dev/fuse",


what if fuse isn't enabled / mounted?

rootless: use fuse-overlayfs snapshotter by default

42e6ce8

The snapshotter can be overridden with `$KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER`. e.g., `KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=overlayfs`. Close 2275 Signed-off-by: Akihiro Suda <[email protected]>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 25, 2021

k8s-ci-robot requested review from amwat and aojea May 25, 2021 06:32

k8s-ci-robot added the area/provider/podman Issues or PRs related to podman label May 25, 2021

AkihiroSuda mentioned this pull request May 25, 2021

rootless: use fuse-overlayfs by default #2275

Closed

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label May 25, 2021

AkihiroSuda closed this May 25, 2021

AkihiroSuda reopened this May 25, 2021

AkihiroSuda closed this May 25, 2021

AkihiroSuda reopened this May 25, 2021

AkihiroSuda closed this May 25, 2021

AkihiroSuda reopened this May 25, 2021

AkihiroSuda closed this May 25, 2021

AkihiroSuda reopened this May 25, 2021

AkihiroSuda closed this May 26, 2021

AkihiroSuda reopened this May 26, 2021

BenTheElder reviewed May 26, 2021

View reviewed changes

images/base/files/etc/systemd/system/containerd-fuse-overlayfs.service Show resolved Hide resolved

BenTheElder reviewed May 26, 2021

View reviewed changes

BenTheElder self-assigned this May 27, 2021

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 27, 2021

AkihiroSuda mentioned this pull request May 27, 2021

REQUEST: New membership for AkihiroSuda kubernetes/org#2740

Closed

6 tasks

BenTheElder approved these changes May 28, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 28, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 28, 2021

BenTheElder merged commit 0d37803 into kubernetes-sigs:main May 28, 2021

BenTheElder mentioned this pull request May 28, 2021

bump images #2290

Merged

aojea mentioned this pull request Nov 3, 2021

Control Plane Fails to Start on Fedora 34 Silverblue #2521

Closed

BenTheElder reviewed Nov 15, 2021

View reviewed changes

jawnsy mentioned this pull request Nov 15, 2021

Graceful degradation if fuse device is missing #2540

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rootless: use fuse-overlayfs snapshotter by default #2278

rootless: use fuse-overlayfs snapshotter by default #2278

AkihiroSuda commented May 25, 2021 •

edited

Loading

k8s-ci-robot commented May 25, 2021

AkihiroSuda commented May 25, 2021

aojea commented May 25, 2021

BenTheElder May 26, 2021

AkihiroSuda May 26, 2021

AkihiroSuda May 26, 2021

BenTheElder May 26, 2021

BenTheElder May 26, 2021

AkihiroSuda May 26, 2021

BenTheElder May 26, 2021

BenTheElder May 26, 2021

AkihiroSuda May 26, 2021

BenTheElder May 26, 2021 •

edited

Loading

AkihiroSuda May 27, 2021

BenTheElder May 27, 2021

BenTheElder Nov 15, 2021

BenTheElder commented May 27, 2021

BenTheElder commented May 27, 2021

aojea commented May 27, 2021

AkihiroSuda commented May 27, 2021

BenTheElder commented May 27, 2021

BenTheElder left a comment

k8s-ci-robot commented May 28, 2021

BenTheElder Nov 15, 2021

rootless: use fuse-overlayfs snapshotter by default #2278

rootless: use fuse-overlayfs snapshotter by default #2278

Conversation

AkihiroSuda commented May 25, 2021 • edited Loading

k8s-ci-robot commented May 25, 2021

AkihiroSuda commented May 25, 2021

aojea commented May 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder May 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder commented May 27, 2021

BenTheElder commented May 27, 2021

aojea commented May 27, 2021

AkihiroSuda commented May 27, 2021

BenTheElder commented May 27, 2021

BenTheElder left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 28, 2021

Choose a reason for hiding this comment

AkihiroSuda commented May 25, 2021 •

edited

Loading

BenTheElder May 26, 2021 •

edited

Loading