Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

images/krte: add Rootless Docker (and systemd as a dependency) #30744

Closed
wants to merge 3 commits into from
Closed

images/krte: add Rootless Docker (and systemd as a dependency) #30744

wants to merge 3 commits into from

Conversation

AkihiroSuda
Copy link
Member

This commit adds the following runtime variables to support running Rootless Docker:

  • KRTE_SYSTEMD=true (default: false): enable systemd
  • KRTE_SYSTEMD_ROOTLESS=true (default: false): switch to a non-root user via systemd.
    The KRTE container itself still has to be run as the root, so DO NOT specify securityContext.runAsUser.

e.g.,

$ docker build -t krte .

$ docker run -t --rm --privileged \
  -e KRTE_SYSTEMD=true \
  -e KRTE_SYSTEMD_ROOTLESS=true \
  -e DOCKER_IN_DOCKER_ENABLED=true \
  krte \
  docker info
wrapper.sh] [INFO] Re-executing in systemd: `docker info`
...
wrapper.sh] [INFO] Switching to rootless: `docker info`
...
wrapper.sh] [TEST] Running Test Command: `docker info`
...
Cgroup Driver: systemd
 Cgroup Version: 2
...
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
...

This is planned to be used for testing the KubeletInUserNamespace feature gate (KEP 2033),
by running Kubernetes inside Rootless Docker.

FAQ: Why is systemd needed?

For delegating cgroup controllers to a non-root user.
https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation

systemd is still opt-in for the default "rootful" mode, so this change will not affect the existing test jobs.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 18, 2023
@k8s-ci-robot k8s-ci-robot added area/images sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Sep 18, 2023
This was referenced Sep 18, 2023
Open
Closed
@AkihiroSuda AkihiroSuda mentioned this pull request Sep 18, 2023
Merged
@AkihiroSuda
Copy link
Member Author

@aojea @BenTheElder Could you take a look?

aojea
aojea reviewed Sep 25, 2023
View reviewed changes
images/krte/Dockerfile Outdated Show resolved Hide resolved
@AkihiroSuda
images/krte/Dockerfile: fulfill GO_VERSION when unspecified
ceb4467 
This helps locally experimenting the image with `docker build`.
Prior to this commit, `docker build` was failing unless
`--build-arg GO_VERSION=...` was explicitly specified.

Signed-off-by: Akihiro Suda <[email protected]>
@AkihiroSuda
images/krte/Dockerfile: move COPY instruction for cache efficiency
69e8cd1 
The `RUN` cache below the `COPY` instruction is invalidated on updating
the script.

Signed-off-by: Akihiro Suda <[email protected]>
@AkihiroSuda
images/krte: add Rootless Docker (and systemd as a dependency)
751a5c4 
This commit adds the following runtime variables to support running Rootless Docker:
- `KRTE_SYSTEMD=true` (default: `false`): enable systemd
- `KRTE_SYSTEMD_ROOTLESS=true` (default: `false`): switch to a non-root user via systemd.
   The KRTE container itself still has to be run as the root,
   so DO NOT specify `securityContext.runAsUser`.

e.g.,
```console
$ docker build -t krte .

$ docker run -t --rm --privileged \
  -e KRTE_SYSTEMD=true \
  -e KRTE_SYSTEMD_ROOTLESS=true \
  -e DOCKER_IN_DOCKER_ENABLED=true \
  krte \
  docker info
wrapper.sh] [INFO] Re-executing in systemd: `docker info`
...
wrapper.sh] [INFO] Switching to rootless: `docker info`
...
wrapper.sh] [TEST] Running Test Command: `docker info`
...
Cgroup Driver: systemd
 Cgroup Version: 2
...
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
...
```

This is planned to be used for testing the `KubeletInUserNamespace` feature gate (KEP 2033),
by running Kubernetes inside Rootless Docker.

[FAQ: Why is systemd needed?]
For delegating cgroup controllers to a non-root user.
https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation

systemd is still opt-in for the default "rootful" mode, so this change will not affect the existing test jobs.

Signed-off-by: Akihiro Suda <[email protected]>
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 25, 2023
@AkihiroSuda AkihiroSuda requested a review from aojea September 30, 2023 06:12
@AkihiroSuda
Copy link
Member Author

ping @aojea @BenTheElder 🙏

BenTheElder
BenTheElder reviewed Oct 13, 2023
View reviewed changes
Copy link
Member

@BenTheElder BenTheElder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

I've never tried to run systemd as anything other than PID1, will that work? In Kubernetes's CI there's an injected PID1 binary wrapping the test command unfortunately. https://docs.prow.k8s.io/docs/components/pod-utilities/entrypoint/
/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 13, 2023
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 13, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AkihiroSuda, BenTheElder

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 13, 2023
@BenTheElder
Copy link
Member

Attempting to catch back up today on all the things, if yes go ahead and /hold cancel, if no then ... it may be tricky to work around that. There's been past discussion of this problem but no immediate plans to resolve it, unit tests etc run oblivious to the entrypoint detail.

@AkihiroSuda
Copy link
Member Author

I've never tried to run systemd as anything other than PID1, will that work?

Unlikely

In Kubernetes's CI there's an injected PID1 binary wrapping the test command unfortunately. https://docs.prow.k8s.io/docs/components/pod-utilities/entrypoint/

Is this a hard requirement?
Would it be possible to disable it for KRTE_SYSTEMD_ROOTLESS tests?

@dims
Copy link
Member

dims commented Oct 15, 2023

@AkihiroSuda the entrypoint process is for injecting a bunch of stuff, collecting logs etc, so someone will have to do the same work somehow i think.

@BenTheElder
Copy link
Member

Is this a hard requirement?

Ish? Without it we don't get the log uploads etc. so ...

The injected entrypoint redirects logs, handles signals, etc.

https://docs.prow.k8s.io/docs/components/pod-utilities/

It should be possible to design the same functionality without this sort of entrypoint wrapper, but it doesn't exist, and would take a bit of work. It's a pretty old unresolved topic #8779 (comment)

@BenTheElder
Copy link
Member

It's possible to handle these things some other way but it will be high-friction.

I think the fastest path to getting rootless CI up is probably to spin up AWS/GCP VMs from a prowjob.

@AkihiroSuda AkihiroSuda marked this pull request as draft October 16, 2023 19:30
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 16, 2023
@AkihiroSuda
Copy link
Member Author

Thank you, will consider using VMs

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Oct 16, 2023 •
edited
Loading

@BenTheElder
Can we extend https://github.com/kubernetes-sigs/kubetest2/tree/master/kubetest2-gce to install rootless kind inside a GCE instance? (Probably it should be a new deployer ?)

@BenTheElder
Copy link
Member

Can we extend https://github.com/kubernetes-sigs/kubetest2/tree/master/kubetest2-gce to install rootless kind inside a GCE instance? (Probably it should be a new deployer ?)

Yes, It would need to be a new deployer, and we're hoping to phase out the kube-up bash in favor of a more mature tool (kops), kubernetes/enhancements#4250.

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Oct 19, 2023 •
edited
Loading

Wrote a deployer: https://github.com/rootless-containers/kubetest2-kindinv

@AkihiroSuda AkihiroSuda mentioned this pull request Oct 20, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BenTheElder BenTheElder BenTheElder left review comments

@munnerz munnerz Awaiting requested review from munnerz

@aojea aojea Awaiting requested review from aojea

Assignees

@BenTheElder BenTheElder

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/images cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@AkihiroSuda @k8s-ci-robot @BenTheElder @dims @aojea