Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libct/cg/sd: fix SkipDevices for systemd #2958

Merged
merged 2 commits into from
May 28, 2021

Conversation

kolyshkin
Copy link
Contributor

@kolyshkin kolyshkin commented May 22, 2021

  1. libct/cg/sd: fix SkipDevices for systemd

    Commit 108ee85 (PR libct/cgroups: add SkipDevices to Resources #2490) adds SkipDevices flag, which is used by kubernetes
    to create cgroups for pods.

    Unfortunately the above commit falls short, and systemd DevicePolicy and
    DeviceAllow properties are still set, which requires kubernetes to set
    "allow everything" rule.

    This commit fixes this: if SkipDevices flag is set, we return
    Device* properties to allow all devices.

  2. libct/cg/sd: add SkipDevices unit test

    The idea is to mimic what kubelet is doing, with minimum amount of code.

    First, create a slice with SkipDevices=true. It should have access to
    all devices.

    Next, create a scope within the above slice, allowing access to /dev/full
    only.

    Check that within that scope we can only access /dev/full and not other
    devices (such as /dev/null).

    Repeat the test with SkipDevices=false, make sure we can not access any
    devices (as they are disallowed by a parent cgroup). This is done only
    to assess the test correctness.

    NOTE that cgroup v1 and v2 behave differently for SkipDevices=false
    case, and thus the check is different. Cgroup v1 returns EPERM on
    writing to devices.allow, so cgroup manager's Set() fails, and we check
    for a particular error from m.Set(). Cgroup v2 allows to create a child
    cgroup, but denies access to any device (despite access being enabled)
    -- so we check the error from the shell script running in that cgroup.
    Again, this is only about SkipDevices=false case.

Previous discussions on topic: #2490 (comment), kubernetes/kubernetes#92862 (comment)

Fixes: 108ee85

Commit 108ee85 adds SkipDevices flag, which is used by kubernetes
to create cgroups for pods.

Unfortunately the above commit falls short, and systemd DevicePolicy and
DeviceAllow properties are still set, which requires kubernetes to set
"allow everything" rule.

This commit fixes this: if SkipDevices flag is set, we return
Device* properties to allow all devices.

Fixes: 108ee85
Signed-off-by: Kir Kolyshkin <[email protected]>
cyphar
cyphar previously approved these changes May 25, 2021
Copy link
Member

@cyphar cyphar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@AkihiroSuda
Copy link
Member

Would it be possible to have some tests?

@kolyshkin
Copy link
Contributor Author

Would it be possible to have some tests?

I started working on something that mimics the kubelet behavior...

@kolyshkin kolyshkin force-pushed the cgroup-skip-devices branch 3 times, most recently from 99fde82 to b9347d1 Compare May 26, 2021 03:28
@cyphar
Copy link
Member

cyphar commented May 26, 2021

CentOS failure seems legit -- SELinux?

=== RUN   TestSkipDevicesFalse
    systemd_test.go:129: failed to write "c 1:7 rwm": write /sys/fs/cgroup/devices/system.slice/system-runc_test_pods.slice/test-SkipDevices.scope/devices.allow: operation not permitted
--- FAIL: TestSkipDevicesFalse (0.20s)

@cyphar
Copy link
Member

cyphar commented May 26, 2021

Once this is merged I will send out the v1.0.0 vote (it would be nice to do the release -- as @h-vetinari suggested -- on the 3rd of June since that's the 5-year anniversary of 1.0.0-rc1).

The idea is to mimic what kubelet is doing, with minimum amount of code.

First, create a slice with SkipDevices=true. It should have access to
all devices.

Next, create a scope within the above slice, allowing access to /dev/full
only.

Check that within that scope we can only access /dev/full and not other
devices (such as /dev/null).

Repeat the test with SkipDevices=false, make sure we can not access any
devices (as they are disallowed by a parent cgroup). This is done only
to assess the test correctness.

NOTE that cgroup v1 and v2 behave differently for SkipDevices=false
case, and thus the check is different. Cgroup v1 returns EPERM on
writing to devices.allow, so cgroup manager's Set() fails, and we check
for a particular error from m.Set(). Cgroup v2 allows to create a child
cgroup, but denies access to any device (despite access being enabled)
-- so we check the error from the shell script running in that cgroup.
Again, this is only about SkipDevices=false case.

Signed-off-by: Kir Kolyshkin <[email protected]>
@kolyshkin
Copy link
Contributor Author

Test case fixed, PR description updated.

Copy link
Member

@cyphar cyphar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Will send the v1.0.0 vote out today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants