Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drivers/exec: Fix handling of capabilities for unprivileged tasks #16643

Merged
merged 3 commits into from
Mar 28, 2023
Merged

drivers/exec: Fix handling of capabilities for unprivileged tasks #16643

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
559b360
drivers/exec: Fix handling of capabilities for unprivileged tasks
elprans Mar 23, 2023
8bf0677
Fix test
elprans Mar 27, 2023
d626fed
Expand the test and add a changelog entry
elprans Mar 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .changelog/16643.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:bug
driver/exec: Fixed a bug where `cap_drop` and `cap_add` would not expand capabilities
```
11 changes: 10 additions & 1 deletion drivers/shared/executor/executor_linux.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -526,8 +526,17 @@ func configureCapabilities(cfg *lconfigs.Config, command *ExecCommand) {
}
default:
// otherwise apply the plugin + task capability configuration
//
// The capabilities must be set in the Ambient set as libcontainer
// performs `execve`` as an unprivileged user. Ambient also requires
// that capabilities are Permitted and Inheritable. Setting Effective
// is unnecessary, because we only need the capabilities to become
// effective _after_ execve, not before.
cfg.Capabilities = &lconfigs.Capabilities{
Bounding: command.Capabilities,
Bounding: command.Capabilities,
Permitted: command.Capabilities,
Inheritable: command.Capabilities,
Ambient: command.Capabilities,
}
}
}
Expand Down
43 changes: 33 additions & 10 deletions drivers/shared/executor/executor_linux_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -628,27 +628,40 @@ func TestExecutor_Capabilities(t *testing.T) {
testutil.ExecCompatible(t)

cases := []struct {
user string
caps string
user string
capAdd []string
capDrop []string
capsExpected string
}{
{
user: "nobody",
caps: `
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
capsExpected: `
CapInh: 00000000a80405fb
CapPrm: 00000000a80405fb
CapEff: 00000000a80405fb
CapBnd: 00000000a80405fb
CapAmb: 0000000000000000`,
CapAmb: 00000000a80405fb`,
},
{
user: "root",
caps: `
capsExpected: `
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000`,
},
{
user: "nobody",
capDrop: []string{"all"},
capAdd: []string{"net_bind_service"},
capsExpected: `
CapInh: 0000000000000400
CapPrm: 0000000000000400
CapEff: 0000000000000400
CapBnd: 0000000000000400
CapAmb: 0000000000000400`,
Comment on lines +659 to +663
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was doing a little testing locally without Nomad to compare this approach to what Docker does, and I was a bit surprised to see they seem to have the same problem we do!

$ docker run -it \
    --user nobody \
    --sysctl net.ipv4.ip_unprivileged_port_start=1024 \
    --cap-drop ALL \
    --cap-add NET_BIND_SERVICE \
    busybox:1 nc -lvp 443
nc: bind: Permission denied

When I dug into that, I found that it looks like they decided not to fix this moby/moby#38664 moby/moby#8460 in lieu of telling folks to set the net.ipv4.ip_unprivileged_port_start, which is obviously not the only capability that people want (Docker sets it to 0 by default!). I think we're doing the right thing here but I want to pull in @picatz and @shoenig for a consultation on this just to make sure I'm not missing something obvious.

Copy link
Contributor Author

@elprans elprans Mar 27, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which is obviously not the only capability that people want

Yes, another example of where this issue hits us: we sometimes need to use a template to generate a wrapper shell script over an artifact-downloaded binary. There is no way to specify the desired file mode in the artifact stanza, so the binary ends up without an executable bit set. Consequently, without a working cap_add = ["fowner"] the exec driver is effectively broken for this use case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've opened #16692 with some notes about how this same issue impacts the docker driver and can't be fixed because the problem is upstream, which will result in some docs updates. I'm going to give this one more pass and make sure it works as I'd expect with a restrictive allow_caps.

},
}

for _, c := range cases {
Expand All @@ -662,7 +675,17 @@ CapAmb: 0000000000000000`,
execCmd.ResourceLimits = true
execCmd.Cmd = "/bin/bash"
execCmd.Args = []string{"-c", "cat /proc/$$/status"}
execCmd.Capabilities = capabilities.NomadDefaults().Slice(true)

capsBasis := capabilities.NomadDefaults()
capsAllowed := capsBasis.Slice(true)
if c.capDrop != nil || c.capAdd != nil {
calcCaps, err := capabilities.Calculate(
capsBasis, capsAllowed, c.capAdd, c.capDrop)
require.NoError(t, err)
execCmd.Capabilities = calcCaps
} else {
execCmd.Capabilities = capsAllowed
}

executor := NewExecutorWithIsolation(testlog.HCLogger(t))
defer executor.Shutdown("SIGKILL", 0)
Expand Down Expand Up @@ -690,7 +713,7 @@ CapAmb: 0000000000000000`,
return s
}

expected := canonical(c.caps)
expected := canonical(c.capsExpected)
tu.WaitForResult(func() (bool, error) {
output := canonical(testExecCmd.stdout.String())
if !strings.Contains(output, expected) {
Expand Down