docker driver allow_caps and cap_add don't work as documented (fails closed)

[tgross]

While testing #16643 which fixes the behavior of cap_add for the exec driver, I discovered that the docker driver has the same bug. This is because upstream Docker has the same bug (see moby/moby#38664 moby/moby#8460) but has intentionally decided not to fix it. This results in unexpected behavior in the docker task driver as well.

Note: this bug "fails closed". It does not open capabilities unexpectedly, so it's not a security bug.

tl;dr: for unprivileged users:

• allow_caps can only reduce, not expand, the permitted set of capabilities.
• cap_add can only remove selected capabilities from the cap_drop set (making them allowed again).

Unfortunately there's no workaround for this issue short of getting Docker to fix it upstream, which they've previously rejected. So I'm going to update the documentation to note the current behavior and point to this issue.

---

To verify this behavior, I used the following jobspec. Note that in order to properly test net_bind_service you need to set the sysctl net.ipv4.ip_unprivileged_port_start 1024. This is because upstream Docker has worked around this whole thing for the common case of binding to low ports by setting the default sysctl net.ipv4.ip_unprivileged_port_start 0.

job spec

job "example" {

  group "group" {

    task "task" {
      driver = "docker"
      # user   = "nobody"

      config {
        image    = "busybox:1"
        # cap_add  = ["net_bind_service"]
        # cap_drop = ["all"]

        command  = "/bin/sleep"
        args = ["600"]
        sysctl = {
          "net.ipv4.ip_unprivileged_port_start" = "1024"
        }
      }
    }
  }
}

I tested this jobspec along several dimensions: user namespace remapping, the allow_caps field on the plugin, the user field on the task, and the cap_add/cap_drop field. For each test I ran the job, alloc exec'd into the allocation (if it ran at all), and ran two tests for the netbind and netraw capabilities:

/ $ nc -lvp 443
nc: bind: Permission denied

/ $ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: permission denied (are you root?)

Some combinations would not run at all, because cap_add is restricted to those capabilities allowed in allow_caps. But as it turns out, allow_caps doesn't work to expand the permitted set either. This results in task events like the following:

2023-03-28T10:21:05-04:00 Driver Failure Failed to create container configuration for image "busybox:1" ("sha256:7cfbbec8963d8f13e6c70416d6592e1cc10f47a348131290a55d43c3acab3fb9"): driver does not allow the following capabilities: netraw

I didn't test every combination out of practicality, but there's more than enough data to draw conclusions.

userns remapping?	user	allow_caps	cap_add	cap_drop	runs?	netbind?	netraw?
yes	root	[<defaults>]	[]	[]	yes ✔️	yes ✔️	no ✔️
yes	root	[<defaults>, "net_raw"]	[]	[]	yes ✔️	yes ✔️	yes ✔️
yes	nobody	[<defaults>]	[]	[]	yes ✔️	no ❌	no ❌
yes	nobody	[<defaults>, "net_raw"]	[]	[]	yes ✔️	no ❌	no ❌
no	root	[<defaults>]	[]	[]	yes ✔️	yes ✔️	no ✔️
no	root	[<defaults>]	[]	["net_bind_service"]	yes ✔️	no ✔️	no ✔️
no	root	[<defaults>]	["net_bind_service"]	["all"]	yes ✔️	yes ✔️	no ✔️
no	root	["chown"]	["all"]	[]	no ✔️	-	-
no	root	["all"]	["net_raw"]	[]	yes ✔️	yes ✔️	yes ✔️
no	root	[<defaults>]	["net_raw"]	[]	no ✔️	-	-
no	nobody	[<defaults>]	["net_bind_service"]	[]	yes ✔️	no ❌	no ✔️
yes	nobody	["all"]	[]	[]	yes ✔️	no ❌	no ❌
yes	nobody	["all"]	["net_raw"]	[]	yes ✔️	no ❌	no ❌
yes	nobody	[<defaults>]	["net_bind_service"]	[]	yes ✔️	no ❌	no ❌