Recent test failures in CI (apply caps: operation not permitted) #42892
Labels
Comments
|
Looks like it may be related to Ubuntu 20.04 / kernel 5.11. Diff between "passing" and "arm64"; diff --git a/./test-pass.txt b/./test-fail.txt
index 0712495..6ea9559 100644
--- a/./test-pass.txt
+++ b/./test-fail.txt
@@ -8,7 +8,7 @@
Go version: go1.16.6
Git commit: 3967b7d
Built: REDACTED
- OS/Arch: linux/amd64
+ OS/Arch: linux/arm64
Context: default
Experimental: true
@@ -19,7 +19,7 @@
Go version: go1.16.6
Git commit: 75249d8
Built: REDACTED
- OS/Arch: linux/amd64
+ OS/Arch: linux/arm64
Experimental: true
containerd:
Version: 1.4.9
@@ -40,7 +40,6 @@
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
- scan: Docker Scan (Docker Inc., v0.8.0)
Server:
Containers: 0
@@ -72,11 +71,11 @@
apparmor
seccomp
Profile: default
- Kernel Version: 5.4.0-1057-aws
- Operating System: Ubuntu 18.04.6 LTS
+ Kernel Version: 5.11.0-1017-aws
+ Operating System: Ubuntu 20.04.3 LTS
OSType: linux
- Architecture: x86_64
- CPUs: 2
+ Architecture: aarch64
+ CPUs: 4
Total Memory: 7.488GiB
Name: REDACTED
ID: REDACTED
@@ -89,7 +88,6 @@
127.0.0.0/8
Live Restore Enabled: true
- WARNING: No swap limit support
@@ -98,7 +96,7 @@
+ curl -fsSL -o /home/ubuntu/workspace/moby_PR-42890/check-config.sh https://raw.githubusercontent.com/moby/moby/2b0755b936416834e14208c6c37b36977e67ea35/contrib/check-config.sh
+ bash /home/ubuntu/workspace/moby_PR-42890/check-config.sh
warning: /proc/config.gz does not exist, searching other paths for kernel config ...
- info: reading kernel config from /boot/config-5.4.0-1057-aws ...
+ info: reading kernel config from /boot/config-5.11.0-1017-aws ...
Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
@@ -135,8 +133,7 @@
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- - CONFIG_MEMCG_SWAP_ENABLED: missing
- (cgroup swap accounting is currently not enabled, you can enable it by setting boot option "swapaccount=1")
+ (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
@@ -183,7 +180,7 @@
- CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
- "aufs":
- - CONFIG_AUFS_FS: enabled (as module)
+ - CONFIG_AUFS_FS: missing
- "btrfs":
- CONFIG_BTRFS_FS: enabled (as module)
- CONFIG_BTRFS_FS_POSIX_ACL: enabledDiff between "passing" and "group v2" (remove the "check-config.sh" output, as that's not ran on cgroup2 in jenkins): diff --git a/./test-pass.txt b/./test-fail2.txt
index 0712495..c2d3191 100644
--- a/./test-pass.txt
+++ b/./test-fail2.txt
@@ -1,3 +1,5 @@
+Also failing (cgroupv2);
+
<details>
@@ -55,8 +57,8 @@
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
- Cgroup Driver: cgroupfs
- Cgroup Version: 1
+ Cgroup Driver: systemd
+ Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
@@ -72,8 +74,9 @@
apparmor
seccomp
Profile: default
- Kernel Version: 5.4.0-1057-aws
- Operating System: Ubuntu 18.04.6 LTS
+ cgroupns
+ Kernel Version: 5.11.0-1017-aws
+ Operating System: Ubuntu 20.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
@@ -89,118 +92,6 @@
127.0.0.0/8
Live Restore Enabled: true
- WARNING: No swap limit support
|
|
I was wondering if it would be related to #42736, but that has been merged for some time, and CI passed on that PR (running on ubuntu 20.04). However it looks that;
Info from a "pass" on that PR:
OS:
|
|
So the error is emitted here; https://github.com/opencontainers/runc/blob/v1.0.1/libcontainer/init_linux.go#L195-L197 if err := w.ApplyCaps(); err != nil {
return errors.Wrap(err, "apply caps")
}And comes from; https://github.com/opencontainers/runc/blob/v1.0.1/libcontainer/capabilities/capabilities.go#L104-L111 // Apply sets all the capabilities for the current process in the config.
func (c *Caps) ApplyCaps() error {
c.pid.Clear(allCapabilityTypes)
for _, g := range capTypes {
c.pid.Set(g, c.caps[g]...)
}
return c.pid.Apply(allCapabilityTypes)
} |
|
@AkihiroSuda @kolyshkin any ideas? This something you've seen before? Seems like it's either related to |
|
Also curious where the 10 seconds timeout comes from in the moby/integration/container/health_test.go Lines 60 to 62 in 7b9275c Ah; looks like that's the config of moby/vendor/gotest.tools/v3/poll/poll.go Lines 39 to 41 in 776cadc |
This was referenced Oct 8, 2021
Open
This was referenced Oct 18, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Started to see these tests fail; not sure if something changed in our code, or if Jenkins agents were updated leading to this issue;
e.g. https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-42888/5/pipeline/
and this one failing on cgroups v2
Failing on (arm54, kernel 5.11):
Passing on (amd64, kernel 5.4):
Also failing (cgroupv2, kernel 5.11);
The text was updated successfully, but these errors were encountered: