[1.11-rc2] Read-only /proc/sys breaks compatibility

[errordeveloper]

Output of docker version:

Client:
 Version:      1.11.0-rc2
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   388f544
 Built:        Fri Mar 25 20:30:45 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0-rc2
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   388f544
 Built:        Fri Mar 25 20:30:45 2016
 OS/Arch:      linux/amd64```

Output of docker info:

Containers: 19
 Running: 0
 Paused: 0
 Stopped: 19
Images: 3
Server Version: 1.11.0-rc2
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 56
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.1.19-boot2docker
Operating System: Boot2Docker 1.11.0-rc2 (TCL 7.0); HEAD : 9afac98 - Fri Mar 25 20:42:01 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996 MiB
Name: kube-dev
ID: RRPD:TK6H:7JNN:JHTZ:IRB5:KIXL:UZ2D:YK54:IG7I:A7ZU:I23E:NVUF
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 12
 Goroutines: 30
 System Time: 2016-03-30T09:45:11.114379566Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=vmwarefusion

Steps to reproduce the issue:

1. Run any container either with --privileged and try changing settings on eth0 interface

$ docker run --privileged alpine sh -c 'echo 5 >/proc/sys/net/ipv4/neigh/eth0/base_reachable_time'
sh: can't create /proc/sys/net/ipv4/neigh/eth0/base_reachable_time: Read-only file system

Describe the results you received:

Not able to changing network interface settings.

Describe the results you expected:

Expected to work. Used to work up to 1.10.

Additional information you deem important (e.g. issue happens only occasionally):

This breaks compatibility with all prior releases.

[crosbymichael]

@errordeveloper since you are a privileged container is it feasible for you to use umount /proc/sys when you want to make these changes or is that not an option?

[rade]

@crosbymichael Code similar to @errordeveloper's is in Weave Net. It is technically possible for us to remount the fs, but that requires a) us to make a change and publish new images, and b) all our users to upgrade to these new images in order to run Docker 1.11.

I image there might be other images out there with similar code that might be harder to change.

[crosbymichael]

@rade thanks. I'll see if we can find a better solution for this.

[icecrime]

@rade Thanks for your input. In agree we requirement a), but I don't quite understand b): I believe the modification to that script wouldn't have any negative impact for users using a version older than 1.11.

[rade]

I believe the modification to that script wouldn't have any negative impact for users using a version older than 1.11.

That is correct.

The script is embedded in a docker image, which is part of what makes up a Weave Net release. Any user running a current or previously released version of Weave Net will encounter an error when they upgrade to Docker 1.11.

[icecrime]

@rade Oh sorry I see, you meant "upgrade (that is: repull) the image", not "upgrade to Docker 1.11". Yes then I agree 😉

[rade]

Yes. I have updated my comment to clarify.

[thaJeztah]

Looks like this is currently blocked on changes in the OCI specs; opencontainers/runtime-spec#320

[ibuildthecloud]

This impacts me too, although remount rw is probably not a big deal. But in general a read-only /proc/sys/net makes me sad as I really like flipping bits in there.

[thaJeztah]

[ibuildthecloud]

As I've been testing rc3 more, this is coming up more than I thought. I think it should be considered breaking compatibility.

[ibuildthecloud]

What's the reasoning behind this? Why would something in proc be ro if it's a privileged system. Also if I add NET_ADMIN on a non-privileged container, will I get a writeable /proc/sys/net?

[thaJeztah]

@ibuildthecloud given that the pull request was not yet accepted in opencontainers, we didn't want to delay RC3 to allow people testing other fixes, and plan to include the fix in the next RC

[ibuildthecloud]

@thaJeztah Thanks!

[ibuildthecloud]

@thaJeztah @crosbymichael Is there a PR somewhere I can pull to fix this? This is blocking me from more widely deploying 1.11 to test.

[thaJeztah]

PR's for the specs, and runC were merged; opencontainers/runtime-spec#364 opencontainers/runc#716. I think it needs to be bumped in the Dockerfiles in this repository to use those changes

[ibuildthecloud]

@thaJeztah As always, Thanks!

[anusha-ragunathan]

I'm running into another flavor of this issue. Its related to /proc/fs mounted ro. When running daemon with aufs on privileged containers, umount of the inner container (eg docker stop) results in auplink error (which is behaviourally only a warning) .

ERRO[0022] Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/c5a3d779e4d84707a38d0abb8cdc58a80c8ee31a1703832f754f00324c36bd4a:  exit status 30 

The error is because auplink attempts to write to /proc/fs/aufs/plink_maint, but fails because its not mounted rw. Remounting as rw fixes the issue.