Docker stop deadlocks intermittently

[kstaken]

There seems to be an issue in docker stop that causes it to intermittently deadlock and hang the execution context performing the stop. It seems to be a race condition as it's very sporadic.

I've tracked the issue down to a block on container.Wait() here: https://github.com/dotcloud/docker/blob/master/container.go#L886 The prior step is that docker tries to directly kill the process which doesn't error but also doesn't actually succeed in killing it either.

In this scenario the lxc container is already gone but the process is still running and docker thinks the container still exists. When it's in this state docker stop on the container just hangs because it waits to acquire the lock. Killing the process manually will cause docker to think the container exited and will free the lock. It requires a kill -9 to kill the process but I'm not sure why Process.Kill isn't able to do it directly.

Here's the log of a hung session (the last line is a debug I added right before L886 of container.go):

2013/07/25 21:41:10 POST /v1.3/containers/2b1213eb763b/stop?t=1
2013/07/25 21:41:10 lxc-kill: '2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd' is not running
lxc-kill: failed to get the init pid
2013/07/25 21:41:10 Failed to send SIGTERM to the process, force killing
2013/07/25 21:41:10 error killing container 2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd (lxc-kill: '2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd' is not running
lxc-kill: failed to get the init pid
, exit status 255)
2013/07/25 21:41:20 Container 2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd failed to exit within 10 seconds of lxc SIGKILL - trying direct SIGKILL
2013/07/25 21:41:20 Waiting on Container 2b1213eb763bb27440859a161c847352ead090e17748a49429813e01864ba5bd to exit

[vieux]

Hi,

Do you know any wy to reproduce this ? Or at least see it more often ?

[shykes]

Ping. Are you still seeing this @kstaken? Anything else you can give us to reproduce?

[pwaller]

I'm getting this on a Ubuntu 13.10 daily image. Seems to happen for every container which is docker started. Will happily respond with more information, but haven't looked into it further than the same log message that @kstaken got.

[pwaller]

To add another bit of information: lxc-info -n 667233ead56667ed06fa920f99d988b014f8865e775f4faf361e3d9549521a5 returns state: STOPPED even though I'm still connected to it and interacting with it.

[mahendra]

@pwaller @shykes @vieux I was looking at another issue. I feel these two are related. Do have a look at my observations at : #1906 (comment)

Also, I am seeing this on Ubuntu 13.10. Not tried it on older versions yet.

[durdn]

I have the same problem with stopping and attaching to a container running postgresql (sudo docker pull zaiste/postgresql).

OSX Mavericks running Vagrant -> Ubuntu 13.10 (saucy) -> running 'base' docker container.

[metakermit]

I am experiencing this issue in a Digital Ocean docker droplet. The first time I run sudo docker stop <container> it hangs. Then I have to Ctrl + C it and rerun the command to actually get the container to stop. Same behaviour when accessing docker from a remote API (docker-py) manifested through HTTP request time-outs. In the end I have to send the call two times by refreshing the page to actually stop the container.

[sundbp]

Running into this one as well it looks like, host is ubuntu 13.10 (running a custom 13.10 container).

[stuartpb]

Is this bug still present now that Docker uses straight libcontainer by default?

[crosbymichael]

I'm not sure, maybe someone who can reproduce can help.

[metakermit]

I just upgraded to docker 0.9 to give it a try, but unfortunately the
problem is still here. The client hangs on the first call to docker stop
(causing me connection timeouts on the API interface) and succeeds normally
on the second call.

On Thu, Mar 13, 2014 at 8:37 PM, Michael Crosby [email protected]:

I'm not sure, maybe someone who can reproduce can help.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1300#issuecomment-37577149
.

[crosbymichael]

@Kermit666 do you have a reliable way to reproduce?

[metakermit]

Well, it's happening on my server (a Digital Ocean Docker droplet). It's an
Ubuntu 13.04.

This is the image I am using (though I think the same thing happened with
other images as well):

https://index.docker.io/u/cloudfleet/simple-ldap/

I start it with:

sudo docker run -d -p 3001:3000 cloudfleet/simple-ldap

And stop it with

sudo docker stop <id>

On Fri, Mar 14, 2014 at 12:21 AM, Michael Crosby
[email protected]:

@Kermit666 https://github.com/kermit666 do you have a reliable way to
reproduce?

Reply to this email directly or view it on GitHubhttps://github.com//issues/1300#issuecomment-37599204
.

[crosbymichael]

@Kermit666 awesome, i'll take a look

[developit]

I'm getting this error too. I'm not sure if this is related, but I'm seeing this in my docker logs:

[error] mount.go:11 [warning]: couldn't run auplink before unmount: exec: "auplink": executable file not found in $PATH

... It looks like auplink flush is failing? (I have no idea what that is)
https://github.com/dotcloud/docker/blob/master/daemon/graphdriver/aufs/mount.go#L11

[unclejack]

@developit Are you sure that aufs-tools is installed? That error states quite clearly that auplink is missing.

[developit]

@unclejack I can't seem to get aufs-tools to install (also, there doesn't seem to be much help for that around the web and I'm not as knowledgeable about Linux as I would like to be). I'm running a 3.10 kernel, though.

[unclejack]

@developit What's your Linux distribution?

[developit]

@unclejack CentOS 6.

$ uname -a
Linux ... 3.10.5-3.el6.x86_64 #1 SMP Tue Aug 20 14:10:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[unclejack]

Multiple problems are being discussed on this issue and I'll try to address them all.

The original problem around docker stop deadlocking:
This shouldn't be a problem any more. The default native driver doesn't seem to be running into this problem at all. lxc might still have problems, but those are strictly lxc related.
Docker has also been improved a lot when it comes to stability and reliability. Many races, deadlocks and bugs have been fixed.

@developit's problem:
You have installed a kernel which seems to be AUFS enabled on CentOS 6. CentOS isn't a Linux distribution on which AUFS is in any way supported. The aufs-tools package isn't available there and using AUFS on that Linux distribution isn't recommended, nor supported.

That auplink problem isn't fixable without aufs-tools.

Please upgrade to CentOS 7 if you wish to use a recent kernel in a supported setup.

I'll close this issue now.

[mrdfuse]

When you say Docker has improved a lot, what version are you talking about? We are still regularly experiencing this issue on Docker 1.0. In combination with #5684 we lose our data everytime. I checked the changelogs for later versions but didn't see anything mentioned about stability and reliability?

[dmitry]

I'm experiencing the same issue with Docker version 1.3.2, build 39fa2fa

[unclejack]

@mrdfuse It's not only one thing, there have been a lot of changes which would fix that.

The kernel matters as well. Installing the updates provided by your distribution is important.

Would you mind providing the full output of docker info, docker version and uname -a?

@dmitry Could you provide the full output of docker info and docker version, please?

[mrdfuse]

docker info
Containers: 5
Images: 168
Storage Driver: devicemapper
Pool Name: docker-253:2-2465794-pool
Data file: /data/jenkins/docker/devicemapper/devicemapper/data
Metadata file: /data/jenkins/docker/devicemapper/devicemapper/metadata
Data Space Used: 6606.6 Mb
Data Space Total: 102400.0 Mb
Metadata Space Used: 10.2 Mb
Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 2.6.32-431.23.3.el6.x86_64

docker version:
Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.2
Git commit (client): 63fe64c/1.0.0
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.2
Git commit (server): 63fe64c/1.0.0

uname -a:
Linux sv-arg-bld-d1 2.6.32-431.23.3.el6.x86_64 #1 SMP Wed Jul 16 06:12:23 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)

It scares me a bit that dmitry has the same problem with the latest Docker version, and unfortunately upgrading our kernel is not an option (not in our control).

[unclejack]

@mrdfuse There are known issues with older kernels and devicemapper. Not keeping up with the latest kernel updates of your distribution (what you get when you install system updates) is how you can run into these problems. Since @dmitry hasn't provided the information regarding his environment, I'll stop discussing his problems any further.

2.6.32 is a kernel which is supported by Red Hat. That kernel needs to be kept up to date with the latest system updates.

The kernel you are running is one of the newest versions provided by RHEL 6.5. Red Hat has released RHEL 6.6 and the kernel version I'm seeing on a CentOS 6.6 system is 2.6.32-504.1.3.el6. I've also just received updates for devicemapper related packages. Updates for systems running RHEL/RHEL derivatives aren't optional when it comes to running containers on these systems, they're vital because they provide vital bug fixes from newer kernels and packages.

@vbatts Can you also look into this, provide some advice or both, please?

[dmitry]

@unclejack sorry, haven't time to provide such information.

Containers: 19
Images: 39
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 77
Execution Driver: native-0.2
Kernel Version: 3.8.0-30-generic
Operating System: Ubuntu precise (12.04.5 LTS)
WARNING: No swap limit support

Client version: 1.3.0
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): c78088f
OS/Arch (client): linux/amd64
Server version: 1.3.0
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): c78088f

Linux host.com 3.8.0-30-generic #44~precise1-Ubuntu SMP Fri Aug 23 18:32:41 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[mrdfuse]

Ok thanks for the explanation. We outsource our infrastructure and that company wants to keep its system in a stable condition, ie not update constantly (only security patches). Since I can't guarantee this issue won't occur again even if they would upgrade (again, I see @dmitry's data, he seems uptodate), I'm gonna stop using Docker for the more critical applications (with regret).

[unclejack]

@dmitry The kernel you are running is no longer supported by Canonical on any of their distributions.
There's no other Linux distribution which supports kernel 3.8. If you take a look at kernel.org, 3.8 s not maintained any more.
A kernel which is currently supported by Canonical can be installed using apt-get update && apt-get install linux-image-generic-lts-trusty on Ubuntu 12.04.

Bugs have been reported on this issue tracker which affect kernel 3.8 :

• various btrfs corruption bugs
• aufs problems: stale nfs file handle, mount problems, crashes (these have been reported a few times on various revisions of the 3.8-xy kernel packages)
• deadlocks and stuck processes
• kernel panics
• potential data loss and corruption

Many other smaller and less visible bugs have been fixed in newer kernels, and many of them also affect 3.8

WARNING: Kernel 3.8 has bugs and it's not supported in any way on any distribution since August, 2014. Kernel 3.8 isn't updated to fix severe bugs which can cause data loss and it's also not updated to fix security vulnerabilities. Canonical stopped supporting this kernel in favor of a newer and better kernel. Updating to that linux-image-generic-lts-trusty kernel is recommended.

You can find out more about this here: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

As you can see in the image above, the support for kernel 3.8 ended officially in August 2014.

Even for a 3.8 kernel, you have an old kernel. I recall seeing 3.8.0-3x on one of my systems before the upgrade to 3.13.

Please upgrade to kernel 3.13 (linux-image-generic-lts-trusty) to get fixes and security patches.

No amount of Docker updates will fix the kernel bugs on kernel 3.8.

@mrdfuse If you don't keep your system up to date with updates from Red Hat, there isn't much to do. Red Hat is updating kernel 2.6.32 to make sure it's stable and that the system they're offering to their customers is stable.
You've asked for help with the problems you're seeing. Please try to reproduce these issues on a fully updated system and comment.

Keeping systems fully patched and updated is necessary.

[dmitry]

@unclejack thank you for such a great explanation! I will defenetelly upgrade the server.

[mrdfuse]

I understand completely, it's just not under my control

[ashahab-altiscale]

@mrdfuse
I also cannot reproduce this in your kernel(using centos 6)

12:10:15-ashahab~$ sudo docker run -itd ubuntu sleep 600
[sudo] password for ashahab: 
6750311e81a36017e4551986d4f6af87e4aa94c95f6a519b09c8436f746d935c
12:10:20-ashahab~$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
6750311e81a3        ubuntu:14.04        sleep 600           8 seconds ago       Up 8 seconds                            stoic_turing        
12:10:28-ashahab~$ sudo ps -eaf | grep lxc
root      1815     1  0 Dec13 ?        00:06:13 /usr/bin/docker -d -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock -e lxc
root     63294  1815  0 12:10 pts/2    00:00:00 lxc-start -n 6750311e81a36017e4551986d4f6af87e4aa94c95f6a519b09c8436f746d935c -f /var/lib/docker/containers/6750311e81a36017e4551986d4f6af87e4aa94c95f6a519b09c8436f746d935c/config.lxc -- /.dockerinit -driver lxc -g 172.17.42.1 -i 172.17.0.15/16 -mtu 1500 -- sleep 600
ashahab  63378 53843  0 12:10 pts/0    00:00:00 grep lxc
12:10:39-ashahab~$ sudo lxc-info -n 6750311e81a36017e4551986d4f6af87e4aa94c95f6a519b09c8436f746d935c
state:   RUNNING
pid:     63312
12:11:05-ashahab~$ sudo docker stop 6750311e81a36017e4551986d4f6af87e4aa94c95f6a519b09c8436f746d935c
6750311e81a36017e4551986d4f6af87e4aa94c95f6a519b09c8436f746d935c
12:11:22-ashahab~$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
12:11:37-ashahab~$ uname -a
Linux localhost.localdomain 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
12:11:46-ashahab~$ sudo docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.2
Git commit (client): d84a070/1.1.2
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.2
Git commit (server): d84a070/1.1.2

[mrdfuse]

It only happened a few times when running about 6 containers for a few months non-stop. I don't it will be easy to reproduce.

[mulle-nat]

I have something happening like this just now on ubuntu. Ubuntu itself is running in a kvm. I dockered a lot of stuff already over the last month and never had this problem so far:

# docker info
Containers: 10
Images: 99
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 119
Execution Driver: native-0.2
Kernel Version: 3.13.0-43-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 2
Total Memory: 1.955 GiB

# docker ps
CONTAINER ID        IMAGE                             COMMAND                CREATED             STATUS              PORTS                                      NAMES
f32bedd01e50        mulle/firefox-syncserver:latest   "/usr/bin/make serve   14 hours ago        Up 14 hours         0.0.0.0:5000->5000/tcp                     firefox-syncserver   

but docker stop firefox-syncserver just hangs there since now 15 minutes. There is nothing interesting in /var/log/syslog.

...

So now I CTRL-Ced it and tried to do it again and this is what happened:

^Croot@muhbuntu:~/docker# docker stop firefox-syncserver
Error response from daemon: Cannot stop container firefox-syncserver: no such process
FATA[0000] Error: failed to stop one or more containers 
root@muhbuntu:~/docker# docker ps
CONTAINER ID        IMAGE                             COMMAND                CREATED             STATUS              PORTS                                      NAMES
f32bedd01e50        mulle/firefox-syncserver:latest   "/usr/bin/make serve   14 hours ago        Up 14 hours         0.0.0.0:5000->5000/tcp                     firefox-syncserver   

Definetly inconsistent. I did service docker restart to fix this.

[wznoinski]

hi,
I'm having the same problem on Docker version 1.5.0, build a8a31ef/1.5.0 on Fedora 21 3.17.7-300.fc21.x86_64 #1 SMP Wed Dec 17 03:08:44 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux.

I'm not yet sure whether that's lxc's or docker problem. Please check artifacts below.

[root@silpixa00385138 docker]# docker info
Containers: 47
Images: 5
Storage Driver: devicemapper
 Pool Name: docker-253:2-7602336-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 36.73 GB
 Data Space Total: 150.3 GB
 Metadata Space Used: 35.98 MB
 Metadata Space Total: 5.369 GB
 Udev Sync Supported: true
 Data loop file: /home/docker/devicemapper/devicemapper/data
 Metadata loop file: /home/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93 (2015-01-30)
Execution Driver: lxc-1.0.7
Kernel Version: 3.17.7-300.fc21.x86_64
Operating System: Fedora 21 (Twenty One)
CPUs: 40
Total Memory: 188.8 GiB


[root@silpixa00385138 docker]# strace docker stop ec6d53bb7666

futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
mmap(0xc208100000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc208100000
mmap(0xc207fe0000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc207fe0000
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0x1166bb0, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
futex(0xc2080bb2f0, FUTEX_WAKE, 1)      = 1
getuid()                                = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1722, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f94acf71000
read(3, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1722
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f94acf71000, 4096)            = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=46750, ...}) = 0
mmap(NULL, 46750, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f94acf66000
close(3)                                = 0
open("/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\"\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=62072, ...}) = 0
mmap(NULL, 2148456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f94a87b2000
mprotect(0x7f94a87be000, 2093056, PROT_NONE) = 0
mmap(0x7f94a89bd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb000) = 0x7f94a89bd000
close(3)                                = 0
mprotect(0x7f94a89bd000, 4096, PROT_READ) = 0
munmap(0x7f94acf66000, 46750)           = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2416, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f94acf71000
read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 2416
close(3)                                = 0
munmap(0x7f94acf71000, 4096)            = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, 23) = 0
epoll_create1(EPOLL_CLOEXEC)            = 4
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2901533608, u64=140276533420968}}) = 0
getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0
getpeername(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, [23]) = 0
read(3, 0xc208092000, 4096)             = -1 EAGAIN (Resource temporarily unavailable)
write(3, "POST /v1.17/containers/ec6d53bb7"..., 164) = 164
epoll_wait(4, {{EPOLLOUT, {u32=2901533608, u64=140276533420968}}}, 128, 0) = 1
epoll_wait(4, 

[root@silpixa00385138 docker]# ls -l /var/run/docker.sock
srw-rw----. 1 root docker 0 Feb 25 15:36 /var/run/docker.sock

1. When I hit that deadlock (happens only during docker stop/lxc-stop by the looks of things), other docker commands, i.e.: docker ps, don't work. docker info works ok.

[root@silpixa00385138 docker]# strace docker ps

setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, 23) = 0
epoll_create1(EPOLL_CLOEXEC)            = 4
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1395209120, u64=140202012652448}}) = 0
getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0
getpeername(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, [23]) = 0
read(3, 0xc208092000, 4096)             = -1 EAGAIN (Resource temporarily unavailable)
write(3, "GET /v1.17/containers/json HTTP/"..., 100) = 100
epoll_wait(4, {{EPOLLOUT, {u32=1395209120, u64=140202012652448}}}, 128, 0) = 1
epoll_wait(4, 

2. kill $PID (of the lxc-stop command) kills the process, docker ps returns the results, then lxc-stop command is back in process list (with a different PID) and stalls and blocks other (i.e. docker ps) commands again. Killing that second occurrence of the lxc-stop causes docker ps to work again, lxc-stop is not coming back for that container anymore. The container in question is sometimes stopped sometimes not. UPDATE: killing (with SIGTERM) makes the lxc-stop to come back with different PID every time. killing with SIGKILL makes it go away for good.

[MrMMorris]

Hitting this on Linux ip-10-0-1-144 3.19.0-18-generic #18~14.04.1-Ubuntu SMP Wed May 20 09:38:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Docker version 1.6.2, build 7c8fca2

sudo service docker restart does not fix it for me (or at least didn't seem to the first time)

docker info

Containers: 6
Images: 66
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 78
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.19.0-18-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 14.94 GiB
Name: ip-10-0-1-144
ID: 47EM:TJ4Q:GQUD:32ZF:7G7B:OIFY:B6GO:7W75:TNTX:IGPC:5BV7:O7AU
WARNING: No swap limit support

strace

$ strace docker stop mongo
execve("/usr/bin/docker", ["docker", "stop", "mongo"], [/* 21 vars */]) = 0
uname({sys="Linux", node="ip-10-0-1-144", ...}) = 0
brk(0)                                  = 0x1dc7000
brk(0x1dc81c0)                          = 0x1dc81c0
arch_prctl(ARCH_SET_FS, 0x1dc7880)      = 0
set_tid_address(0x1dc7b50)              = 359
set_robust_list(0x1dc7b60, 24)          = 0
futex(0x7ffd294eb440, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 1dc7880) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x9af430, [], SA_RESTORER|SA_SIGINFO, 0x9afaf0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x9af4c0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x9afaf0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
readlink("/proc/self/exe", "/usr/bin/docker", 4096) = 15
brk(0x1de91c0)                          = 0x1de91c0
brk(0x1dea000)                          = 0x1dea000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
sched_getaffinity(0, 128, {3, 0, 0, 0}) = 32
mmap(0xc000000000, 65536, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000000000
munmap(0xc000000000, 65536)             = 0
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f809d805000
mmap(0xc208000000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc208000000
mmap(0xc207ff0000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc207ff0000
mmap(0xc000000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000000000
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f809d7f5000
clock_gettime(CLOCK_MONOTONIC, {180929, 343344820}) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 343447993}) = 0
mmap(NULL, 1439992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f809d695000
sigaltstack({ss_sp=0xc208002000, ss_flags=0, ss_size=32768}, NULL) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGHUP, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGHUP, {0x44cc80, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x44ccf0}, NULL, 8) = 0
rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x44cc80, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x44ccf0}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {0x44cc80, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x44ccf0}, NULL, 8) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 351049853}) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 351166215}) = 0
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f809ce94000
mprotect(0x7f809ce94000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f809d693e70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f809d6949d0, tls=0x7f809d694700, child_tidptr=0x7f809d6949d0) = 360
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 351794740}) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 351941639}) = 0
mmap(0xc208100000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc208100000
mmap(0xc207fe0000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc207fe0000
clock_gettime(CLOCK_MONOTONIC, {180929, 371673394}) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 371752683}) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 371861342}) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 371931313}) = 0
clock_gettime(CLOCK_REALTIME, {1446059353, 31316199}) = 0
futex(0x134e480, FUTEX_WAKE, 1)         = 1
clock_gettime(CLOCK_MONOTONIC, {180929, 372263461}) = 0
futex(0x134e400, FUTEX_WAKE, 1)         = 1
clock_gettime(CLOCK_MONOTONIC, {180929, 372419929}) = 0
capget({0 /* _LINUX_CAPABILITY_VERSION_??? */, 194}, NULL) = 0
open("/proc/sys/kernel/cap_last_cap", O_RDONLY|O_CLOEXEC) = 3
read(3, "37\n", 11)                     = 3
clock_gettime(CLOCK_MONOTONIC, {180929, 372877646}) = 0
close(3)                                = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 373113031}) = 0
futex(0xc208020ed8, FUTEX_WAKE, 1)      = 1
futex(0xc208020ed8, FUTEX_WAKE, 1)      = 1
futex(0xc208020ed8, FUTEX_WAKE, 1)      = 1
futex(0x134f1b8, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_MONOTONIC, {180929, 403190606}) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 403343132}) = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
clock_gettime(CLOCK_REALTIME, {1446059353, 63115358}) = 0
clock_gettime(CLOCK_REALTIME, {1446059353, 63216926}) = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, 23) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 404413608}) = 0
clock_gettime(CLOCK_REALTIME, {1446059353, 63842622}) = 0
epoll_create1(EPOLL_CLOEXEC)            = 4
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2642513848, u64=140190375055288}}) = 0
getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0
getpeername(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, [23]) = 0
clock_gettime(CLOCK_MONOTONIC, {180929, 404974866}) = 0
read(3, 0xc20800f000, 4096)             = -1 EAGAIN (Resource temporarily unavailable)
write(3, "POST /v1.18/containers/mongo/sto"..., 157) = 157
epoll_wait(4, {{EPOLLOUT, {u32=2642513848, u64=140190375055288}}}, 128, 0) = 1
epoll_wait(4,

docker.log:

INFO[0484] POST /v1.18/containers/mongo/stop?t=10
INFO[0484] +job stop(mongo)
Container already stopped
INFO[0484] -job stop(mongo) = ERR (1)
INFO[0485] GET /containers/json
INFO[0485] +job containers()
INFO[0485] -job containers() = OK (0)
INFO[0485] GET /containers/f0364e4e881f7696f94fa9dd918bf86fba12b2dce1ad68dd605/json
INFO[0485] +job container_inspect(f0364e4e881f7696f94fa973d5ada61e0dd918bf86fba12b2dce1ad68dd605)
INFO[0485] -job container_inspect(f0364e4e881f763d5ada61e0dd918bf86fba12b2dce1ad68dd605) = OK (0)
INFO[0485] GET /containers/928e761c0528e728619065c89451a69b089eec46f6e54d9056447b0/json
INFO[0485] +job container_inspect(928e761c0528e7286197c681389451a69b089eec46f6e54d9056447b0)
INFO[0485] -job container_inspect(928e761c0528e728619451a69b089eec46f6e54d9056447b0) = OK (0)
INFO[0485] GET /containers/3a1ab439bec525afd282f03584a74db2c4c4b672bbc40d5ab19a8cb3/json
INFO[0485] +job container_inspect(3a1ab439bec52fb7ba06cd03584a74db2c4c4b672bbc40d5ab19a8cb3)
INFO[0485] -job container_inspect(3a1ab439bec506cd03584a74db2c4c4b672bbc40d5ab19a8cb3) = OK (0)
INFO[0485] GET /containers/5fbb24dac754ccff7c64e0b71cf81903129e58903714a9c5d96931/json
INFO[0485] +job container_inspect(5fbb24dac754ccff7e0b71cf81903129e58903714a9c5d96931)
INFO[0485] -job container_inspect(5fbb24dac754ccff7c64860858903714a9c5d96931) = OK (0)

[MrMMorris]

Alright, that container seems to be gone now for whatever reason. I can stop other containers fine. Will keep an eye out for it again.

EDIT: after being able to stop/rm and recreate that container on the same host, I am getting deadlocks again

logs are the same:

INFO[1870] POST /v1.18/containers/mongo/stop?t=10
INFO[1870] +job stop(mongo)
Container already stopped
INFO[1870] -job stop(mongo) = ERR (1)

Aaaaaand it's gone again

SIGKILL on the docker process doesn't fix anything

EDIT2: restart of host allowed me to stop/rm the container, but yet AGAIN if I recreate it, I cannot stop it. This is very bad

[MrMMorris]

so I tried reinstalling the docker package as suggested in #10684 (comment)

And even that didn't work! I have now concluded the only way to fix this is to destroy and recreate my host which is very disturbing.

Can this please be reopened? @jfrazelle @unclejack @crosbymichael ?

I will keep this host up for debugging

[jasonahooper]

I have the same issue on:
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty
with:
Client:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64

Has this been re-opened?

Regards,

[michaelb990]

I'm seeing the same thing for one of our docker hosts. Any guidance? We're using:

Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:    14.04
Codename:   trusty

$ docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:37:18 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:37:18 UTC 2015
 OS/Arch:      linux/amd64

[magicphip]

I am experiencing the same problem, with strace output similar to @MrMMorris

connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, 23) = 0
getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0
getpeername(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, [23]) = 0
read(3, 0xc2100f5000, 4096)             = -1 EAGAIN (Resource temporarily unavailable)
write(3, "POST /v1.18/containers/XXXX"..., 165) = 165
epoll_wait(4, {{EPOLLOUT, {u32=3810742824, u64=140234492957224}}}, 128, 0) = 1
epoll_wait(4,  
^CProcess 4880 detached
 <detached ...>

The ps command shows interesting things (in my case that was a recursive rm that was running when the ctrl-c has been done):

$ ps -ef | grep rm
philipp+  5054  3383  0 16:47 pts/12   00:00:00 grep rm
root     10864 29668 99 16:24 ?        00:23:08 rm -rf /opt/(...)
$ ps -ef | grep 29668
root     29668  1516  0 16:23 ?        00:00:00 [bash]

We can see that the rm's parent process bash is zombified, so the signals sent to him get lost in nature.
The only way to use again that container is to reboot the PC.
There is no need to re-install anything.

[AJNOURI]

Same here cannot stop a centos container:

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4cb482d2c1f9 centos:latest "/bin/sh" 19 minutes ago Up 16 minutes serene_ptolemy

$ docker stop 4cb482d2c1f9

<... hangs...>

strace docker stop centos

execve("/usr/bin/docker", ["docker", "stop", "centos"], [/* 74 vars */]) = 0
brk(0) = 0x32b4000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa629be1000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=177871, ...}) = 0
mmap(NULL, 177871, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fa629bb5000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
...
connect(4, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, 23) = 0
epoll_create1(EPOLL_CLOEXEC) = 5
epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=649757624, u64=140351591056312}}) = 0
getsockname(4, {sa_family=AF_LOCAL, NULL}, [2]) = 0
getpeername(4, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, [23]) = 0
futex(0xc8201cae90, FUTEX_WAKE, 1) = 1
read(4, 0xc8201cc000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
write(4, "POST /v1.22/containers/centos/st"..., 147) = 147
epoll_wait(5, {}, 128, 0) = 0
epoll_wait(5, Failed to stop container (centos): Error response from daemon: No such container: centos
<unfinished ...>
+++ exited with 1 +++

The complete file herehttps://drive.google.com/open?id=0BzYXUSsUVFR_RGlWLVZBY2ZDWjQ

Docker host

docker version
Client:
Version: 1.10.2
API version: 1.22
Go version: go1.5.3
Git commit: c3959b1
Built: Mon Feb 22 21:37:01 2016
OS/Arch: linux/amd64

Server:
Version: 1.10.2
API version: 1.22
Go version: go1.5.3
Git commit: c3959b1
Built: Mon Feb 22 21:37:01 2016
OS/Arch: linux/amd64

Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty

Linux gns3-iouvm 3.13.0-75-generic #119-Ubuntu SMP Mon Jan 11 13:49:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[thaJeztah]

That strace looks to be incorrect; strace docker stop centos, because there's no such container?

[sleaze]

@thaJeztah How about this strace for my hanged container:

ubuntu@jenkins2a:~$ sudo strace docker stop efbd748834815227493d326e6308825e9f796127fa597f11bdb610f742171d47
execve("/usr/bin/docker", ["docker", "stop", "efbd748834815227493d326e6308825e"...], [/* 15 vars */]) = 0
brk(0)                                  = 0x2474000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0aa01000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=36274, ...}) = 0
mmap(NULL, 36274, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f6f0a9f8000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libsystemd-journal.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`1\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=108632, ...}) = 0
mmap(NULL, 2205808, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f0a5c6000
mprotect(0x7f6f0a5df000, 2097152, PROT_NONE) = 0
mmap(0x7f6f0a7df000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7f6f0a7df000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libapparmor.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\30\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=47984, ...}) = 0
mmap(NULL, 2143152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f0a3ba000
mprotect(0x7f6f0a3c5000, 2093056, PROT_NONE) = 0
mmap(0x7f6f0a5c4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7f6f0a5c4000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0po\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=141574, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9f7000
mmap(NULL, 2217264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f0a19c000
mprotect(0x7f6f0a1b5000, 2093056, PROT_NONE) = 0
mmap(0x7f6f0a3b4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f6f0a3b4000
mmap(0x7f6f0a3b6000, 13616, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a3b6000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdevmapper.so.1.02.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260Z\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=228992, ...}) = 0
mmap(NULL, 2328416, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f09f63000
mprotect(0x7f6f09f98000, 2093056, PROT_NONE) = 0
mmap(0x7f6f0a197000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x34000) = 0x7f6f0a197000
mmap(0x7f6f0a19b000, 1888, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a19b000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\37\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1840928, ...}) = 0
mmap(NULL, 3949248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f09b9e000
mprotect(0x7f6f09d59000, 2093056, PROT_NONE) = 0
mmap(0x7f6f09f58000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ba000) = 0x7f6f09f58000
mmap(0x7f6f09f5e000, 17088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6f09f5e000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0[\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=134296, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9f6000
mmap(NULL, 2238192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f0997b000
mprotect(0x7f6f0999b000, 2093056, PROT_NONE) = 0
mmap(0x7f6f09b9a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1f000) = 0x7f6f09b9a000
mmap(0x7f6f09b9c000, 5872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6f09b9c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcgmanager.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0206\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=108480, ...}) = 0
mmap(NULL, 2203688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f09760000
mprotect(0x7f6f09779000, 2093056, PROT_NONE) = 0
mmap(0x7f6f09978000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f6f09978000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnih.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`G\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=96280, ...}) = 0
mmap(NULL, 2191776, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f09548000
mprotect(0x7f6f0955f000, 2093056, PROT_NONE) = 0
mmap(0x7f6f0975e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f6f0975e000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libnih-dbus.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340,\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=38920, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9f5000
mmap(NULL, 2134040, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f0933e000
mprotect(0x7f6f09346000, 2097152, PROT_NONE) = 0
mmap(0x7f6f09546000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f6f09546000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdbus-1.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@h\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=281552, ...}) = 0
mmap(NULL, 2377408, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f090f9000
mprotect(0x7f6f0913d000, 2093056, PROT_NONE) = 0
mmap(0x7f6f0933c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x43000) = 0x7f6f0933c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320 \0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=137400, ...}) = 0
mmap(NULL, 2232456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f08ed7000
mprotect(0x7f6f08ef8000, 2093056, PROT_NONE) = 0
mmap(0x7f6f090f7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x20000) = 0x7f6f090f7000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libgcrypt.so.11", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\177\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=524704, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9f4000
mmap(NULL, 2621056, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f08c57000
mprotect(0x7f6f08cd3000, 2097152, PROT_NONE) = 0
mmap(0x7f6f08ed3000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7c000) = 0x7f6f08ed3000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P#\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=31792, ...}) = 0
mmap(NULL, 2129016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f08a4f000
mprotect(0x7f6f08a56000, 2093056, PROT_NONE) = 0
mmap(0x7f6f08c55000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f6f08c55000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libudev.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0200\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=67600, ...}) = 0
mmap(NULL, 2164816, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f0883e000
mprotect(0x7f6f0884e000, 2093056, PROT_NONE) = 0
mmap(0x7f6f08a4d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f6f08a4d000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9f3000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\27\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=252032, ...}) = 0
mmap(NULL, 2347200, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f08600000
mprotect(0x7f6f0863d000, 2093056, PROT_NONE) = 0
mmap(0x7f6f0883c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3c000) = 0x7f6f0883c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14664, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f083fc000
mprotect(0x7f6f083ff000, 2093056, PROT_NONE) = 0
mmap(0x7f6f085fe000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f6f085fe000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\t\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18416, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9f2000
mmap(NULL, 2113648, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6f081f7000
mprotect(0x7f6f081fb000, 2093056, PROT_NONE) = 0
mmap(0x7f6f083fa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f6f083fa000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9f1000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9f0000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9ee000
arch_prctl(ARCH_SET_FS, 0x7f6f0a9ee880) = 0
mprotect(0x7f6f09f58000, 16384, PROT_READ) = 0
mprotect(0x7f6f083fa000, 4096, PROT_READ) = 0
mprotect(0x7f6f085fe000, 4096, PROT_READ) = 0
mprotect(0x7f6f0883c000, 4096, PROT_READ) = 0
mprotect(0x7f6f0a3b4000, 4096, PROT_READ) = 0
mprotect(0x7f6f08c55000, 4096, PROT_READ) = 0
mprotect(0x7f6f0975e000, 4096, PROT_READ) = 0
mprotect(0x7f6f0933c000, 4096, PROT_READ) = 0
mprotect(0x7f6f09546000, 4096, PROT_READ) = 0
mprotect(0x7f6f09978000, 8192, PROT_READ) = 0
mprotect(0x7f6f08a4d000, 4096, PROT_READ) = 0
mprotect(0x7f6f08ed3000, 4096, PROT_READ) = 0
mprotect(0x7f6f090f7000, 4096, PROT_READ) = 0
mprotect(0x7f6f09b9a000, 4096, PROT_READ) = 0
mprotect(0x7f6f0a197000, 4096, PROT_READ) = 0
mprotect(0x7f6f0a5c4000, 4096, PROT_READ) = 0
mprotect(0x7f6f0a7df000, 4096, PROT_READ) = 0
mprotect(0x2045000, 8192, PROT_READ)    = 0
mprotect(0x7f6f0aa03000, 4096, PROT_READ) = 0
munmap(0x7f6f0a9f8000, 36274)           = 0
set_tid_address(0x7f6f0a9eeb50)         = 4773
set_robust_list(0x7f6f0a9eeb60, 24)     = 0
futex(0x7ffd38c4b750, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7f6f0a9ee880) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x7f6f0a1a29f0, [], SA_RESTORER|SA_SIGINFO, 0x7f6f0a1ac340}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7f6f0a1a2a80, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f6f0a1ac340}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(0)                                  = 0x2474000
brk(0x2495000)                          = 0x2495000
statfs("/sys/fs/selinux", 0x7ffd38c4b720) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7ffd38c4b720)      = -1 ENOENT (No such file or directory)
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0aa00000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 1024) = 322
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f6f0aa00000, 4096)            = 0
sched_getaffinity(0, 8192, {f, 0, 0, 0}) = 32
mmap(0xc000000000, 65536, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000000000
munmap(0xc000000000, 65536)             = 0
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a9ae000
mmap(0xc820000000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc820000000
mmap(0xc81fff8000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc81fff8000
mmap(0xc000000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000000000
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a99e000
clock_gettime(CLOCK_MONOTONIC, {2437, 165041772}) = 0
mmap(NULL, 1439992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f0a83e000
clock_gettime(CLOCK_MONOTONIC, {2437, 165550216}) = 0
clock_gettime(CLOCK_MONOTONIC, {2437, 165760578}) = 0
sigaltstack({ss_sp=0xc820002000, ss_flags=0, ss_size=32672}, NULL) = 0
gettid()                                = 4773
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGHUP, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGHUP, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGHUP, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGILL, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGILL, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGTRAP, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGTRAP, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGABRT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGABRT, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGBUS, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGBUS, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGFPE, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGFPE, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGUSR1, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGUSR1, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGSEGV, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGSEGV, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGUSR2, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGUSR2, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGPIPE, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGPIPE, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGALRM, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGALRM, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGTERM, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGSTKFLT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGSTKFLT, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGCHLD, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGURG, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGURG, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGXCPU, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGXCPU, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGXFSZ, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGXFSZ, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGVTALRM, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGVTALRM, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGPROF, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGPROF, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGWINCH, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGWINCH, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGIO, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGIO, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGPWR, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGPWR, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGSYS, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGSYS, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRTMIN, NULL, {0x7f6f0a1a29f0, [], SA_RESTORER|SA_SIGINFO, 0x7f6f0a1ac340}, 8) = 0
rt_sigaction(SIGRTMIN, NULL, {0x7f6f0a1a29f0, [], SA_RESTORER|SA_SIGINFO, 0x7f6f0a1ac340}, 8) = 0
rt_sigaction(SIGRTMIN, {0x7f6f0a1a29f0, [], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x7f6f0a1ac340}, NULL, 8) = 0
rt_sigaction(SIGRT_1, NULL, {0x7f6f0a1a2a80, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f6f0a1ac340}, 8) = 0
rt_sigaction(SIGRT_1, NULL, {0x7f6f0a1a2a80, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f6f0a1ac340}, 8) = 0
rt_sigaction(SIGRT_1, {0x7f6f0a1a2a80, [], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x7f6f0a1ac340}, NULL, 8) = 0
rt_sigaction(SIGRT_2, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_2, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_3, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_3, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_4, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_4, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_5, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_5, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_6, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_6, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_7, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_7, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_8, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_8, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_9, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_9, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_10, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_10, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_11, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_11, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_12, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_12, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_13, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_13, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_14, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_14, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_15, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_15, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_16, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_16, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_17, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_17, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_18, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_18, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_19, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_19, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_20, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_20, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_21, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_21, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_22, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_22, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_23, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_23, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_24, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_24, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_25, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_25, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_26, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_26, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_27, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_27, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_28, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_28, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_29, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_29, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_30, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_30, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_31, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_31, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
rt_sigaction(SIGRT_32, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGRT_32, {0x474270, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x474290}, NULL, 8) = 0
clock_gettime(CLOCK_MONOTONIC, {2437, 192236290}) = 0
rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f6f079f6000
mprotect(0x7f6f079f6000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f6f081f5e70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f6f081f69d0, tls=0x7f6f081f6700, child_tidptr=0x7f6f081f69d0) = 4774
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f6f071f5000
mprotect(0x7f6f071f5000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f6f079f4e70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f6f079f59d0, tls=0x7f6f079f5700, child_tidptr=0x7f6f079f59d0) = 4775
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f069b4000
clock_gettime(CLOCK_MONOTONIC, {2437, 195378309}) = 0
clock_gettime(CLOCK_MONOTONIC, {2437, 195664379}) = 0
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
openat(AT_FDCWD, "/proc/sys/net/core/somaxconn", O_RDONLY|O_CLOEXEC) = 3
read(3, "128\n", 4096)                  = 4
read(3, "", 4092)                       = 0
close(3)                                = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
close(3)                                = 0
socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
setsockopt(3, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
bind(3, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
bind(4, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
close(4)                                = 0
close(3)                                = 0
clock_gettime(CLOCK_REALTIME, {1457980615, 60710107}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
stat("/usr/local/sbin/adduser", 0xc820092120) = -1 ENOENT (No such file or directory)
stat("/usr/local/bin/adduser", 0xc8200921b0) = -1 ENOENT (No such file or directory)
stat("/usr/sbin/adduser", {st_mode=S_IFREG|0755, st_size=35125, ...}) = 0
lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/sbin", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/sbin/adduser", {st_mode=S_IFREG|0755, st_size=35125, ...}) = 0
stat("/usr/local/sbin/addgroup", 0xc820092480) = -1 ENOENT (No such file or directory)
stat("/usr/local/bin/addgroup", 0xc820092510) = -1 ENOENT (No such file or directory)
stat("/usr/sbin/addgroup", {st_mode=S_IFREG|0755, st_size=35125, ...}) = 0
lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/sbin", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/sbin/addgroup", {st_mode=S_IFLNK|0777, st_size=7, ...}) = 0
readlinkat(AT_FDCWD, "/usr/sbin/addgroup", "adduser", 128) = 7
lstat("/usr/sbin/adduser", {st_mode=S_IFREG|0755, st_size=35125, ...}) = 0
stat("/usr/local/sbin/groupadd", 0xc820092870) = -1 ENOENT (No such file or directory)
stat("/usr/local/bin/groupadd", 0xc820092900) = -1 ENOENT (No such file or directory)
stat("/usr/sbin/groupadd", {st_mode=S_IFREG|0755, st_size=55576, ...}) = 0
lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/sbin", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/sbin/groupadd", {st_mode=S_IFREG|0755, st_size=55576, ...}) = 0
syscall_318(0xc82005fabf, 0x1, 0x1, 0, 0, 0) = 0x1
syscall_318(0xc82000f5d0, 0x8, 0, 0, 0, 0) = 0x8
mmap(0xc820100000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc820100000
mmap(0xc81fff0000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc81fff0000
clock_gettime(CLOCK_MONOTONIC, {2437, 206577777}) = 0
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f06173000
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f06133000
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
futex(0xc820030e90, FUTEX_WAKE, 1)      = 1
clock_gettime(CLOCK_MONOTONIC, {2437, 213878984}) = 0
clock_gettime(CLOCK_MONOTONIC, {2437, 214406705}) = 0
mmap(0xc820200000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc820200000
mmap(0xc81ffe8000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc81ffe8000
clock_gettime(CLOCK_MONOTONIC, {2437, 215149257}) = 0
clock_gettime(CLOCK_MONOTONIC, {2437, 215652953}) = 0
mmap(0xc820300000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc820300000
mmap(0xc81ffe0000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc81ffe0000
clock_gettime(CLOCK_MONOTONIC, {2437, 217409880}) = 0
capget({0 /* _LINUX_CAPABILITY_VERSION_??? */, 0}, NULL) = 0
openat(AT_FDCWD, "/proc/sys/kernel/cap_last_cap", O_RDONLY|O_CLOEXEC) = 3
read(3, "37\n", 11)                     = 3
close(3)                                = 0
openat(AT_FDCWD, "/sys/kernel/mm/hugepages", O_RDONLY|O_CLOEXEC) = 3
getdents64(3, /* 3 entries */, 4096)    = 88
getdents64(3, /* 0 entries */, 4096)    = 0
lstat("/sys/kernel/mm/hugepages/hugepages-2048kB", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
close(3)                                = 0
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0x20844d0, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0x20844d0, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0x20844d0, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0x20844d0, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0x20844d0, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820030a10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
clock_gettime(CLOCK_REALTIME, {1457980615, 105639840}) = 0
mmap(0xc820400000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc820400000
mmap(0xc81ffd8000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc81ffd8000
mmap(0xc000001000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000001000
clock_gettime(CLOCK_MONOTONIC, {2437, 246665274}) = 0
mmap(0xc820500000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc820500000
mmap(0xc81ffd0000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc81ffd0000
clock_gettime(CLOCK_MONOTONIC, {2437, 247912639}) = 0
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0x20844d0, FUTEX_WAIT, 0, NULL)   = 0
futex(0xc820382e90, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820382e90, FUTEX_WAKE, 1)      = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0x20844d0, FUTEX_WAIT, 0, NULL)   = 0
clock_gettime(CLOCK_MONOTONIC, {2437, 252880589}) = 0
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
clock_gettime(CLOCK_MONOTONIC, {2437, 253415015}) = 0
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
sched_yield()                           = 0
futex(0x208dde0, FUTEX_WAKE, 1)         = 0
sched_yield()                           = 0
clock_gettime(CLOCK_MONOTONIC, {2437, 253823447}) = 0
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
clock_gettime(CLOCK_MONOTONIC, {2437, 254374913}) = 0
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
clock_gettime(CLOCK_MONOTONIC, {2437, 254793402}) = 0
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
clock_gettime(CLOCK_MONOTONIC, {2437, 255247650}) = 0
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
clock_gettime(CLOCK_MONOTONIC, {2437, 255914774}) = 0
futex(0x208dde0, FUTEX_WAKE, 1)         = 1
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
futex(0xc820382e90, FUTEX_WAKE, 1)      = 1
futex(0x20844d0, FUTEX_WAIT, 0, NULL)   = 0
futex(0xc820065c10, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0xc820136590, FUTEX_WAKE, 1)      = 1
futex(0x20844d0, FUTEX_WAIT, 0, NULL)   = 0
mmap(0xc820600000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc820600000
mmap(0xc81ffc8000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc81ffc8000
clock_gettime(CLOCK_MONOTONIC, {2437, 263504006}) = 0
clock_gettime(CLOCK_REALTIME, {1457980615, 124660973}) = 0
clock_gettime(CLOCK_REALTIME, {1457980615, 124888376}) = 0
uname({sys="Linux", node="jenkins2a", ...}) = 0
uname({sys="Linux", node="jenkins2a", ...}) = 0
clock_gettime(CLOCK_REALTIME, {1457980615, 125881519}) = 0
clock_gettime(CLOCK_REALTIME, {1457980615, 126159208}) = 0
clock_gettime(CLOCK_REALTIME, {1457980615, 126359512}) = 0
socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/run/systemd/journal/socket"}, 30) = -1 ENOENT (No such file or directory)
close(3)                                = 0
stat("/home/ubuntu/.docker/config.json", 0xc8203781b0) = -1 ENOENT (No such file or directory)
stat("/home/ubuntu/.dockercfg", 0xc820378240) = -1 ENOENT (No such file or directory)
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
clock_gettime(CLOCK_REALTIME, {1457980615, 128931348}) = 0
clock_gettime(CLOCK_REALTIME, {1457980615, 129150649}) = 0
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, 23) = 0
clock_gettime(CLOCK_REALTIME, {1457980615, 130003272}) = 0
epoll_create1(EPOLL_CLOEXEC)            = 4
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=102120360, u64=140114820217768}}) = 0
getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0
getpeername(3, {sa_family=AF_LOCAL, sun_path="/var/run/docker.sock"}, [23]) = 0
futex(0xc820064110, FUTEX_WAKE, 1)      = 1
read(3, 0xc820233000, 4096)             = -1 EAGAIN (Resource temporarily unavailable)
futex(0x20844d0, FUTEX_WAIT, 0, NULL

Is that more helpful?

[sleaze]

See also: #18758

[thaJeztah]

@sleaze just asked around, but that strace probably doesn't contain useful information, because thats from the client, not the daemon.

[sleaze]

@thaJeztah I appreciate you looking into this for me, seriously you're the man!!!

I actually found a hack-ish workaround based on @SamSaffron's brilliant observations.

The "problem" goes away as long as pkill is invoked against any straggler processes before Jenkins initiates docker stop on the container (for example, Riak leaves an "epmd" process running even after the service is stopped).

I wonder how many other people in this thread are experiencing the same thing as @SamSaffron and I...

This certainly makes docker seem ill-equipped to handle many real-life situations compared things like LX[CD] which natively support multiple process environments and use-cases.

[magicphip]

@sleaze

I wonder how many other people in this thread are experiencing the same thing as @SamSaffron and I...

I do. My previous comment gets explained by @SamSaffron observations