node_exporter prevents docker from removing monitored containers?

[keyolk]

Host operating system:

Linux css 4.4.68-nx #122 SMP Mon May 15 09:46:11 KST 2017 x86_64 GNU/Linux

node_exporter version:

  build user:       root@bb6d0678e7f3
  build date:       20170321-12:12:54
  go version:       go1.7.5

Are you running node_exporter in Docker?

yes

What did you do that produced an error?

Seems it is related to this issue from cadvisor
and this http://blog.hashbangbash.com/2014/11/docker-devicemapper-fix-for-device-or-resource-busy-ebusy/

1. use devicemapper as a storage driver
2. launch any of a container
3. deploy node exporter

$ docker run -d \
  --net=host \
  -v "/proc:/host/proc:ro" \
  -v "/sys:/host/sys:ro" \
  -v "/":/rootfs:ro \
  registry.navercorp.com/cocofarm/node-exporter:latest \
    -collector.procfs /host/proc \
    -collector.sysfs /host/sys \
    -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

4. delete the container deployed first

Result is

time="2017-06-13T15:58:36.031272839+09:00" level=error msg="Error unmounting container 05b7a3e4ec8706415e866c3e52b1530b8c0dd4d1d614530c0d40934f01c5f144: Device is Busy"

and I found that the node_exporter container holds the container's devicemapper mount point

$ cat /proc/`/sbin/pidof node_exporter`/mounts | grep xfs
...
/dev/mapper/docker-8:3-6816084-f692988f2d272fe116016cde6a43d5228032deeeb40590a2b390c196492a6802 /rootfs/var/lib/docker/devicemapper/mnt/f692988f2d272fe116016cde6a43d5228032deeeb40590a2b390c196492a6802 xfs 
rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
...

What I assume is ..
if new container is created with volume including /var/lib/docker,
all of mount namespace of existing containers is leaked to new container. : (

and is it ok to use node exporter container with out " -v "/:/rootfs:ro" \ " ?

[matthiasr]

This seems like a Docker issue to me. Am I missing anything that node exporter is doing for this case, or does the same happen if you run

docker run -d \
  --net=host \
  -v "/proc:/host/proc:ro" \
  -v "/sys:/host/sys:ro" \
  -v "/":/rootfs:ro \
  busybox \
    sleep 3600

is it ok to use node exporter container with out " -v "/:/rootfs:ro" \ " ?

I honestly don't know, try it out?

[matthiasr]

Ah, I understand now, the /rootfs mount gets you the filesystem metrics mentioned in #600. I'm afraid there is no way to get the host mount metrics except by leaking the host mount namespace into the container, otherwise the whole kernel namespacing would be pointless.

I think this can only be fixed in Docker, if at all.

[keyolk]

I see Thanks @mattbostock

[keyolk]

@mattbostock
I think it can be mitigated.
if I can put exact volumes to be used to the container.
what I means just take off /var/lib/docker/devicemapper being mounted.

could you inform me what of exact host data it uses ?

[matthiasr]

See #591 for details on how this data is collected. As far as I understand, each filesystem that is mounted into the container's mount namespace is represented. By mounting in / you get them all. If you know what your actual mount points are you could only mount in a subset of them, but if you want metrics about the root volume you're just back where you started.

Pending the namespace breakout in the other issue, all I can recommend is not to run node exporter in a container if you can help it at all. Put it directly under control of the host supervisor where it has a view of the whole machine.

[maffelbaffel]

I had this issue for a quite a long time, not really knowing where it came from.
This should be noted in the Using Docker part of the Readme.