no devicemapper support in 18.09.x on ubuntu and debian

[akosyrev]

• This is a bug report
• [x ] I searched existing issues before opening this one

I found that last nightly build from this repo https://download.docker.com/linux/ubuntu/dists/xenial/pool/nightly/amd64/ fails with error, when i'm trying to run it with devicemapper:

{
    "storage-driver": "devicemapper"
}

Error message:

failed to start daemon: error initializing graphdriver: driver not supported```

I recompiled it from sources and error is still here. iI built from commit 7718f802c486607f080ab8158953dc668e3f09fb

Also I noticed that bightly build from 2018-09-26 is less than 2018-09-20: 17M vs 24M 

[akosyrev]

https://github.com/docker/docker-ce-packaging/pull/223/files#diff-6670e1ec6985ef6cc31ddd74f1e52e70R43

[domste]

Have the same problem. Is there a reason why devicemapper is excluded?

[Halo9Pan]

docker/cli#1424
"Deprecate", but it was excluded...

[teadur]

Same here devicemapper is excluded on debian stretch starting from 18.09

[cpuguy83]

ping @andrewhsu

[thaJeztah]

This is being tracked internally; opened an issue for the packaging team.

[nemesis545]

Same here devicemapper is excluded on debian buster starting from Docker version 18.09.0, build 4d60db4

[nemesis545]

i got this:
Unable to locate plugin: devicemapper, retrying in 1s
dockerd -s devicemapper --experimental -D

DEBU[2018-11-13T10:38:00.517663555-05:00] Listener created for HTTP on unix (/var/run/docker.sock) 
WARN[2018-11-13T10:38:00.521002154-05:00] failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: file exists. Deleting synchronously 
DEBU[2018-11-13T10:38:00.521222922-05:00] Golang's threads limit set to 114570         
INFO[2018-11-13T10:38:00.521525999-05:00] parsed scheme: "unix"                         module=grpc
INFO[2018-11-13T10:38:00.521542834-05:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2018-11-13T10:38:00.521602212-05:00] parsed scheme: "unix"                         module=grpc
INFO[2018-11-13T10:38:00.521616361-05:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2018-11-13T10:38:00.521683461-05:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
DEBU[2018-11-13T10:38:00.521716188-05:00] Using default logging driver json-file       
DEBU[2018-11-13T10:38:00.521746247-05:00] [graphdriver] trying provided driver: devicemapper 
INFO[2018-11-13T10:38:00.521806617-05:00] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]  module=grpc
INFO[2018-11-13T10:38:00.521846492-05:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2018-11-13T10:38:00.521727080-05:00] ClientConn switching balancer to "pick_first"  module=grpc
WARN[2018-11-13T10:38:00.521924066-05:00] Unable to locate plugin: devicemapper, retrying in 1s 
INFO[2018-11-13T10:38:00.521923492-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42019a160, CONNECTING  module=grpc
INFO[2018-11-13T10:38:00.521959850-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42016c9e0, CONNECTING  module=grpc
INFO[2018-11-13T10:38:00.522111703-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42016c9e0, READY  module=grpc
INFO[2018-11-13T10:38:00.522124829-05:00] pickfirstBalancer: HandleSubConnStateChange: 0xc42019a160, READY  module=grpc
DEBU[2018-11-13T10:38:00.522207455-05:00] processing event stream                       module=libcontainerd namespace=plugins.moby
WARN[2018-11-13T10:38:01.522307695-05:00] Unable to locate plugin: devicemapper, retrying in 2s 
WARN[2018-11-13T10:38:03.522773842-05:00] Unable to locate plugin: devicemapper, retrying in 4s 
^CINFO[2018-11-13T10:38:04.863216849-05:00] Processing signal 'interrupt'                
WARN[2018-11-13T10:38:07.523305063-05:00] Unable to locate plugin: devicemapper, retrying in 8s 
^CINFO[2018-11-13T10:38:08.095129525-05:00] Processing signal 'interrupt'                
^CINFO[2018-11-13T10:38:08.303084600-05:00] Processing signal 'interrupt'                
^CINFO[2018-11-13T10:38:08.558976666-05:00] Processing signal 'interrupt'                
INFO[2018-11-13T10:38:08.559126689-05:00] Forcing docker daemon shutdown without cleanup; 3 interrupts received ```

[sboeuf]

Hi @thaJeztah, what's the status on this issue? Is this resolved now?

[amshinde]

@thaJeztah We rely on the block storage provided by devicemapper storage driver, so would really like to see this resolved.

[sboeuf]

Also, are you planning to support any alternative to devicemapper? Because block device support provides better performances in our case.

[thaJeztah]

I'm not sure what the current status is; I know this was something that was being looked into, but no details yet if they found a solution.

Also, are you planning to support any alternative to devicemapper?

The current roadmap is to complete the migration to using containerd as a runtime, and for image management (see moby/moby#38043). Currently, the functionality of containerd and dockerd overlaps (as in; they both are capable of image-management, and both implement storage drivers). The goal is to hand over the overlapping parts to containerd. Currently containerd has support for overlay, zfs, btrfs, and aufs as storage drivers ("snapshotters") but additional storage drivers could be added upstream in the containerd project. t.b.h., I'm not sure if devicemapper would find its way in there. Maintenance of devicemapper was largely driven by Red Hat (as it was the only option for their kernels), but now that overlay has become mainstream, focus has shifted in that direction.

Because block device support provides better performances in our case.

Do you have more information about your use-case? Is this for writes or reads ? (for writes, the general recommendation is to use a volume, to skip the CoW performance penalty)

[sboeuf]

@thaJeztah

I'm not sure if devicemapper would find its way in there. Maintenance of devicemapper was largely driven by Red Hat (as it was the only option for their kernels), but now that overlay has become mainstream, focus has shifted in that direction.

In our case, we get better performance with block device rootfs. This means that we're not strongly attached to devicemapper itself but to the fact that it exposes block devices. And so it's important for us to keep at least one storage driver that exposes rootfs as block device if we want to maintain this level of performance.

Do you have more information about your use-case?

Because Kata Containers runtime spawn a VM, and because the rootfs needs to be passed inside this VM, we are limited by the device model exposed by QEMU. And in this case, using virtio-block or virtio-scsi is more performant than using 9pfs. But virtio-scsi is block device based... that's the rationale why we need a storage driver exposing rootfs as block device.

[thaJeztah]

Gotcha, thanks for the extra info. /cc @dmcgowan @crosbymichael

[maityh]

Did anyone get rid from this issue? After upgrading the docker system we had faced the same and downgraded the version. Switch to the overlay2 is the only option?

[nemesis545]

that is the only option, i rollback to previous version just for export a docker image, them upgrade again and import the docker image...

[enoch85]

This is not a "one-stop-shop" solution, but might help anyone looking for migrating to overlay2.

https://github.com/nextcloud/vm/blob/master/static/docker_overlay2.sh

[gerroon]

This is a wreckless upgrade, how on earth. I had to downgrade too.

[burka]

I would pin the package to 18.06 until this issue has been resolved.

cat > /etc/apt/preferences.d/docker  <<EOF
Package: docker-ce
Pin: version 18.06*
Pin-Priority: 1000
EOF

apt-get install docker-ce 
journalctl -xe | grep docker

I posted this a suggested solution on askubuntu: https://askubuntu.com/a/1111895/24580.

[nemesis545]

the upgrade's inevitable, the simplest option is a rollback, export, upgrade and import.

[cpuguy83]

So, break existing users for a feature that nobody is using? 👻

[thaJeztah]

FWIW, it's still in the .rpm packages (CentOS, RHEL, Fedora), but not in the .deb (Debian, Ubuntu)

[cpuguy83]

That just makes no sense to me. Are these new features not supported on CentOS/Fedora/RHEL?

This is a pretty clear cut break with no explanation at all, not even "oh we messed up but aren't going to fix it either", just silence.

[andrewhsu]

Docker CE 18.09.x works with CentOS and Fedora configured with devicemapper. Docker does not qualify Docker CE on RHEL before release, but it is likely to work based on how RHEL is related to CentOS.

We intended to have Docker CE 18.09.x work with Ubuntu and Debian configured with devicemapper as well, but we have encountered complications with the way we build the binaries for the packages that was discovered after release. This is because we did not qualify Docker CE 18.09.x on devicemapper, but on the newer-supported overlay2 which we intend to have as the default, hence the deprecation notice for devicemapper.

Apologies for the problem this has caused existing devicemapper users on Ubuntu and Debian. This is not the way we intend to phase out support for something we know as common use. This is a bug that has definitely slipped through the cracks after the initial 18.09.0 release.

I'll circle back around with the team on what we gotta do to get the devicemapper support back in for 18.09.x on Ubuntu and Debian. In the meantime, if migration is an option, there are instructions for howto convert devicemapper to overlay2, to be performed on your system before upgrade to 18.09.x obviously.

[cpuguy83]

Thanks for taking a look, @andrewhsu.

[gerroon]

@andrewhsu

I hope you realize that there are a lot of layman users like me out there, I just use Docker when the app developers recommend as a preferred method. I do not do Docker dev or Linux system configurations. I am just a basic user who needs to use Docker because it is recommended. When the system breaks like this in the middle of nothing, it is very hard to make sense of what is going on in these situations. I did not know that my Docker appps all were down, until when I realized that all my web apps were down.

If the migration is the only option then please provide a way to automatically migrate all this stuff.

[instantlinux]

@andrewhsu there are reasons as a non-layman user I explicitly CHOSE to use devicemapper rather than overlay2. None of the documentation I've seen has stated WHY overlay2 is better than devicemapper, or refutes the reasons that I and many thousands of others chose devicemapper.

My reasons? Once configured properly (which isn't that hard, and is easily automated with ansible), the devicemapper thinpool provides what I believe to be the fastest-possible performance. And it enables automatic LVM volume-size extension; I can allocate (say) 10GB on a terabyte physical volume in set-and-forget fashion. The volume might grow to 50GB or 100GB but I can still use the rest for purposes other than docker images.

In a nutshell: I went through the list of available storage drivers circa Feb 2017, chose devicemapper, and have had no reason to reconsider that choice--until the snafu with your company's CI failure in the 18.09 release cycle. And now I have to assume that your managers are busily reprioritizing engineering tasks and aren't likely to ever revisit devicemapper. In any event, before closing this ticket out as won't-fix, please have someone document the advantages of overlay2 better and explain the disadvantages of devicemapper (if any). Because I still don't see why I'm forced to make this change, and I'm concerned about suddenly losing autoextend or I/O performance afterward.

[svvac]

In the meantime, if migration is an option, there are instructions for howto convert devicemapper to overlay2, to be performed on your system before upgrade to 18.09.x obviously.

The referenced procedure doesn't lay out how to migrate an existing devicemapper install. It basically just tells you to scratch your existing storage and create a new one, which I dont think is an acceptable migration procedure in any case.

[easp]

Also an issue on Xenial.

[ganeshmaharaj]

With docker 18.09 using containerd as the runtime handler, is there a way to docker use a storage-driver that exists in containerd? There are multiple efforts to add block device support to containerd here, here and here. It will be nice to allow one of these to be the default storage driver for docker.

[thaJeztah]

With docker 18.09 using containerd as the runtime handler, is there a way to docker use a storage-driver that exists in containerd?

Not possible yet, as those parts are not yet integrated with the docker daemon (storage-drivers are still handled by dockerd). Work is in progress to integrate those parts; see moby/moby#38738 and moby/moby#38925)

[seemethere]

Hey guys!

The latest nightly builds after docker/docker-ce-packaging#314 was merged should contain devicemapper support on Ubuntu and Debian and should be included in the next release for Docker CE!

You can install the latest nightly builds with get.docker.com using:

curl -fsSL get.docker.com | CHANNEL=nightly sh

Ran a quick test with Ubuntu 18.04 to verify:

~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

~$ docker --version
Docker version 0.0.0-20190329010133-83264de7d5, build 83264de7d5

~$ docker info 2>/dev/null | grep Storage
 Storage Driver: devicemapper

Keep in mind that devicemapper will still be deprecated at some point in the future so moving to overlay2 is still preferred.

[egernst]

@seemethere - can the deprecation be held off until #452 (comment) is addressed?

For some users, overlay2 isn't an option. Once above is addressed, they'll have a viable and supported option.

@thaJeztah WDYT?

[thaJeztah]

Devicemapper will still be there in the upcoming (19.03) release. In addition, work is in progress in containerd to provide block-device snapshotters;

• Integrate device mapper snapshotter containerd/containerd#3022 Integrate device mapper snapshotter (just merged)
• Introduce the rawblock snapshotter containerd/containerd#3130 Introduce the rawblock snapshotter
• Enable LVM based block snapshotter containerd/containerd#3136 Enable LVM based block snapshotter

[egernst]

Yep - I just want to make sure containerd snapshotters work with docker (see Ganesh’s prior comment).

[megakoresh]

Even though docker/docker-ce-packaging#314 has been released, this issue is still present for 18.09.5 for yum repositories

[Hypocritus]

I don't understand how a company with as reputable a name as Docker allows an issue this disruptive, first, to happen, or worse, to continue effectively unaddressed and untreated for over six months, despite the caliber of professionals which continue to (for all intents and purposes) politely petition an appropriate and open-minded response or set of solutions.

Attempting to use the "supported" and functional ZFS driver, which unfortunately no longer works due to this change.

Use Overlay2? No! And my reasons are just as valid as the infinitely diverse other reasons had and given by other sysadmins here and elsewhere.

[dmcgowan]

to continue effectively unaddressed and untreated for over six months

Not sure if you read the prior comments, this was addressed and taken seriously. If you are continuing to have a problem after the proposed solution, please highlight what steps you tried and how they have failed.

Use Overlay2? No!

No one is forcing anyone to use overlay2. The devicemapper driver in Docker is being deprecated due to lack of maintenance, it is not being immediately removed. Rather devicemapper support will be done through containerd in the future and in the long term replaces the deprecated graphdriver implementation. I think this wasn't communicated well and the unintentional package breakage gave the impression that they could not longer use devicemapper with Docker anymore. That is not the case, please see the work being done in containerd to support for block device snapshotters.

@thaJeztah @seemethere if this issue has been fixed can you close this issue?

[instantlinux]

@dmcgowan as noted above already, the issue hasn't been fixed in a way that we users have access to. Until a 19.x version comes out, we're having to run back-ported updates of 18.06.x.

[seemethere]

This issue has been resolved in 19.03.X, will close this issue once a GA version of that package is available.

[tao12345666333]

This issue has been resolved in 19.03.X, will close this issue once a GA version of that package is available.

https://github.com/docker/docker-ce/releases/tag/v19.03.1 19.03 has been released. Should we close this issue?

[hdp7891000]

centos 8.5
docker 18.09.9 and above (https://download.docker.com/linux/static/stable/x86_64/docker-18.09.9.tgz)
also have the same problem...

"Failed to GetDriver graph" driver=devicemapper error="graphdriver plugins are only supported with experimental mode"
Error starting daemon: error initializing graphdriver: driver not supported

[thaJeztah]

I don't think it's part of the static binaries (which are mostly intended for testing purposes); for CentOS, install the dynamically linked packages (RPMs); https://docs.docker.com/engine/install/centos/

[hdp7891000]

Thank you very much, it has been verified that the static build version does not support devicemapper