Dockerised server cannot access bindmounts with the documented group ID

[fedde-s]

Elasticsearch version (docker run --rm docker.elastic.co/elasticsearch/elasticsearch-oss:7.4.0 elasticsearch --version):

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Version: 7.4.0, Build: oss/docker/22e1767283e61a198cb4db791ea66e3f11ab9910/2019-09-27T08:36:48.569419Z, JVM: 13

Plugins installed: []

JVM version (docker run --rm docker.elastic.co/elasticsearch/elasticsearch-oss:7.4.0 /bin/bash -c 'source "`dirname "$0"`"/elasticsearch-env && "$JAVA" --version' /usr/share/elasticsearch/bin/elasticsearch):

openjdk 13 2019-09-17
OpenJDK Runtime Environment AdoptOpenJDK (build 13+33)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13+33, mixed mode, sharing)

OS version (uname -srvm):

Linux 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64

Description of the problem including expected versus actual behavior:

The documentation on bind-mounting configuration into an Elasticsearch Docker container claims that the container will run Elasticsearch as a user with group ID 1000, and the notes on defaults underneath contain detailed instructions to create a bind-mountable data directory to be accessed by such a user. However, when I follow these instructions, Elasticsearch fails with this message:

"org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",

The output below suggests that the Dockerfile did add a user with this default group, but the command that the entrypoint uses to switch to that user's user ID does not reference that configuration to set the group ID for the user.

% docker run --rm -it --entrypoint='/bin/bash' docker.elastic.co/elasticsearch/elasticsearch-oss:7.4.0
[root@0e0c8897af8d elasticsearch]# id
uid=0(root) gid=0(root) groups=0(root)
[root@0e0c8897af8d elasticsearch]# grep elasticsearch /etc/passwd
elasticsearch:x:1000:1000::/usr/share/elasticsearch:/bin/bash
[root@0e0c8897af8d elasticsearch]# grep elasticsearch /etc/group 
root:x:0:elasticsearch
elasticsearch:x:1000:
[root@0e0c8897af8d elasticsearch]# chroot --userspec=1000 / /bin/bash
[elasticsearch@0e0c8897af8d /]$ id
uid=1000(elasticsearch) gid=0(root) groups=0(root)

Steps to reproduce:

1. As a user that does not have user ID 1000 on the host, create a folder as per the documentation linked above

sudo -- sh -c 'mkdir esdatadir && chmod g+rwx esdatadir && chgrp 1000 esdatadir'

2. Run a container that mounts the folder read-write as its data dir

docker run --rm --ulimit nofile=65535:65535 -e "discovery.type=single-node" -v "$PWD/esdatadir:/usr/share/elasticsearch/data" docker.elastic.co/elasticsearch/elasticsearch-oss:7.4.0

Provide logs (if relevant):

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-10-11T15:36:59,291Z", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "889776d11c03", "message":
"uncaught exception in thread [main]",                                                                       
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.0.jar:7.4.0]",                                                                                                      "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.0.jar:7.4.0]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:614) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]",
"... 6 more"
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:209) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:269) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:275) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.0.jar:7.4.0]",
"... 6 more"] }

[fedde-s]

Since the Elasticsearch images are based on CentOS 7, they use a chroot version feature-frozen since 2013. chroot only has only started applying user-associated groups in 2014. I believe that to be the cause of this bug in the Elasticsearch images; recent documentation for chroot will say that --userspec will look up the groups for the user, but that is not the case in 8.22.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[dliappis]

The documentation on bind-mounting configuration into an Elasticsearch Docker container claims that the container will run Elasticsearch as a user with group ID 1000

@fedde-s This is indeed something that should be addressed in the docs, given that as you correctly mentioned the chroot version available in centos:7 doesn't apply supplemental groups of the specified user. The current behavior, i.e. running with gid:0 is somewhat desired to enable running on Openshift/OKD and is a rather common pattern (e.g. also used in gocd).

Practically speaking I think we should adjust the docs in the linked section to:

1. Mention that Elasticsearch runs with uid:1000 and not claim that gid is 1000.
2. For bind mounting, recommend the Openshift guidelines#SUPPORT ARBITRARY USER IDS to ensure bind mounted directories with different UIDs to uid:1000 are gid:0.

For your use case you'd use something like (I chose a random uid:1500 -- uid:0 will work too ):

sudo -- sh -c 'mkdir esdatadir && chmod g+rwx esdatadir && chgrp 0 esdatadir && chown 1500 esdatadir'

which allows the docker image to run using:

docker run --rm -e "discovery.type=single-node" -v "$PWD/esdatadir:/usr/share/elasticsearch/data" docker.elastic.co/elasticsearch/elasticsearch-oss:7.4.0

In the future when we switch to a newer base image with a newer chown that applies supplementary groups we can still readjust the docs from that version onwards.

[dliappis]

Updates to the docs have been done in #49529 . I've also opened #49632 to backport the changes to version 7.6.