Verify Elasticsearch Docker container writes on plugin installation #69533

mieciu · 2021-02-24T13:19:19Z

Elasticsearch version: 8.0.0-SNAPSHOT

OS version: Docker container

When running Elasticsearch Docker container with an arbitrary UID/GID it's impossible to install a plugin:

$ docker run --user 31337:1337 -i -t docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT bash
bash-4.4$ ./bin/elasticsearch-plugin install --batch https://snapshots.elastic.co/8.0.0-000145a6/downloads/elasticsearch-plugins/repository-s3/repository-s3-8.0.0-SNAPSHOT.zip
-> Installing https://snapshots.elastic.co/8.0.0-000145a6/downloads/elasticsearch-plugins/repository-s3/repository-s3-8.0.0-SNAPSHOT.zip
-> Downloading https://snapshots.elastic.co/8.0.0-000145a6/downloads/elasticsearch-plugins/repository-s3/repository-s3-8.0.0-SNAPSHOT.zip
-> Failed installing https://snapshots.elastic.co/8.0.0-000145a6/downloads/elasticsearch-plugins/repository-s3/repository-s3-8.0.0-SNAPSHOT.zip
-> Rolling back https://snapshots.elastic.co/8.0.0-000145a6/downloads/elasticsearch-plugins/repository-s3/repository-s3-8.0.0-SNAPSHOT.zip
-> Rolled back https://snapshots.elastic.co/8.0.0-000145a6/downloads/elasticsearch-plugins/repository-s3/repository-s3-8.0.0-SNAPSHOT.zip
Exception in thread "main" java.nio.file.AccessDeniedException: /usr/share/elasticsearch/plugins/.installing-668324506119825940
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
	at java.base/java.nio.file.Files.createDirectory(Files.java:694)
	at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:135)
	at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:172)
	at java.base/java.nio.file.Files.createTempDirectory(Files.java:970)
	at org.elasticsearch.plugins.InstallPluginCommand.stagingDirectory(InstallPluginCommand.java:738)
	at org.elasticsearch.plugins.InstallPluginCommand.unzip(InstallPluginCommand.java:685)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:227)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:199)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:80)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
	at org.elasticsearch.cli.Command.main(Command.java:79)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:36)

According to docs, there is a handy workaround for this - since /usr/share/elasticsearch is group-writable user can leverage the --group-add 0 option.

That is described as a desired way to go unless you bind-mount config/data/logs directory. The ./bin/elasticsearch-plugin tool however, writes outside of those locations.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-02-24T13:19:21Z

Pinging @elastic/es-delivery (Team:Delivery)

mieciu · 2021-02-24T13:24:38Z

💡An idea to keep consistency
It would be great to have that plugin path configurable, just like it's currently done with logs/conf/data dirs. Therefore Elasticsearch running inside of the Docker container could just receive any plugin path as an option (-Epath.logs=/app/logs -Epath.data=/app/data -Epath.plugins=/app/plugins).

Then, user running Elasticsearch Docker container would be able to supply just one bind mount to the Docker container.

elasticmachine · 2021-02-24T13:36:07Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

pugnascotia · 2021-02-24T13:36:54Z

👍 from me on making the plugins directory configurable. Ping @rjernst / @mark-vieira for their thoughts.

pugnascotia · 2021-02-24T13:38:33Z

I should note that there's also an issue with the docs, since we ought to instruct users to bind-mount plugins as well if they are running the Docker container with an arbitrary UID+GID, if they're not also passing --add-group 0 to Docker.

mark-vieira · 2021-02-24T18:15:15Z

+1 from me on making the plugins directory configurable. Ping @rjernst / @mark-vieira for their thoughts.

I don't have strong feelings about that, it's really a core/infra issue.

I should note that there's also an issue with the docs, since we ought to instruct users to bind-mount plugins as well if they are running the Docker container with an arbitrary UID+GID, if they're not also passing --add-group 0 to Docker.

Right, this seems like the core issue here, which is missing documentation when running the container as user other than root. We might also want to ensure we have packaging test coverage for this scenario.

rjernst · 2021-02-25T06:24:19Z

The plugins directory used to be configurable long ago. We intentionally moved away from that. But it can still be a symlink. I don't think we should add back the configuration option.

mieciu · 2021-02-25T13:01:50Z

The plugins directory used to be configurable long ago. We intentionally moved away from that.

@rjernst do you recall what were the rationale behind not having that path configurable?

jasontedor · 2021-02-25T14:34:11Z

Conceptually, plugins are libraries that Elasticsearch depends on as opposed to user data for which it makes sense for their location to be user-configurable. We don't allow users to relocate the core Elasticsearch JARs and modules, and we want to treat plugins the same. It's a packaging and execution simplification. I'd personally prefer that we remove even the ability for this to be a symlink than we make this user configurable.

pugnascotia · 2021-02-25T15:00:33Z

The difference though is that plugins are installed by the user, which places them somewhere between user data and program data. I don't know if that's a sufficiently meaningful distinction to matter, but it makes sense that a Docker user could regard the plugin directory as non-ephemeral, in the same way as data and logs.

rjernst · 2021-02-25T17:48:03Z

Would a docker user treat installed packages in the OS as non-ephemeral? That is the closest thing to plugins. They are, IMO, part of the image, whether they are installed on the fly or baked in. We make no guarantees one could install on one instance and the directory would be portable to another.

Coming back to the original problem: can this be fixed by changing the permissions on launching of the docker image which I believe we have done in other cases? It seems related to docker alone, in that it has this special ability to run as a different uid/gid.

mark-vieira · 2021-02-25T18:15:37Z

We make no guarantees one could install on one instance and the directory would be portable to another.

This is an important point I think. Making this configurable/relocatable I think brings along all sorts of other implications and opens the ability for use cases we don't explicitly support.

jasontedor · 2021-02-25T18:22:33Z

The difference though is that plugins are installed by the user, which places them somewhere between user data and program data.

Yeah, I'm with @rjernst here. How they are loaded doesn't place them between user data and program data. They are still program data, they just happen to be loaded through explicit user interaction.

pugnascotia · 2021-03-01T12:21:36Z

Alrighty then. I'll update the Docker docs to cover plugins.

alpar-t · 2021-03-01T13:30:54Z

If we consider plugins as part of the image, shouldn't plugin installation just work when using the docker image, regardless of the user and group it's started with? What's the reason behind restricting write access to the plugins directory in the container? AFAIK if ES is compromised and the security manager bypassed wouldn't one still be able to write to the plugins directory (as everything is running with the same user in docker)? I can understand the concern in a non docker world, where a malicious user, other than the one running ES on the machine could compromise ES by installing a plugin. In a docker world, such a user would have to have access to docker AFAIK at which point I don't think there's any protection that could be added at the container level. Could this be solved by making the plugins directory world writable in the docker image only?

mieciu · 2021-03-02T14:41:16Z

Could this be solved by making the plugins directory world writable in the docker image only?

That'd clearly do the trick here.

pugnascotia · 2021-09-10T10:04:52Z

@mieciu is this still an issue?

mieciu · 2021-09-13T10:22:11Z

I believe so:

przemyslaw@serenity cloud % docker run -i -t -u 2137:7312  docker.elastic.co/elasticsearch-ci/elasticsearch-cloud-ess:8.0.0-SNAPSHOT bash
bash-4.4$ ./bin/elasticsearch-plugin install --batch https://snapshots.elastic.co/8.0.0-dea1303e/downloads/elasticsearch-plugins/store-smb/store-smb-8.0.0-SNAPSHOT.zip
-> Installing https://snapshots.elastic.co/8.0.0-dea1303e/downloads/elasticsearch-plugins/store-smb/store-smb-8.0.0-SNAPSHOT.zip
-> Downloading https://snapshots.elastic.co/8.0.0-dea1303e/downloads/elasticsearch-plugins/store-smb/store-smb-8.0.0-SNAPSHOT.zip
-> Failed installing https://snapshots.elastic.co/8.0.0-dea1303e/downloads/elasticsearch-plugins/store-smb/store-smb-8.0.0-SNAPSHOT.zip
-> Rolling back https://snapshots.elastic.co/8.0.0-dea1303e/downloads/elasticsearch-plugins/store-smb/store-smb-8.0.0-SNAPSHOT.zip
-> Rolled back https://snapshots.elastic.co/8.0.0-dea1303e/downloads/elasticsearch-plugins/store-smb/store-smb-8.0.0-SNAPSHOT.zip
Exception in thread "main" java.nio.file.AccessDeniedException: /usr/share/elasticsearch/plugins/.installing-3303704978444359380
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:396)
	at java.base/java.nio.file.Files.createDirectory(Files.java:694)
	at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:133)
	at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:170)
	at java.base/java.nio.file.Files.createTempDirectory(Files.java:970)
	at org.elasticsearch.plugins.InstallPluginAction.stagingDirectory(InstallPluginAction.java:739)
	at org.elasticsearch.plugins.InstallPluginAction.unzip(InstallPluginAction.java:686)
	at org.elasticsearch.plugins.InstallPluginAction.execute(InstallPluginAction.java:214)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:84)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:95)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
	at org.elasticsearch.cli.Command.main(Command.java:79)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:36)
bash-4.4$
bash-4.4$
bash-4.4$
bash-4.4$ ^C
bash-4.4$ exit
przemyslaw@serenity cloud % docker run -i -t -u 2137:7312 --group-add 0  docker.elastic.co/elasticsearch-ci/elasticsearch-cloud-ess:8.0.0-SNAPSHOT bash
bash-4.4$ ./bin/elasticsearch-plugin install --batch https://snapshots.elastic.co/8.0.0-dea1303e/downloads/elasticsearch-plugins/store-smb/store-smb-8.0.0-SNAPSHOT.zip
-> Installing https://snapshots.elastic.co/8.0.0-dea1303e/downloads/elasticsearch-plugins/store-smb/store-smb-8.0.0-SNAPSHOT.zip
-> Downloading https://snapshots.elastic.co/8.0.0-dea1303e/downloads/elasticsearch-plugins/store-smb/store-smb-8.0.0-SNAPSHOT.zip
-> Installed store-smb
-> Please restart Elasticsearch to activate any plugins installed
bash-4.4$

rstatsinger · 2022-01-13T22:18:00Z

@pugnascotia what is the status of this bug? Its affecting Visa and is one of the "thousand paper cuts" that is slowly killing them. Any updates?

mark-vieira · 2022-01-13T23:06:25Z

I'm not sure there is still an existing issue here. As I see it using --group-add 0 or, additionally bind-mounting the plugins directory are both satisfactory options for this use case. I think the only issue is the latter is not properly documented.

pugnascotia · 2022-01-14T10:12:52Z

Mark is right, a user has 3 options:

Create their own images on top of ours, with added plugins
Add --group-add 0 when running the image
Bind-mount the plugins directory (for which I'll update the docs)

If none of these options are working for a particular user, then we'd like to know more.

rstatsinger · 2022-01-14T15:16:26Z

Thanks guys.

…

On Fri, Jan 14, 2022 at 2:13 AM Rory Hunter ***@***.***> wrote: Mark is right, a user has 3 options: - Create their own images on top of ours, with added plugins - Add --group-add 0 when running the image - Bind-mount the plugins directory (for which I'll update the docs) If none of these options are working for a particular user, then we'd like to know more. — Reply to this email directly, view it on GitHub <#69533 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHB3YGE3LUSAQ6LF34WESFLUV7ZLLANCNFSM4YEQBL3Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

-- *Elastic: Search. Observe. Protect.* Robert Statsinger | Principal Solutions Architect | www.elastic.co Self-schedule a meeting with me using Calendly <https://calendly.com/robert-statsinger/meeting-with-robert-statsinger>

Closes #69533. The Docker docs mention bind-mounting the `config`, `data` and `logs` directories when using an arbitrary UID / GID, but they fail to mention that the `plugins` dir must also be mounted in order to install plugins.

jasontedor · 2022-01-24T22:20:31Z

@pugnascotia I think there's one other option which is to use a tmpfs mount for /usr/share/elasticsearch/plugins. I especially like this idea in tandem with using declarative plugin management. A bind mount is useful for situations where you want to manage the lifecycle of the data separately from the lifecycle of the container. For example, user data in config/data/logs, but with plugins and declarative plugin management, Elasticsearch manages all of that for the user, and there isn't really a good reason to manage/persist it separately.

mark-vieira · 2022-01-24T23:05:46Z

For example, user data in config/data/logs, but with plugins and declarative plugin management, Elasticsearch manages all of that for the user, and there isn't really a good reason to manage/persist it separately.

I suppose not, aside from the startup cost of downloading and installing the plugins on every container restart.

jasontedor · 2022-01-25T01:22:12Z

I suppose not, aside from the startup cost of downloading and installing the plugins on every container restart.

In general I think that's okay since in-place container restarts are expected to be rare events.

Kushmaro · 2022-01-25T12:08:45Z

Mark is right, a user has 3 options:

Create their own images on top of ours, with added plugins

Add --group-add 0 when running the image

Bind-mount the plugins directory (for which I'll update the docs)

If none of these options are working for a particular user, then we'd like to know more.

@pugnascotia I want to call something out here from the cloud end -
none of these options are easy to use or implement when it comes to our cloud service (or on-prem ECE product, which is where Visa is experiencing these issues)

Create images: no go since Elastic Cloud only consumes our own stack packs which are standardized on the ES image AFAIK
Add --group-add 0 when running the image: no go since will involve changing orchestration code for Elastic deployments and visa's main issue is we're running using root privileges (group 0 is root's group, right?) please do call me out if I'm misunderstanding.
Bind-mount - will again require some major and complicated changes on how we orchestrate Elasticsearch through our entire 10000 server fleet.

pugnascotia · 2022-01-25T13:25:43Z

GID 0 is not the same as UID 0, and has no special privileges associated with it. It's become a common convention in container orchestration to default to GID 0 unless otherwise configured e.g. Kubernetes' runAsGroup option.

If none of these options work for you, can you suggest any alternatives that would, and we can discuss? Given that GID 0 isn't special, and that we own the orchestration, using --group-add seems a reasonable option?

jasontedor · 2022-01-25T13:30:11Z

@Kushmaro If a consumer of a Docker container (whether that be us orchestrating containers on behalf of users, or an end-user) chooses to deploy a container with --read-only, yet that container does need to write some data (whether it be user data, logs, dynamically loaded code, etc.), the consumer must also provide some means for that container to be able to write said data. That would then have to be through bind mounting, or tmpfs volumes, depending on the lifecycle of the data. (Note: The other options that @pugnascotia listed would be specific to managing plugins.)

That is, Elasticsearch can not get around the fact that it needs to write data somewhere. What Elasticsearch can do is make clear (via documentation) the volumes that it needs mounted to write data to (e.g., data, plugins, logs, etc.). But it still up to the end consumer to provide volume mounts for those. In this case, that would be our orchestration software.

mieciu added >bug needs:triage Requires assignment of a team area label Team:Delivery Meta label for Delivery team labels Feb 24, 2021

mieciu assigned pugnascotia Feb 24, 2021

pugnascotia added the :Core/Infra/Plugins Plugin API and infrastructure label Feb 24, 2021

elasticmachine added the Team:Core/Infra Meta label for core/infra team label Feb 24, 2021

pugnascotia removed the needs:triage Requires assignment of a team area label label Feb 24, 2021

mieciu mentioned this issue Jun 8, 2021

Tighten up write permissions in Docker image #70635

Merged

pugnascotia mentioned this issue Jan 14, 2022

Mention bind-mounting plugins dir in Docker docs #82622

Merged

pugnascotia closed this as completed in #82622 Jan 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Verify Elasticsearch Docker container writes on plugin installation #69533

Verify Elasticsearch Docker container writes on plugin installation #69533

mieciu commented Feb 24, 2021

elasticmachine commented Feb 24, 2021

mieciu commented Feb 24, 2021 •

edited

Loading

elasticmachine commented Feb 24, 2021

pugnascotia commented Feb 24, 2021

pugnascotia commented Feb 24, 2021

mark-vieira commented Feb 24, 2021

rjernst commented Feb 25, 2021

mieciu commented Feb 25, 2021

jasontedor commented Feb 25, 2021

pugnascotia commented Feb 25, 2021

rjernst commented Feb 25, 2021

mark-vieira commented Feb 25, 2021

jasontedor commented Feb 25, 2021

pugnascotia commented Mar 1, 2021

alpar-t commented Mar 1, 2021

mieciu commented Mar 2, 2021

pugnascotia commented Sep 10, 2021

mieciu commented Sep 13, 2021

rstatsinger commented Jan 13, 2022

mark-vieira commented Jan 13, 2022

pugnascotia commented Jan 14, 2022

rstatsinger commented Jan 14, 2022 via email

jasontedor commented Jan 24, 2022

mark-vieira commented Jan 24, 2022

jasontedor commented Jan 25, 2022

Kushmaro commented Jan 25, 2022

pugnascotia commented Jan 25, 2022

jasontedor commented Jan 25, 2022

Verify Elasticsearch Docker container writes on plugin installation #69533

Verify Elasticsearch Docker container writes on plugin installation #69533

Comments

mieciu commented Feb 24, 2021

elasticmachine commented Feb 24, 2021

mieciu commented Feb 24, 2021 • edited Loading

elasticmachine commented Feb 24, 2021

pugnascotia commented Feb 24, 2021

pugnascotia commented Feb 24, 2021

mark-vieira commented Feb 24, 2021

rjernst commented Feb 25, 2021

mieciu commented Feb 25, 2021

jasontedor commented Feb 25, 2021

pugnascotia commented Feb 25, 2021

rjernst commented Feb 25, 2021

mark-vieira commented Feb 25, 2021

jasontedor commented Feb 25, 2021

pugnascotia commented Mar 1, 2021

alpar-t commented Mar 1, 2021

mieciu commented Mar 2, 2021

pugnascotia commented Sep 10, 2021

mieciu commented Sep 13, 2021

rstatsinger commented Jan 13, 2022

mark-vieira commented Jan 13, 2022

pugnascotia commented Jan 14, 2022

rstatsinger commented Jan 14, 2022 via email

jasontedor commented Jan 24, 2022

mark-vieira commented Jan 24, 2022

jasontedor commented Jan 25, 2022

Kushmaro commented Jan 25, 2022

pugnascotia commented Jan 25, 2022

jasontedor commented Jan 25, 2022

mieciu commented Feb 24, 2021 •

edited

Loading