Plugin install fail with Operation not permitted on OpenShift

[pebrc]

Bug Report

What did you do?
When calling elasticsearch plugin commands the following exception is raised, when running on top of OpenShift:

 

sh-4.2$ id

uid=1000560000 gid=0(root) groups=0(root),1000560000

 

sh-4.2$ ls -l
total 548
-rw-r--r--.  1 elasticsearch root        13675 Aug 19 20:18 LICENSE.txt
-rw-r--r--.  1 elasticsearch root       502598 Aug 19 20:23 NOTICE.txt
-rw-r--r--.  1 elasticsearch root         8500 Aug 19 20:17 README.textile
drwxrwsrwx.  2 root          1000560000   4096 Sep 23 18:22 bin
drwxrwsrwx.  5 root          1000560000   4096 Sep 25 15:18 config
drwxrwxrwx.  3 root          root         4096 Sep 17 17:24 data
drwxr-xr-x.  1 elasticsearch root           17 Aug 19 20:23 jdk
drwxr-xr-x.  3 elasticsearch root         4096 Aug 19 20:23 lib
drwxrwsrwx.  2 root          1000560000   4096 Sep 25 03:49 logs
drwxr-xr-x. 33 elasticsearch root         4096 Aug 19 20:23 modules
drwxrwsrwx.  2 root          1000560000   4096 Sep 25 15:18 plugins

sh-4.2$ ps -ef

UID         PID   PPID  C STIME TTY          TIME CMD

1000560+      1      0  3 Sep23 ?        01:39:47 /usr/share/elasticsearch/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSIni

1000560+    217      1  0 Sep23 ?        00:00:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

1000560+ 312661      0  0 15:24 ?        00:00:00 /bin/sh

1000560+ 312710 312661  0 15:24 ?        00:00:00 ps -ef

 

 

sh-4.2$ bin/elasticsearch-plugin install  <PLUGIN URL>

-> Downloading <PLUGIN URL>

[=================================================] 100%??
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission es.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

 

Continue with installation? [y/N]y

Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/repository-s3: Operation not permitted
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setOwners(UnixFileAttributeViews.java:268)
        at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setOwner(UnixFileAttributeViews.java:290)
        at org.elasticsearch.plugins.InstallPluginCommand.setOwnerGroup(InstallPluginCommand.java:928)
        at org.elasticsearch.plugins.InstallPluginCommand.installConfig(InstallPluginCommand.java:901)
        at org.elasticsearch.plugins.InstallPluginCommand.installPluginSupportFiles(InstallPluginCommand.java:835)
        at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:816)
        at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:785)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:231)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
sh-4.2$

What did you expect to see?

What did you see instead? Under which circumstances?

Environment

• ECK version:
  0.9.0

• Kubernetes information:

  • Kubernetes distribution: Openshift

  for each of them please give us the version you are using

$ kubectl version

• Resource definition:

if relevant insert the resource definition

• Logs:

insert operator logs or any relevant message to the issue here

[pebrc]

cc @agup006

[barkbay]

It is not possible to install a plugin using an unprivileged user. Any user using a security context and/or some PSP to secure a K8S cluster is likely to hit this issue (not specific to Openshift).

The root cause is that the plugin installation attempts to change the owner on a directory, which is something only root can do:
https://github.com/elastic/elasticsearch/blob/a8dae10edb67895b9ece741ec484d0f62fa10236/distribution/tools/plugin-cli/src/main/java/org/elasticsearch/plugins/InstallPluginCommand.java#L922-L928

I'm not sure there's much we can do on the ECK side but recommend to build a custom image to install plugins. We may also open an issue in the Elasticsearch repo. to discuss this issue.

[pebrc]

Closing this for now, given the assessment by @barkbay