[BUG] opensearch (sec plugin) crashes after upgrading to 2.18.0 with Unable to read the file root-ca.pem

[proligde]

What is the bug?

Since quite a while I have a, basically default, installation of opensearch running, using the official charts from https://opensearch-project.github.io/helm-charts/.

Whenever a new chart version came out, I updated the deployment accordingly. This worked flawlessly for months until now I upgraded to chart version 2.27.0, containing opensearch 2.18.0

Now the pod crashloops with essentially these errors (see full stack trace below)

[2024-11-20T17:09:29,387][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-0] uncaught exception in thread [main]
[...]
Caused by: org.opensearch.OpenSearchException: Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions

Just turning the statefulset's image back to 2.17.0 makes it work again.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. install a default opensearch setup (activated security, demo configuration) using a helm chart < 2.27.0
2. switch chart to 2.27.0

What is the expected behavior?
The upgrade works like it did before

What is your host/environment?

• Host OS: Ubuntu 22.04 / Kubernetes
• 2.18.0 (Helm Chart 2.27.0)

** Full stack trace **

Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.4.0-200-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.18.0
Detected OpenSearch Security Version: 2.18.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Nov 20, 2024 5:09:28 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-11-20T17:09:28,411][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] version[2.18.0], pid[1], build[tar/99a9a81da366173b0c2b963b26ea92e15ef34547/2024-10-31T19:08:39.157471098Z], OS[Linux/5.4.0-200-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.5/21.0.5+11-LTS]
[2024-11-20T17:09:28,413][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-11-20T17:09:28,413][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-7151848204992045269, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx30g, -Xms30g, -XX:MaxDirectMemorySize=16106127360, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-11-20T17:09:28,556][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [opensearch-cluster-master-0] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
[2024-11-20T17:09:29,222][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled
[2024-11-20T17:09:29,222][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] OpenSearch Config path is /usr/share/opensearch/config
[2024-11-20T17:09:29,299][WARN ][o.o.s.s.SslSettingsManager] [opensearch-cluster-master-0] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of {}
java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLContext
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593) ~[?:?]
at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
at java.base/java.lang.Class.forName(Class.java:534) ~[?:?]
at java.base/java.lang.Class.forName(Class.java:513) ~[?:?]
at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:95) ~[netty-handler-4.1.114.Final.jar:4.1.114.Final]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin$4.run(OpenSearchSecuritySSLPlugin.java:218) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:213) [opensearch-security-2.18.0.0.jar:2.18.0.0]
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) [opensearch-security-2.18.0.0.jar:2.18.0.0]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.18.0.jar:2.18.0]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.18.0.jar:2.18.0]


[2024-11-20T17:09:29,387][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-0] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.18.0.jar:2.18.0]
uncaught exception in thread [main]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.18.0.jar:2.18.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
Caused by: org.opensearch.OpenSearchException: Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions
at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165) ~[?:?]
at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
... 6 more
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions]
at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165)
at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85)
at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137)
at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93)
at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80)
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249)
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796)
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744)
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545)
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197)
at org.opensearch.node.Node.<init>(Node.java:523)
at org.opensearch.node.Node.<init>(Node.java:450)
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

[cwperks]

Looks like this is due to the bug fix in #4793

The demo installation script thinks security is already configured (meaning certs pre-installed) so it aborts w/o copying the demo certs to the config/ directory. FYi @prudhvigodithi

[cwperks]

Fixed in the helm-charts repo: opensearch-project/helm-charts#618