Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Security TLS configuration fails #690

Open
ericklife28 opened this issue Dec 23, 2023 · 6 comments
Open

Security TLS configuration fails #690

ericklife28 opened this issue Dec 23, 2023 · 6 comments
Labels
bug Something isn't working

Comments

@ericklife28
Copy link

ericklife28 commented Dec 23, 2023 •
edited
Loading

What is the bug?

When I enable tls, I have decided to use self signed certificates using the opensearch cluster CA for the transport layer. However I want to create a custom certificate for the http layer. I am using this configuration:

`security:

config:  # Everything related to the securityconfig

  securityConfigSecret:

    name:  ${security_config_secret}

  adminCredentialsSecret:

    name:  ${admin_credentials_secret}

tls:

  transport:

    generate: true

    perNode: true

  http:

    generate: false # --> here I disable certificate generation 

    secret:

      name: ${tls_rest_secret_name} # Name of the secret that contains the provided certificate`

However, the opensearch-cluster-bootstrap fails with this error:

init-sysctl vm.max_map_count = 262144
opensearch [2023-12-23T13:50:23,243][WARN ][stderr ] [opensearch-cluster-bootstrap-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
opensearch [2023-12-23T13:50:23,244][WARN ][stderr ] [opensearch-cluster-bootstrap-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
opensearch [2023-12-23T13:50:23,244][WARN ][stderr ] [opensearch-cluster-bootstrap-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
opensearch [2023-12-23T13:50:23,254][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-bootstrap-0] SSL dual mode is disabled
opensearch [2023-12-23T13:50:23,254][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-bootstrap-0] OpenSearch Config path is /usr/share/opensearch/config
opensearch [2023-12-23T13:50:23,501][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-bootstrap-0] JVM supports TLSv1.3
opensearch [2023-12-23T13:50:23,503][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-bootstrap-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
opensearch [2023-12-23T13:50:23,555][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-bootstrap-0] uncaught exception in thread [main]
opensearch org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.3.0.jar:2.3.0]
opensearch at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.3.0.jar:2.3.0]
opensearch Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.node.Node.(Node.java:420) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.node.Node.(Node.java:347) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0]
opensearch ... 6 more
opensearch Caused by: java.lang.reflect.InvocationTargetException
opensearch at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
Stream closed EOF for opensearch-cluster/opensearch-cluster-bootstrap-0 (init)
Stream closed EOF for opensearch-cluster/opensearch-cluster-bootstrap-0 (init-sysctl)
opensearch at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
opensearch at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
opensearch at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
opensearch at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
opensearch at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.node.Node.(Node.java:420) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.node.Node.(Node.java:347) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
opensearch uncaught exception in thread [main]
opensearch at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0]
opensearch ... 6 more
opensearch Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.
opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419) ~[?:?]
opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:255) ~[?:?]
opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:176) ~[?:?]
opensearch at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
opensearch at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:262) ~[?:?]
opensearch at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
opensearch at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
opensearch at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
opensearch at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
opensearch at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
opensearch at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.node.Node.(Node.java:420) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.node.Node.(Node.java:347) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0]
opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0]
opensearch ... 6 more
opensearch java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.]
opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419)
opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:255)
opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:176)
opensearch at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218)
opensearch at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:262)
opensearch at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
opensearch at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
opensearch at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
opensearch at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
opensearch at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
opensearch at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
opensearch at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730)
opensearch at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532)
opensearch at org.opensearch.plugins.PluginsService.(PluginsService.java:195)
opensearch at org.opensearch.node.Node.(Node.java:420)
opensearch at org.opensearch.node.Node.(Node.java:347)
opensearch at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242)
opensearch at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
opensearch at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
opensearch at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
opensearch at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
opensearch at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
opensearch at org.opensearch.cli.Command.main(Command.java:101)
opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
opensearch For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log
opensearch Killing performance analyzer process 34
opensearch OpenSearch exited with code 1
opensearch Performance analyzer exited with code 143
Stream closed EOF for opensearch-cluster/opensearch-cluster-bootstrap-0 (opensearch)

How can one reproduce the bug?

Using the security setup above

What is the expected behavior?

The startup process should run successfully

Do you have any additional context?

I am using the opensearch-cluster helm chart to apply this configuration. The values above correspond to the values.yaml used.

According to the documentation, It supposed that the tls configuraton for transport and http could be configured in an independent way. However, I wonder either if once you disable the generation for any of them we need to use our own CA for both of them. or if there is a missing setup that is not documented.

On the other hand. If I set the http > generate parameter to true

http:
generate: true

I don't get any error, and the cluster start up process is executed successfully.

I appreciate your help.

Cheers.
@ericklife28 ericklife28 added the bug Something isn't working label Dec 23, 2023
@ericklife28
Copy link
Author

Hi, looking at the opensearch cluster chart. I noticed that this line could be the reason of the problem

opensearch-k8s-operator/charts/opensearch-cluster/templates/opensearch-cluster-cr.yaml

Line 284 in f88d40d

{{- if .Values.opensearchCluster.security.tls.http.generate }}

As you can see there the transport.generate value is dependen of the http.generate value and it shouldn't be. Is there someone working on this?

@ericklife28
Copy link
Author

Hi I made the change above on the chart locally and it seems to work. However, I get a new error on the opensearch-cluster-securityconfig-update pod:

** This tool will be deprecated in the next major release of OpenSearch **
** opensearch-project/security#1755 **

Will connect to opensearch-cluster.opensearch-cluster.svc.cluster.local:9200 ... done
ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetSecurity Admin v7
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:947)
at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:288)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:356)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:547)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 24 more

Is there anyone working on this?. Let me know if you need more info to continue.

Cheers.

@cyxou
Copy link

cyxou commented Mar 26, 2024

@ericklife28 have you managed to find a solution? Facing the same issue with Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target error ((

@Siradjedd
Copy link

@cyxou any updates ?

@cyxou
Copy link

cyxou commented Apr 12, 2024

@cyxou any updates ?

Nope((

@getsaurabh02 getsaurabh02 moved this from 🆕 New to Backlog in Engineering Effectiveness Board Jul 18, 2024
@CoderYellow
Copy link

CoderYellow commented Dec 7, 2024 •
edited
Loading

Hi @saketmht @stefan-heilig-mw i think the root cause is here, you are using the node transport CA

opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/securityconfig.go

Line 30 in 2e78967

caCert = "/certs/ca.crt"

opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/securityconfig.go

Line 161 in 2e78967

fmt.Sprintf(ApplyAllYmlCmdTmpl, caCert, adminCert, adminKey, securityconfigPath, clusterHostName, securityConfigPort)

when user have a custom CA for http under /usr/share/opensearch/config/tls-http/ca.crt we should honor and trust this cert

as per this doc

opensearch-k8s-operator/charts/opensearch-operator/files/opensearch.opster.io_opensearchclusters.yaml

Line 5355 in 2e78967

description: Optional, name of a TLS secret that contains

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Engineering Effectiveness Board
Status: 📦 Backlog
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cyxou @ericklife28 @CoderYellow @Siradjedd