Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TLS not working in OCP cluster with self signed cert. #5

Closed
whitingjr opened this issue Mar 24, 2023 · 9 comments
Closed

TLS not working in OCP cluster with self signed cert. #5

whitingjr opened this issue Mar 24, 2023 · 9 comments
Assignees
@barreiro
Labels
bug Something isn't working

Comments

@whitingjr
Copy link

A deployed Horreum replica is reporting this error in the log.

2023-03-23 19:34:29,728 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-1) HTTP Request to /api/user/administrators failed, error id: 086eb16d-d056-4b99-a1e0-6b8ff8122923-1: java.util.concurrent.CompletionException: javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
712at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
713at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)

Caused by: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
1059at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:130)
1060... 25 more
@whitingjr whitingjr changed the title TLS not working in OCP cluster with self signed certs. TLS not working in OCP cluster with self signed cert. Mar 24, 2023
@johnaohara
Copy link
Member

This issue is caused by the Keycloak cert being signed by a self cert CA, which has not been imported into the jdk cacerts truststore in the horreum pod;

2023-03-23 19:34:29,728 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-1) HTTP Request to /api/user/administrators failed, error id: 086eb16d-d056-4b99-a1e0-6b8ff8122923-1: java.util.concurrent.CompletionException: javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
	at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)
	at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:970)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
	at io.vertx.core.Future.lambda$toCompletionStage$3(Future.java:386)
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153)
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
	at io.hyperfoil.tools.horreum.svc.UserServiceImpl.wrapException(UserServiceImpl.java:372)
	at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:986)
	... 14 more
Caused by: javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
	at org.jboss.resteasy.reactive.client.handlers.ClientSendRequestHandler$3.accept(ClientSendRequestHandler.java:217)
	at org.jboss.resteasy.reactive.client.handlers.ClientSendRequestHandler$3.accept(ClientSendRequestHandler.java:209)
	at io.smallrye.context.impl.wrappers.SlowContextualConsumer.accept(SlowContextualConsumer.java:21)
	at io.smallrye.mutiny.helpers.UniCallbackSubscriber.onFailure(UniCallbackSubscriber.java:62)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at org.jboss.resteasy.reactive.client.AsyncResultUni.lambda$subscribe$1(AsyncResultUni.java:37)
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153)
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69)
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:86)
	at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:163)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:66)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230)
	at io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23)
	at io.vertx.core.http.impl.HttpClientImpl.lambda$doRequest$6(HttpClientImpl.java:658)
	at io.vertx.core.net.impl.pool.Endpoint.lambda$getConnection$0(Endpoint.java:52)
	at io.vertx.core.http.impl.SharedClientHttpStreamEndpoint$Request.handle(SharedClientHttpStreamEndpoint.java:162)
	at io.vertx.core.http.impl.SharedClientHttpStreamEndpoint$Request.handle(SharedClientHttpStreamEndpoint.java:123)
	at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:55)
	at io.vertx.core.impl.ContextBase.emit(ContextBase.java:239)
	at io.vertx.core.net.impl.pool.SimpleConnectionPool$ConnectFailed$1.run(SimpleConnectionPool.java:384)
	at io.vertx.core.net.impl.pool.CombinerExecutor.submit(CombinerExecutor.java:50)
	at io.vertx.core.net.impl.pool.SimpleConnectionPool.execute(SimpleConnectionPool.java:245)
	at io.vertx.core.net.impl.pool.SimpleConnectionPool.lambda$connect$2(SimpleConnectionPool.java:259)
	at io.vertx.core.http.impl.SharedClientHttpStreamEndpoint.lambda$connect$2(SharedClientHttpStreamEndpoint.java:104)
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230)
	at io.vertx.core.impl.future.Composition$1.onFailure(Composition.java:66)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75)
	at io.vertx.core.impl.future.FailedFuture.addListener(FailedFuture.java:98)
	at io.vertx.core.impl.future.Composition.onFailure(Composition.java:55)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230)
	at io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23)
	at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:55)
	at io.vertx.core.impl.ContextBase.emit(ContextBase.java:239)
	at io.vertx.core.net.impl.NetClientImpl.failed(NetClientImpl.java:320)
	at io.vertx.core.net.impl.NetClientImpl.lambda$connectInternal2$5(NetClientImpl.java:288)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
	at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:109)
	at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:132)
	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:400)
	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)
	at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368)
	at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1260)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1241)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	... 4 more
Caused by: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
	at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:130)
	... 25 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:641)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:460)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995)
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
	... 20 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:619)
	... 31 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
	... 37 more

@johnaohara johnaohara added the bug Something isn't working label Mar 24, 2023
@rafaeltuelho
Copy link

rafaeltuelho commented Mar 24, 2023 •
edited
Loading

adding some additional info related to this...

I see the Operator [1] sets the quarkus [2] QUARKUS_OIDC_TLS_VERIFICATION to none which means it should not verify the TLS certificate used by the OIDC server (Keycloak in this case). Maybe it is worth investigating if this issue is really related to the communication between horreum backend and keycloak through the Quarkus OIDC extension.

[1]

horreum-operator/controllers/app.go

Lines 35 to 40 in 104e3a4

{
// TODO: it's not possible to set up custom CA for OIDC
// https://github.com/quarkusio/quarkus/issues/18002
Name: "QUARKUS_OIDC_TLS_VERIFICATION",
Value: "none",
},

[2] Quarkus OIDC extension reference: https://quarkus.io/guides/security-openid-connect

@johnaohara
Copy link
Member

The error comes from a request to rest endpoint /api/user/administrators. The exception from a backend call is wrapped here: https://github.com/Hyperfoil/Horreum/blob/0.7/horreum-backend/src/main/java/io/hyperfoil/tools/horreum/svc/UserServiceImpl.java#L372
when an exception occurs from this call: https://github.com/Hyperfoil/Horreum/blob/0.7/horreum-backend/src/main/java/io/hyperfoil/tools/horreum/svc/UserServiceImpl.java#L452

the client call to keycloak is throwing the exception: https://github.com/Hyperfoil/Horreum/blob/0.7/horreum-backend/src/main/java/io/hyperfoil/tools/horreum/svc/UserServiceImpl.java#L448

it might be that QUARKUS_OIDC_TLS_VERIFICATION=none is no longer working as expected, or there is something else broken

@rafaeltuelho
Copy link

Quakus seems to have a global runtime property for TLS called quarkus.tls.trust-all. Maybe it is worth a try.

@dustinblack
Copy link
Contributor

dustinblack commented Aug 16, 2023 •
edited
Loading

I'm running into this problem when trying to do anything in the Administration tab of the Horreum UI after doing an initial install of the operator 0.7.9.

I attempted as a test to set QUARKUS_TLS_TRUST_ALL = true as and environment variable for the horreum-app pod, but I got this:

2023-08-16 13:27:18,627 WARN  [io.qua.run.con.ConfigRecorder] (main) Build time property cannot be changed at runtime:
 - quarkus.tls.trust-all is set to 'true' but it is build time fixed to 'false'. Did you change the property quarkus.tls.trust-all aft
er building the application?

@dustinblack
Copy link
Contributor

@barreiro This may be a useful relevant reference: https://developers.redhat.com/blog/2017/11/22/dynamically-creating-java-keystores-openshift

@johnaohara
Copy link
Member

It looks like we should be able to register a truststore for the OIDC client: quarkusio/quarkus#18012

I haven't looked at the mechanics of how to do it though

@barreiro barreiro mentioned this issue Aug 24, 2023
Merged
@barreiro
Copy link
Collaborator

In this case, the problem is not the OIDC client.

Horreum uses Keycloak admin client directly. A resteasy client is not provided, so a new one is created with defaults.

We would have to provide a resteasy client with the right trust store there.

In the scope of the operator, an alternative would be to add the service-ca certificate to the system-wide trust store, so that all SSL connections to other services are trusted. See #17.

@barreiro
Copy link
Collaborator

barreiro commented Sep 4, 2023

fixed in #17

@barreiro barreiro closed this as completed Sep 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@barreiro barreiro

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@rafaeltuelho @whitingjr @barreiro @johnaohara @dustinblack