MutualTLS error="not Authorized"

[rekcaiot]

• I have searched the issues of this repository and believe that this is not a duplicate.

What happened?

Is the Chirpstack Gateweway Bridge tested for Mutual TLS based generic MQTT connections?
After many tests I can't get authorization without user/password to work.
I want to authorize the Bridge against our broker, which supports MTLS.
When using the certificates in the MQTT Explorer client application or mosquitto_sub the connection is established without any issue.
But using the same files in the Bridge didn't work.

Bridge configuration:

[integration.mqtt.auth]
type="generic"
[integration.mqtt.auth.generic]
server="ssl://placeholder:31377"
qos=2
clean_session=true
client_id="1231231231231231"

ca_cert="/srv/dev-disk-by-uuid-242aa529-f752-41a2-84ca-358448997e18/Transfer/bridge/truststore.pem"
tls_cert="/srv/dev-disk-by-uuid-242aa529-f752-41a2-84ca-358448997e18/Transfer/bridge/smc_cert.pem"
tls_key="/srv/dev-disk-by-uuid-242aa529-f752-41a2-84ca-358448997e18/Transfer/bridge/smc_key.pem"

smc_cert.pem includes all three certs of the chain.
smc_key.pem includes the decrypted key.

Bridge logs:

INFO[0000] starting ChirpStack Gateway Bridge            docs="https://www.chirpstack.io/gateway-bridge/" version=3.14.6
INFO[0000] backend/semtechudp: starting gateway udp listener  addr="127.0.0.1:1701"
DEBU[0000] backend/semtechudp: cleanup gateway registry
INFO[0000] integration/mqtt: gateway id provided by authentication method  gateway_id=1231231231231231
INFO[0000] integration/mqtt: setting last will and testament  gateway_id=1231231231231231 topic=chirpstack/gateway/1231231231231231/state/conn
DEBU[0000] [client]   Connect()                          module=mqtt
DEBU[0000] [store]    memorystore initialized            module=mqtt
DEBU[0000] [client]   about to write new connect msg     module=mqtt
DEBU[0000] [client]   socket connected to broker         module=mqtt
DEBU[0000] [client]   Using MQTT 3.1.1 protocol          module=mqtt
DEBU[0000] [net]      connect started                    module=mqtt
DEBU[0000] [net]      received connack                   module=mqtt
ERRO[0000] [client]   Connecting to ssl://placeholder:31377 CONNACK was not CONN_ACCEPTED, but rather Connection Refused: Not Authorised  module=mqtt
ERRO[0000] [client]   Failed to connect to a broker      module=mqtt
DEBU[0000] [store]    memorystore closed                 module=mqtt
ERRO[0000] integration/mqtt: connection error            error="not Authorized"`

Broker logs:

2023-04-14 11:52:19,688 [client-dispatcher-1-1] TRACE o.t.m.b.s.a.p.SslMqttClientAuthProvider - [1231231231231231] Authenticating client with SSL credentials
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:556)
at org.thingsboard.mqtt.broker.service.auth.providers.SslMqttClientAuthProvider.authWithSSLCredentials(SslMqttClientAuthProvider.java:76)
at org.thingsboard.mqtt.broker.service.auth.providers.SslMqttClientAuthProvider.authenticate(SslMqttClientAuthProvider.java:59)
at org.thingsboard.mqtt.broker.service.auth.DefaultAuthenticationService.authenticate(DefaultAuthenticationService.java:80)
at org.thingsboard.mqtt.broker.service.auth.DefaultAuthenticationService.authenticate(DefaultAuthenticationService.java:57)
at org.thingsboard.mqtt.broker.actors.client.service.ActorProcessorImpl.authenticateClient(ActorProcessorImpl.java:174)
at org.thingsboard.mqtt.broker.actors.client.service.ActorProcessorImpl.onInit(ActorProcessorImpl.java:68)
at org.thingsboard.mqtt.broker.actors.client.ClientActor.doProcess(ClientActor.java:121)
at org.thingsboard.mqtt.broker.actors.service.ContextAwareActor.process(ContextAwareActor.java:50)
at org.thingsboard.mqtt.broker.actors.TbActorMailbox.processMailbox(TbActorMailbox.java:141)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
2023-04-14 11:52:19,688 [client-dispatcher-1-1] ERROR o.t.m.b.s.a.p.SslMqttClientAuthProvider - Failed to authenticate client with SSL credentials! No SSL credentials were found!
2023-04-14 11:52:19,688 [client-dispatcher-1-1] WARN o.t.m.b.a.c.s.ActorProcessorImpl - [1231231231231231] Connection is not established due to: CONNECTION_REFUSED_NOT_AUTHORIZED

When testing with openssl s_server I can see that there might be something missing in the tls handshake:

ACCEPT
SSL_accept:before/accept initialization
SSL_accept:SSLv3 read client hello A
SSL_accept:SSLv3 write server hello A
SSL_accept:SSLv3 write certificate A
SSL_accept:SSLv3 write key exchange A
SSL_accept:SSLv3 write certificate request A
SSL_accept:SSLv3 flush data
SSL_accept:failed in SSLv3 read client certificate A
ERROR
shutting down SSL
CONNECTION CLOSED
ACCEPT