v6.4.0: Cannot connect to local Greengrass broker (moquette) #57

jaxer · 2024-03-05T12:01:45Z

Here is an agent config:

devices: devices

modules:
    - mqtt-relay:
          server: host.docker.internal
          port: 8883
          clientId: alex-printer-8
          certificateAuthority: certificates/certificateAuthority.pem
          pemCertificate: certificates/certificate.pem
          pemPrivateKey: certificates/privateKey.pem
#          Uncommenting this does not change much
#          allowUntrustedCertificates: true

observationBufferSize: 150000
assetBufferSize: 1000

# Sets whether the Agent buffers are durable and retain state after restart
durable: true

defaultVersion: 2.2

Starting agent with docker compose up compose config:

version: "3.8"

services:
    agent:
        image: trakhound/mtconnect.net-agent:6.0.9-beta-alpine-3.18-amd64
        volumes:
            - ./agent.config.yaml:/app/agent.config.yaml
            - ./devices:/app/devices
            - ./certificates:/app/certificates

Output on console:

⋊> docker compose up                                                                                                                   
[+] Running 1/0
 ⠿ Container mtconnectnet-agent-1  Created                                                                                                                        0.0s
Attaching to mtconnectnet-agent-1
mtconnectnet-agent-1  | --------------------
mtconnectnet-agent-1  | Copyright 2023 TrakHound Inc., All Rights Reserved
mtconnectnet-agent-1  | MTConnect.NET Agent : Version 6.0.9.0
mtconnectnet-agent-1  | --------------------
mtconnectnet-agent-1  | This application is licensed under the MIT License (https://choosealicense.com/licenses/mit/)
mtconnectnet-agent-1  | Source code available at Github.com (https://github.com/TrakHound/MTConnect.NET)
mtconnectnet-agent-1  | --------------------
mtconnectnet-agent-1  | 2024-03-05 11:50:03.2538|INFO|application|Configuration File Read Successfully from: /app/agent.config.yaml
mtconnectnet-agent-1  | 2024-03-05 11:50:03.3169|INFO|agent|Loading Observations from File Buffer...
mtconnectnet-agent-1  | 2024-03-05 11:50:03.3461|INFO|agent|15 Observations Loaded from File Buffer in (0.025s)
mtconnectnet-agent-1  | 2024-03-05 11:50:03.3476|INFO|agent|Loading Assets from File Buffer...
mtconnectnet-agent-1  | 2024-03-05 11:50:03.6038|INFO|application|Module Loaded : MQTT Relay
mtconnectnet-agent-1  | 2024-03-05 11:50:03.7549|INFO|agent|Device (M12346) Read From File : /app/devices/test.xml
mtconnectnet-agent-1  | 2024-03-05 11:50:04.1271|WARN|modules.mqtt-relay|MQTT Relay Connection Error : Authentication failed, see inner exception.
mtconnectnet-agent-1  | 2024-03-05 11:50:04.1406|INFO|modules.mqtt-relay|MQTT Relay Disconnected from External Broker (host.docker.internal:8883)
^CGracefully stopping... (press Ctrl+C again to force)

Logs from greengrass:

2024-03-05T11:50:04.109Z [ERROR] (nioEventLoopGroup-5-4) io.moquette.broker.NewNettyMQTTHandler: Unexpected exception while processing MQTT message. Closing Netty channel. CId=null. {}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.moquette.broker.metrics.BytesMetricsHandler.channelRead(BytesMetricsHandler.java:51)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:347)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:303)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:294)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:390)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1076)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1063)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1010)
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1548)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1394)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDe
coder.java:449)
	... 25 more

2024-03-05T11:50:04.110Z [INFO] (nioEventLoopGroup-5-4) io.moquette.b
roker.metrics.MQTTMessageLogger: Channel Inactive. {}

I am guessing it's something about MQTT library configuration, as I can connect to same broker with same certificates using MQTT Explorer:

In which case, Greengrass logs look like:

2024-03-05T11:56:37.269Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B CONNECT <null>. {}
2024-03-05T11:56:37.877Z [INFO] (nioEventLoopGroup-5-1) com.aws.greengrass.mqtt.moquette.ClientDeviceAuthorizer: Successfully authenticated client device. {clientId=alex-printer-8, sessionId=2bfac112-3e14-4e25-9996-eeb5802dca9b}
2024-03-05T11:56:37.893Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <alex-printer-8> to topics [MqttTopicSubscription[topicFilter=#, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
2024-03-05T11:56:37.894Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <alex-printer-8> to topics [MqttTopicSubscription[topicFilter=$SYS/#, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
2024-03-05T11:56:37.895Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <alex-printer-8> to topics [MqttTopicSubscription[topicFilter=$aws/#, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}
2024-03-05T11:57:20.889Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: C->B DISCONNECT <alex-printer-8>. {}
2024-03-05T11:57:20.891Z [INFO] (nioEventLoopGroup-5-1) io.moquette.broker.metrics.MQTTMessageLogger: Channel Inactive. {}

The text was updated successfully, but these errors were encountered:

jaxer · 2024-03-05T12:05:29Z

I think there is not much about greengrass configuration, I am using mostly default settings.

Please let me know if I can provide more info or can try it out on some test build of a docker image.

PatrickRitchie · 2024-03-05T16:43:26Z

Thanks for the information. Are you able to connect using MQTT Explorer from inside a Docker container? I'm wondering if that might be causing an issue. Have you installed the certificate in the Docker container? I'm not an expert on certificates but I get similar errors if I don't have the client certificate installed when running on Windows.

jaxer · 2024-03-06T11:26:37Z

Thanks for quick replys. Thats an interesting tought. I am also not an expert on certificates.
I think AWS root certificates should be trusted by modern systems by default. But I still tried to do somthing similar to this (took AWS IoT root CA for that) with no noticable changes.

Here is what I tried after that (with same setup as in description):

docker compose run --entrypoint sh agent
apk add mosquitto-clients
mosquitto_sub -t "#" -h host.docker.internal -p 8883 --cafile certificates/certificateAuthority.pem --cert certificates/certificate.pem --key certificates/privateKey.pem -i alex-printer-8

That connected sucessfully and greengrass logs are:

2024-03-06T10:59:50.726Z [INFO] (nioEventLoopGroup-5-3) io.moquette.broker.metrics.MQTTMessageLogger: C->B CONNECT <null>. {}
2024-03-06T10:59:51.308Z [INFO] (nioEventLoopGroup-5-3) com.aws.greengrass.mqtt.moquette.ClientDeviceAuthorizer: Successfully authenticated client device. {clientId=alex-printer-8, sessionId=20447946-fe71-49cf-8069-9732dcb015f1}
2024-03-06T10:59:51.311Z [INFO] (nioEventLoopGroup-5-3) io.moquette.broker.metrics.MQTTMessageLogger: C->B SUBSCRIBE <alex-printer-8> to topics [MqttTopicSubscription[topicFilter=#, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]. {}

Running dotnet agent.dll right after that on same instance of container produces exactly same result as in description.
So I am guessing that it's not related to system certificates chain.

PatrickRitchie · 2024-03-27T04:56:33Z

I just created a new release (v6.2.0) that may fix some of the issues you were having. I made a number of changes to configuring TLS for both HTTP and MQTT. Note that the configuration file format for setting certificate paths has changed to the below:

- mqtt-relay:
    server: localhost
    port: 8883
    clientId: mtconnect-test # Set the ClientId to the AWS Thing ID
    tls:
      pem:
        certificateAuthority: certs/AmazonRootCA1.pem
        certificatePath: certs/2316549874654321654984984158961634984794-certificate.pem.crt
        privateKeyPath: certs/2316549874654321654984984158961634984794-private.pem.key
    documentFormat: json-cppagent
    currentInterval: 5000
    sampleInterval: 500

There may be some work left to do so let me know if you are still having issues with it.

jaxer · 2024-05-14T15:38:33Z

Sorry for taking it so long to re-test.

Unfortunately the issue still remains for me.
Same exception with latest 6.4.0:

2024-05-14T15:30:06.497Z [ERROR] (nioEventLoopGroup-3-25) io.moquette.broker.NewNettyMQTTHandler: Unexpected exception while processing MQTT message. Closing Netty channel. CId=null. {}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.moquette.broker.metrics.BytesMetricsHandler.channelRead(BytesMetricsHandler.java:51)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:347)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:303)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:294)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:390)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1076)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1063)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1010)
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	... 25 more

I could help with reproducing the issue by putting Greengrass gateway into public and giving you temporary certificates to connect with. Let me know if that would help. Our instance of Greegrass gateway is nothing special, it runs default Moquette broker and does not do any magic with root certs (using default AWS root certs).

Would be great to have this fixed as that will allow us to use MTConnect.NET agent in emulators (linux/docker based) simplifying MTConnect integration.

I think issue only present on non-windows platforms.

jaxer · 2024-11-26T16:06:28Z

#73 works for my needs so far, I'll close this.

jaxer changed the title ~~Beta branch: Cannot connect to local Greengrass broker (mosquitto)~~ Beta branch: Cannot connect to local Greengrass broker (moquette) Mar 5, 2024

jaxer changed the title ~~Beta branch: Cannot connect to local Greengrass broker (moquette)~~ v6.4.0: Cannot connect to local Greengrass broker (moquette) May 15, 2024

PatrickRitchie added this to MTConnect.NET-Issues May 16, 2024

PatrickRitchie moved this to Todo in MTConnect.NET-Issues May 16, 2024

PatrickRitchie mentioned this issue Jun 19, 2024

mqtt-relay cannot connect to broker with TLS enabled and PEM certificates #66

Open

Pranay265 mentioned this issue Sep 26, 2024

Add Option to Skip CA Certificate Validation in MTConnect Agent Configuration #73

Merged

jaxer closed this as completed Nov 26, 2024

github-project-automation bot moved this from Todo to Done in MTConnect.NET-Issues Nov 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v6.4.0: Cannot connect to local Greengrass broker (moquette) #57

v6.4.0: Cannot connect to local Greengrass broker (moquette) #57

jaxer commented Mar 5, 2024

jaxer commented Mar 5, 2024

PatrickRitchie commented Mar 5, 2024

jaxer commented Mar 6, 2024

PatrickRitchie commented Mar 27, 2024

jaxer commented May 14, 2024 •

edited

Loading

jaxer commented Nov 26, 2024

v6.4.0: Cannot connect to local Greengrass broker (moquette) #57

v6.4.0: Cannot connect to local Greengrass broker (moquette) #57

Comments

jaxer commented Mar 5, 2024

jaxer commented Mar 5, 2024

PatrickRitchie commented Mar 5, 2024

jaxer commented Mar 6, 2024

PatrickRitchie commented Mar 27, 2024

jaxer commented May 14, 2024 • edited Loading

jaxer commented Nov 26, 2024

jaxer commented May 14, 2024 •

edited

Loading