jmxfetch always sets JMXConnector.CREDENTIALS, leading to NullPointerException with jboss_wildfly integration on JBoss 7.0.7

[deejgregor]

Please reference support case #1778852.

When we followed the instructions on the JBoss/WildFly integration page to collect metrics from one of our JBoss instances, we get a NullPointerException (full exception included later):

java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: java.lang.NullPointerException

We are seeing this with Datadog Agent v7.55.3, JMX Fetch 0.49.1, JBoss 7.0.x, and using the service:jmx:remote+http://<HOST>:<PORT> form of the jmx_url. We aren't setting a user or password in our JMX configuration.

Regardless of whether a user and password are set in the config file, jmxfetch always adds JMXConnector.CREDENTIALS, and this seems to be the cause of our NPE:

        environment.put(JMXConnector.CREDENTIALS, new String[] { user, password });

-- link

Changing that to this fixes the problem:

        if (user !=  null && password != null) {
            environment.put(JMXConnector.CREDENTIALS, new String[] { user, password });
        }

I'm not sure if there are valid cases where only one of user or password would be set, so maybe that should be a ||? I'm not deeply familiar JMX, so I'm not sure what the best solution should be here.

Some notes on JMX URLs

Note: as listed in the documentation for Wildfly 9 and older, we first used set jmx_url to service:jmx:http-remoting-jmx://<HOST>:<PORT>. When were were testing jmxterm, we got the warning WARN: The protocol 'http-remoting-jmx' is deprecated, instead you should use 'remote+http'. To avoid the warning, we so we switched to the newer form. We got the same NPE with both.

Also, we did need to use the special JBoss http-remoting-jmx/'remote+http schemes to connect to JMX on the server, so we had to make sure the JBoss CLI client library JAR was on the jmxfetch (and jmxterm) classpath (/opt/jboss/v7.0/bin/client/jboss-cli-client.jar in our case).

Full exception

2024-10-28 13:38:20 EDT | JMX | WARN | App | Could not initialize instance: conf:
java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: java.lang.NullPointerException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.datadog.jmxfetch.App.processRecoveryResults(App.java:1101)
	at org.datadog.jmxfetch.App$5.invoke(App.java:1046)
	at org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
	at org.datadog.jmxfetch.App.init(App.java:1036)
	at org.datadog.jmxfetch.App.run(App.java:247)
	at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
Caused by: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: java.lang.NullPointerException
	at org.jboss.remoting3.remote.ClientConnectionOpenListener.allMechanismsFailed(ClientConnectionOpenListener.java:114)
	at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:389)
	at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:241)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:198)
	at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:112)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
	at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:571)
	at ...asynchronous invocation...(Unknown Source)
	at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:294)
	at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:276)
	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:393)
	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:377)
	at org.jboss.remotingjmx.RemotingConnector.internalRemotingConnect(RemotingConnector.java:239)
	at org.jboss.remotingjmx.RemotingConnector.internalConnect(RemotingConnector.java:158)
	at org.jboss.remotingjmx.RemotingConnector.connect(RemotingConnector.java:105)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at org.datadog.jmxfetch.Connection.createConnection(Connection.java:57)
	at org.datadog.jmxfetch.RemoteConnection.<init>(RemoteConnection.java:101)
	at org.datadog.jmxfetch.ConnectionFactory.createConnection(ConnectionFactory.java:40)
	at org.datadog.jmxfetch.Instance.getConnection(Instance.java:425)
	at org.datadog.jmxfetch.Instance.init(Instance.java:438)
	at org.datadog.jmxfetch.InstanceInitializingTask.call(InstanceInitializingTask.java:15)
	at org.datadog.jmxfetch.InstanceInitializingTask.call(InstanceInitializingTask.java:3)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

[carlosroman]

Also, we did need to use the special JBoss http-remoting-jmx/'remote+http schemes to connect to JMX on the server, so we had to make sure the JBoss CLI client library JAR was on the jmxfetch (and jmxterm) classpath (/opt/jboss/v7.0/bin/client/jboss-cli-client.jar in our case).

Hi @deejgregor, the documentation does mention that you need to add the a jar to classpath (in your case jboss-cli-client.jar) to connect to the server so that is expected:

Depending on your server setup (particularly when using the remote+http JMX scheme), you may need to specify a custom JAR to connect to the server. Place the JAR on the same machine as your Agent, and add its path to the custom_jar_paths option in your jboss_wildfly.d/conf.yaml file.

I'll double check the spec for credentials and see if just having a username is allowed or if just a password.