JMX Exporter Container Fails to Connect to Rest-Server JMX Port (Connection Refused)

[abdelrahman-a-ibrahim]

Description: I am using the following helm chart to deploy Kafka rest proxy with the kafka-rest image tag: 7.0.1. We are encountering an issue with internal communication between two containers in our Kubernetes deployment: the kafka-rest-server-container and the jmx-exporter-container. The kafka-rest-server-container is exposing a JMX port, configured through an environment variable, for the jmx-exporter-container to scrape metrics. However, whenever the jmx-exporter-container attempts to connect to the JMX port on localhost, it encounters a "Connection Refused" error.

Logs: Below are the logs from the jmx-exporter-container that indicate the connection refusal:

SEVERE: [!dt dt.entity.host=HOST-BF4FF85A33F813E3,dt.entity.process_group=PROCESS_GROUP-E763B61EAEB2F724,dt.entity.process_group_instance=PROCESS_GROUP_INSTANCE-9C7A9A7773EF297F,dt.kubernetes.cluster.id=e3e50ceb-9180-4ac5-937e-e87169bb2b16,dt.kubernetes.workload.kind=Deployment,dt.kubernetes.workload.name=kafka-dev-shared-services-kafka-staging-cp-kafka-rest-proxy,k8s.namespace.name=kafka,k8s.pod.name=kafka-dev-shared-services-kafka-staging-cp-kafka-rest-proxcmhh6,k8s.pod.uid=a7b3a612-4dd5-4ff3-8afe-2fa8267f482b] JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
        java.net.ConnectException: Connection refused (Connection refused)]
        ...
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
        java.net.ConnectException: Connection refused (Connection refused)

Expected Behavior: The jmx-exporter-container should successfully connect to the JMX port on the kafka-rest-server-container to retrieve metrics data.

Current Behavior: The connection to the JMX port on localhost is being refused, as shown in the logs above.

Environment:

• Kubernetes Deployment
• Rest-Server Container JMX Port: Configured via KAFKAREST_JMX_PORT environment variable
• JMX Exporter Image: solsson/kafka-prometheus-jmx-exporter@sha256:6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
• JMX Exporter Port: 5556

Relevant Configuration:

JMX Configuration:

jmx:
    enabled: true
    image: solsson/kafka-prometheus-jmx-exporter@sha256
    imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
    imagePullPolicy: IfNotPresent
    port: 5556

Kubernetes Deployment Environment Variables for the kafka-rest-server-container:

Note: tried to use JMX_PORT and got same issue !

- name: KAFKAREST_JMX_PORT
  value: "5555"

[hifly81]

Hi @abdelrahman-a-ibrahim is it for you an option to use the official Confluent for Kubernetes operator instead?
Helm charts are archived.
Here you can find an example including the rest proxy
https://github.com/confluentinc/confluent-kubernetes-examples/blob/master/quickstart-deploy/confluent-platform-singlenode.yaml

and in this repo you can also find a quickstart:
https://github.com/confluentinc/jmx-monitoring-stacks/tree/main/jmxexporter-prometheus-grafana/cfk