Add OTel java agent with JMX Metric Insights to kafka (open-telemetry…

…#654) * Add otel agent with JMX metric insights to kafka Signed-off-by: svrnm <[email protected]> * Increase memory limit for kafka * Remove pb files Signed-off-by: svrnm <[email protected]> * Add docs for kafka service Signed-off-by: svrnm <[email protected]> * Revert removal of demo.proto Signed-off-by: svrnm <[email protected]> * Update CHANGELOG Signed-off-by: svrnm <[email protected]> * Fix markdownlint error Signed-off-by: svrnm <[email protected]> Signed-off-by: svrnm <[email protected]> Co-authored-by: Carter Socha <[email protected]> Co-authored-by: Pierre Tessier <[email protected]>
jmichalak9 · Dec 24, 2022 · ccd82b6 · ccd82b6
1 parent 565c514
commit ccd82b6
Show file tree

Hide file tree

Showing 4 changed files with 35 additions and 2 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -158,6 +158,8 @@ significant modifications will be credited to OpenTelemetry Authors.
 ([#628](https://github.com/open-telemetry/opentelemetry-demo/pull/628))
 * Update `quoteservice` to use opentelemetry-php beta release
 ([#644](https://github.com/open-telemetry/opentelemetry-demo/pull/644))
+* Add OTel java agent with JMX Metric Insights to kafka
+([#654](https://github.com/open-telemetry/opentelemetry-demo/pull/654))
 * Add resource detectors to payment service
 ([#651](https://github.com/open-telemetry/opentelemetry-demo/pull/651))
 * Add resource detectors to frontend service

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -537,10 +537,14 @@ services:
     deploy:
       resources:
         limits:
-          memory: 600M
+          memory: 800M
     restart: always
     environment:
-      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
+      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
+      - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
+      - OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
+      - OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE
+      - OTEL_SERVICE_NAME=kafka
     logging: *logging
     healthcheck:
       test: nc -z kafka 9092

diff --git a/docs/services/kafka.md b/docs/services/kafka.md
@@ -0,0 +1,21 @@
+# Kafka
+
+This is used as a message queue service to connect the checkout service with
+the accounting and fraud detection services.
+
+[Kafka service source](../../src/kafka/)
+
+## Auto-instrumentation
+
+This service relies on the OpenTelemetry Java Agent and the built in
+[JMX Metric Insight Module](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent)
+to capture [kafka broker metrics](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/jmx-metrics/javaagent/kafka-broker.md)
+and send them off to the collector via OTLP.
+
+The agent is passed into the process using the `-javaagent` command line
+argument. Command line arguments are added through the `KAFKA_OPTS`
+in the `Dockerfile`.
+
+```dockerfile
+ENV KAFKA_OPTS="-javaagent:/tmp/opentelemetry-javaagent.jar -Dotel.jmx.target.system=kafka-broker"
+```
diff --git a/src/kafka/Dockerfile b/src/kafka/Dockerfile
@@ -1,5 +1,10 @@
 FROM confluentinc/cp-kafka:latest-ubi8
 
+USER root
+ARG version=1.21.0
+ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v$version/opentelemetry-javaagent.jar /tmp/opentelemetry-javaagent.jar
+RUN chmod go+r /tmp/opentelemetry-javaagent.jar
+
 USER appuser
 WORKDIR /tmp
 COPY ./src/kafka/clusterID clusterID
@@ -18,5 +23,6 @@ ENV KAFKA_METADATA_LOG_MAX_RECORD_BYTES_BETWEEN_SNAPSHOTS=2800
 ENV KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
 ENV KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
 ENV KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
+ENV KAFKA_OPTS="-javaagent:/tmp/opentelemetry-javaagent.jar -Dotel.jmx.target.system=kafka-broker"
 
 ENTRYPOINT ["/bin/sh", "-c", "ls -lh /tmp && /tmp/update_run.sh && /etc/confluent/docker/run"]