Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove experimental Kafka metrics #338

Merged
merged 3 commits into from
Sep 21, 2023
Merged

Remove experimental Kafka metrics #338

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5250162
Remove experimental Kafka metrics
pyohannes Sep 21, 2023
5f226c9
Fix TOC
pyohannes Sep 21, 2023
a70ac6a
Merge branch 'main' into remove-experimental-kafka-metrics
reyang Sep 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 0 additions & 75 deletions docs/messaging/kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,6 @@ linkTitle: Kafka
- [Span attributes](#span-attributes)
- [Examples](#examples)
* [Apache Kafka with Quarkus or Spring Boot Example](#apache-kafka-with-quarkus-or-spring-boot-example)
- [Metrics](#metrics)
* [General Metrics](#general-metrics)
* [Producer Metrics](#producer-metrics)
* [Consumer Metrics](#consumer-metrics)

<!-- tocstop -->

Expand Down Expand Up @@ -88,75 +84,4 @@ Process CB: | Span Rcv2 |
| `messaging.kafka.partition` | `"1"` | `"1"` | `"1"` | `"3"` | `"3"` |
| `messaging.kafka.message.offset` | `"12"` | `"12"` | `"12"` | `"32"` | `"32"` |

## Metrics

This section defines how to apply semantic conventions when collecting Kafka metrics.

### General Metrics

**Description:** General Kafka metrics.

| Name | Instrument | Value type | Unit | Unit ([UCUM](/docs/general/metrics.md#instrument-units)) | Description | Attribute Key | Attribute Values |
| ---------------------------------------------| ------------- | ---------- | ------ | -------------------------------------------- | -------------- | ------------- | ---------------- |
| messaging.kafka.messages | Counter | Int64 | messages | `{message}` | The number of messages received by the broker. | | |
| messaging.kafka.requests.failed | Counter | Int64 | requests | `{request}` | The number of requests to the broker resulting in a failure. | `type` | `produce`, `fetch` |
| messaging.kafka.requests.queue | UpDownCounter | Int64 | requests | `{request}` | The number of requests in the request queue. | | |
| messaging.kafka.network.io | Counter | Int64 | bytes | `By` | The bytes received or sent by the broker. | `state` | `in`, `out` |
| messaging.kafka.purgatory.size | UpDownCounter | Int64 | requests | `{request}` | The number of requests waiting in the purgatory. | `type` | `produce`, `fetch` |
| messaging.kafka.partitions.all | UpDownCounter | Int64 | partitions | `{partition}` | The number of partitions in the broker. | | |
| messaging.kafka.partitions.offline | UpDownCounter | Int64 | partitions | `{partition}` | The number of offline partitions. | | |
| messaging.kafka.partitions.under-replicated | UpDownCounter | Int64 | partition | `{partition}` | The number of under replicated partitions. | | |
| messaging.kafka.isr.operations | Counter | Int64 | operations | `{operation}` | The number of in-sync replica shrink and expand operations. | `operation` | `shrink`, `expand` |
| messaging.kafka.lag_max | Gauge | Int64 | lag max | `{message}` | Max lag in messages between follower and leader replicas. | | |
| messaging.kafka.controllers.active | UpDownCounter | Int64 | controllers | `{controller}` | The number of active controllers in the broker. | | |
| messaging.kafka.leader.elections | Counter | Int64 | elections | `{election}` | Leader election rate (increasing values indicates broker failures). | | |
| messaging.kafka.leader.unclean-elections | Counter | Int64 | elections | `{election}` | Unclean leader election rate (increasing values indicates broker failures). | | |
| messaging.kafka.brokers | UpDownCounter | Int64 | brokers | `{broker}` | Number of brokers in the cluster. | | |
| messaging.kafka.topic.partitions | UpDownCounter | Int64 | partitions | `{partition}` | Number of partitions in topic. | `topic` | The ID (integer) of a topic |
| messaging.kafka.partition.current_offset | Gauge | Int64 | partition offset | `{partition offset}` | Current offset of partition of topic. | `topic` | The ID (integer) of a topic |
| | | | | | | `partition` | The number (integer) of the partition |
| messaging.kafka.partition.oldest_offset | Gauge | Int64 | partition offset | `{partition offset}` | Oldest offset of partition of topic | `topic` | The ID (integer) of a topic |
| | | | | | | `partition` | The number (integer) of the partition |
| messaging.kafka.partition.replicas.all | UpDownCounter | Int64 | replicas | `{replica}` | Number of replicas for partition of topic | `topic` | The ID (integer) of a topic |
| | | | | | | `partition` | The number (integer) of the partition |
| messaging.kafka.partition.replicas.in_sync | UpDownCounter | Int64 | replicas | `{replica}` | Number of synchronized replicas of partition | `topic` | The ID (integer) of a topic |
| | | | | | | `partition` | The number (integer) of the partition|

### Producer Metrics

**Description:** Kafka Producer level metrics.

| Name | Instrument | Value type | Unit | Unit ([UCUM](/docs/general/metrics.md#instrument-units)) | Description | Attribute Key | Attribute Values |
| --------------------------------------------- | ------------- | ---------- | ------ | -------------------------------------------- | -------------- | ------------- | ---------------- |
| messaging.kafka.producer.outgoing-bytes.rate | Gauge | Double | bytes per second | `By/s` | The average number of outgoing bytes sent per second to all servers. | `client-id` | `client-id` value |
| messaging.kafka.producer.responses.rate | Gauge | Double | responses per second | `{response}/s` | The average number of responses received per second. | `client-id` | `client-id` value |
| messaging.kafka.producer.bytes.rate | Gauge | Double | bytes per second | `By/s` | The average number of bytes sent per second for a specific topic. | `client-id` | `client-id` value |
| | | | | | | `topic` | topic name |
| messaging.kafka.producer.compression-ratio | Gauge | Double | compression ratio | `{compression}` | The average compression ratio of record batches for a specific topic. | `client-id` | `client-id` value |
| | | | | | | `topic` | topic name |
| messaging.kafka.producer.record-error.rate | Gauge | Double | error rate | `{error}/s` | The average per-second number of record sends that resulted in errors for a specific topic. | `client-id` | `client-id` value |
| | | | | | | `topic` | topic name |
| messaging.kafka.producer.record-retry.rate | Gauge | Double | retry rate | `{retry}/s` | The average per-second number of retried record sends for a specific topic. | `client-id` | `client-id` value |
| | | | | | | `topic` | topic name |
| messaging.kafka.producer.record-sent.rate | Gauge | Double | records sent rate | `{record_sent}/s` | The average number of records sent per second for a specific topic. | `client-id` | `client-id` value |
| | | | | | | `topic` | topic name |

### Consumer Metrics

**Description:** Kafka Consumer level metrics.

| Name | Instrument | Value type | Unit | Unit ([UCUM](/docs/general/metrics.md#instrument-units)) | Description | Attribute Key | Attribute Values |
| --------------------------------------------- | ------------- | ---------- | ------ | -------------------------------------------- | -------------- | ------------- | ---------------- |
| messaging.kafka.consumer.members | UpDownCounter | Int64 | members | `{member}` | Count of members in the consumer group | `group` | The ID (string) of a consumer group |
| messaging.kafka.consumer.offset | Gauge | Int64 | offset | `{offset}` | Current offset of the consumer group at partition of topic | `group` | The ID (string) of a consumer group |
| | | | | | | `topic` | The ID (integer) of a topic |
| | | | | | | `partition` | The number (integer) of the partition |
| messaging.kafka.consumer.offset_sum | Gauge | Int64 | offset sum | `{offset sum}` | Sum of consumer group offset across partitions of topic | `group` | The ID (string) of a consumer group |
| | | | | | | `topic` | The ID (integer) of a topic |
| messaging.kafka.consumer.lag | Gauge | Int64 | lag | `{lag}` | Current approximate lag of consumer group at partition of topic | `group` | The ID (string) of a consumer group |
| | | | | | | `topic` | The ID (integer) of a topic |
| | | | | | | `partition` | The number (integer) of the partition |
| messaging.kafka.consumer.lag_sum | Gauge | Int64 | lag sum | `{lag sum}` | Current approximate sum of consumer group lag across all partitions of topic | `group` | The ID (string) of a consumer group |
| | | | | | | `topic` | The ID (integer) of a topic |

[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/document-status.md
Loading