diff --git a/README.md b/README.md index 480c970..2d90e6b 100644 --- a/README.md +++ b/README.md @@ -52,39 +52,38 @@ In addition, certain aspects can be overridden dynamically at runtime from a JSO Where this is possible, a field name is provided in the table. The configuration file described in more detail the next section. -| Environment variable | Description | Default | Dynamic Configuration field name | -|---|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|----------------------------------| -| `KAFKA_BOOTSTRAP_SERVERS` | Comma separated bootstrap servers of the Kafka cluster to connect to. | `localhost:9092` | | -| `KAFKA_BOOTSTRAP_BACKOFF_MAX_ATTEMPTS` | Maximum number of attempts for connecting to the Kafka cluster if it is not ready yet. | `10` | | -| `KAFKA_BOOTSTRAP_BACKOFF_SCALE` | The scale used to delay between attempts to connect to the Kafka cluster (in ms) | `5000` | | -| `TOPIC` | The name of the topic used by the tool to send and receive messages. | `__strimzi_canary` | | -| `TOPIC_CONFIG` | Topic configuration defined as a list of semicolon separated `key=value` pairs (i.e. `retention.ms=600000;segment.bytes=16384`). | empty | | -| `RECONCILE_INTERVAL_MS` | It defines how often the tool has to send and receive messages (in ms). | `30000` | | -| `CLIENT_ID` | The client id used for configuring producer and consumer. | `strimzi-canary-client` | | -| `CONSUMER_GROUP_ID` | Group id for the consumer group joined by the canary consumer. | `strimzi-canary-group` | | -| `PRODUCER_LATENCY_BUCKETS` | Buckets of the histogram related to the producer latency metric (in ms). | `100,200,400,800,1600` | | -| `ENDTOEND_LATENCY_BUCKETS` | Buckets of the histogram related to the end to end latency metric between producer and consumer (in ms). | `100,200,400,800,1600` | | -| `EXPECTED_CLUSTER_SIZE` | Expected number of brokers in the Kafka cluster where the canary connects to. This parameter avoids that the tool runs more partitions reassignment of the topic while the Kafka cluster is starting up and the brokers are coming one by one. `-1` means "dynamic" reassignment as described above. When greater than 0, the canary waits for the Kafka cluster having the expected number of brokers running before creating the topic and assigning the partitions | `-1` | | -| `KAFKA_VERSION` | Version of the Kafka cluster | `3.1.0` | | -| `SARAMA_LOG_ENABLED` | Enables the Sarama client logging. | `false` | `saramaLogEnabled` | -| `VERBOSITY_LOG_LEVEL` | Verbosity of the tool logging. Allowed values 0 = INFO, 1 = DEBUG, 2 = TRACE | `0` | `verbosityLogLevel` | -| `TLS_ENABLED` | If the canary has to use TLS to connect to the Kafka cluster. | `false` | | -| `TLS_CA_CERT` | TLS CA certificate, in PEM format, to use to connect to the Kafka cluster. When this parameter is empty (default behaviour) and the TLS connection is enabled, the canary uses the system certificates trust store. When a TLS CA certificate is specified, it is added to the system certificates trust store | empty | | -| `TLS_CLIENT_CERT` | TLS client certificate, in PEM format, to use for enabling TLS client authentication against the Kafka cluster. | empty | | -| `TLS_CLIENT_KEY` | TLS client private key, in PEM format, to use for enabling TLS client authentication against the Kafka cluster. | empty | | -| `TLS_INSECURE_SKIP_VERIFY` | if the underneath Sarama client has to verify the server's certificate chain and host name. | `false` | | -| `SASL_MECHANISM` | Mechanism to use for SASL authentication against the Kafka cluster. Supported are `PLAIN`, `SCRAM-SHA-256` and `SCRAM-SHA-512`. | empty | | -| `SASL_USER` | Username for SASL authentication against the Kafka cluster when one of `PLAIN`, `SCRAM-SHA-256` or `SCRAM-SHA-512` is used. | empty | | -| `SASL_PASSWORD` | Password for SASL authentication against the Kafka cluster when one of `PLAIN`, `SCRAM-SHA-256` or `SCRAM-SHA-512` is used. | empty | | -| `CONNECTION_CHECK_INTERVAL_MS` | It defines how often the tool has to check the connection with brokers (in ms). | `120000` | | -| `CONNECTION_CHECK_LATENCY_BUCKETS` | Buckets of the histogram related to the broker's connection latency metric (in ms). | `100,200,400,800,1600` | | -| `STATUS_CHECK_INTERVAL_MS` | It defines how often (in ms) the tool updates internal status information (i.e. percentage of consumed messages) to expose outside on the corresponding HTTP endpoint. | `30000` | | -| `STATUS_TIME_WINDOW_MS` | It defines the sliding time window size (in ms) in which status information are sampled. | `300000` | | -| `DYNAMIC_CONFIG_FILE` | Location of an optional external config file that provides configuration at runtime. | empty | | -| `DYNAMIC_CONFIG_WATCHER_INTERVAL` | Interval that dynamic config file is examined for changes in content (in ms) | `30000` | | -| `OTEL_EXPORTER_OTLP_ENDPOINT` | Activate OTLP tracing | empty | | -| `OTEL_EXPORTER_JAEGER_ENDPOINT` | Activate Jaeger tracing | empty | | -| `ACTIVATE_TRACING` | Activate tracing. When true OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_JAEGER_ENDPOINT | `false` | `activateTracing` | +| Environment variable | Description | Default | Dynamic Configuration field name | +|---|---|---|---| +| `KAFKA_BOOTSTRAP_SERVERS` | Comma separated bootstrap servers of the Kafka cluster to connect to. | `localhost:9092` | | +| `KAFKA_BOOTSTRAP_BACKOFF_MAX_ATTEMPTS` | Maximum number of attempts for connecting to the Kafka cluster if it is not ready yet. | `10` | | +| `KAFKA_BOOTSTRAP_BACKOFF_SCALE` | The scale used to delay between attempts to connect to the Kafka cluster (in ms) | `5000` | | +| `TOPIC` | The name of the topic used by the tool to send and receive messages. | `__strimzi_canary` | | +| `TOPIC_CONFIG` | Topic configuration defined as a list of semicolon separated `key=value` pairs (i.e. `retention.ms=600000;segment.bytes=16384`). | empty | | +| `RECONCILE_INTERVAL_MS` | It defines how often the tool has to send and receive messages (in ms). | `30000` | | +| `CLIENT_ID` | The client id used for configuring producer and consumer. | `strimzi-canary-client` | | +| `CONSUMER_GROUP_ID` | Group id for the consumer group joined by the canary consumer. | `strimzi-canary-group` | | +| `PRODUCER_LATENCY_BUCKETS` | Buckets of the histogram related to the producer latency metric (in ms). | `100,200,400,800,1600` | | +| `ENDTOEND_LATENCY_BUCKETS` | Buckets of the histogram related to the end to end latency metric between producer and consumer (in ms). | `100,200,400,800,1600` | | +| `EXPECTED_CLUSTER_SIZE` | Expected number of brokers in the Kafka cluster where the canary connects to. This parameter avoids that the tool runs more partitions reassignment of the topic while the Kafka cluster is starting up and the brokers are coming one by one. `-1` means "dynamic" reassignment as described above. When greater than 0, the canary waits for the Kafka cluster having the expected number of brokers running before creating the topic and assigning the partitions | `-1` | | +| `KAFKA_VERSION` | Version of the Kafka cluster | `3.1.0` | | +| `SARAMA_LOG_ENABLED` | Enables the Sarama client logging. | `false` | `saramaLogEnabled` | +| `VERBOSITY_LOG_LEVEL` | Verbosity of the tool logging. Allowed values 0 = INFO, 1 = DEBUG, 2 = TRACE | `0` | `verbosityLogLevel` | +| `TLS_ENABLED` | If the canary has to use TLS to connect to the Kafka cluster. | `false` | | +| `TLS_CA_CERT` | TLS CA certificate, in PEM format, to use to connect to the Kafka cluster. When this parameter is empty (default behaviour) and the TLS connection is enabled, the canary uses the system certificates trust store. When a TLS CA certificate is specified, it is added to the system certificates trust store | empty | | +| `TLS_CLIENT_CERT` | TLS client certificate, in PEM format, to use for enabling TLS client authentication against the Kafka cluster. | empty | | +| `TLS_CLIENT_KEY` | TLS client private key, in PEM format, to use for enabling TLS client authentication against the Kafka cluster. | empty | | +| `TLS_INSECURE_SKIP_VERIFY` | if the underneath Sarama client has to verify the server's certificate chain and host name. | `false` | | +| `SASL_MECHANISM` | Mechanism to use for SASL authentication against the Kafka cluster. Supported are `PLAIN`, `SCRAM-SHA-256` and `SCRAM-SHA-512`. | empty | | +| `SASL_USER` | Username for SASL authentication against the Kafka cluster when one of `PLAIN`, `SCRAM-SHA-256` or `SCRAM-SHA-512` is used. | empty | | +| `SASL_PASSWORD` | Password for SASL authentication against the Kafka cluster when one of `PLAIN`, `SCRAM-SHA-256` or `SCRAM-SHA-512` is used. | empty | | +| `CONNECTION_CHECK_INTERVAL_MS` | It defines how often the tool has to check the connection with brokers (in ms). | `120000` | | +| `CONNECTION_CHECK_LATENCY_BUCKETS` | Buckets of the histogram related to the broker's connection latency metric (in ms). | `100,200,400,800,1600` | | +| `STATUS_CHECK_INTERVAL_MS` | It defines how often (in ms) the tool updates internal status information (i.e. percentage of consumed messages) to expose outside on the corresponding HTTP endpoint. | `30000` | | +| `STATUS_TIME_WINDOW_MS` | It defines the sliding time window size (in ms) in which status information are sampled. | `300000` | | +| `DYNAMIC_CONFIG_FILE` | Location of an optional external config file that provides configuration at runtime. | empty | | +| `DYNAMIC_CONFIG_WATCHER_INTERVAL` | Interval that dynamic config file is examined for changes in content (in ms) | `30000` | | +| `EXPORTER_TYPE_TRACING` | Tracing Exporter use. Empty value disable tracing, other possible values are `jaeger` or `otlp` | `` | | + ## Dynamic Configuration file @@ -128,6 +127,8 @@ The `Consuming` field provides information about the `Percentage` of messages co } ``` +If the time window has not ended, the `/status` endpoint cannot report a percentage of correctly consumed messages. Instead, it returns `Percentage: -1`. The canary also logs `Error processing consumed records percentage: No data samples available in the time window ring`. In this case, you wait until the time window has ended for the sampling to complete. + ## Metrics In order to check how your Apache Kafka cluster is behaving, the Canary provides the following metrics on the corresponding HTTP endpoint.