[ci] Make Kafka integration test more self-siufficient

[yurishkuro]

The test script scripts/kafka-integration-test.sh is currently very simple, it could use some improvements:

• Instead of starting Kafka in the CI it could be started from the script when an extra -k parameter is passed. This makes it easier to run the test locally. Example of starting Kafka manually in [test]: make Kafka consumer read from the earliest messages #4981
• The CI workflow .github/workflows/ci-kafka.yml starts both Kafka and ZK. Is ZK actually needed with the latest versions of Kafka?
• The script has an infinite while true loop waiting for Kafka to start - if should instead use a loop counter and be time bound, say no more than 3min, with checks every 5sec or so

[RipulHandoo]

Hey @yurishkuro, I've made some improvements to the Kafka integration test script. Now, you can start Kafka from within the script using the -k parameter, making it easier to run tests locally. Here's the updated script:

#!/bin/bash

set -e

export STORAGE=kafka

# Function to start Kafka
start_kafka() {
    echo "Starting Kafka..."
    
    docker run --name kafka \
    --network jaeger \
    -p 9092:9092 \
    -e KAFKA_CFG_NODE_ID=0 \
    -e KAFKA_CFG_PROCESS_ROLES=controller,broker \
    -e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093 \
    -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
    -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
    -e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
    -e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
    -e KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT \
    bitnami/kafka:3.6
}

# Check if the -k parameter is provided
if [ "$1" == "-k" ]; then
    start_kafka
fi

# Set the timeout in seconds
timeout=180
# Set the interval between checks in seconds
interval=5

# Calculate the end time
end_time=$((SECONDS + timeout))

while [ $SECONDS -lt $end_time ]; do
    if nc -z localhost 9092; then
        break
    fi
    sleep $interval
done

# Check if Kafka is still not available after the timeout
if ! nc -z localhost 9092; then
    echo "Timed out waiting for Kafka to start"
    exit 1
fi

make storage-integration-test

Changes in ci_kafka.yml

- name: Run kafka integration tests
  run: bash scripts/kafka-integration-test.sh -k

Let me know if any further adjustments are needed. Thanks!

[yurishkuro]

Looks good, please create a pull request

[RipulHandoo]

@yurishkuro I've raised a PR.
PTAL.