Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added information how to run integration tests against Confluence stack #223

Merged
merged 3 commits into from
May 19, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,23 @@ create a configuration template with the correct working directory.
The integration tests are run in parallel e.g. in the CI-pipeline.
The tests need to be engineered taking this in mind.

Since the integration tests run against a Kafka cluster, Kafka REST and Kafka Schema Registry using
their APIs, it's possible to run them not only against Karapace that the tests internally
setup and teardown but against long-running services. This allows you to start
Karapace independently from the integration tests and e.g. inspect Kafka contents after the tests have run.

The integration tests can be configured to use a running (ZooKeeper),
Kafka (:code:`--kafka-bootstrap-servers`), Kafka REST (:code:`--rest-url`)
and Schema Registry (:code:`--registry-url`), e.g. like this::

python -m pytest -vvv --registry-url http://127.0.0.1:8081 --rest-url http://127.0.0.1:8082/ --kafka-bootstrap-servers 127.0.0.1:9092 tests

You can run the integration tests against Kafka REST and Schema Registry from Confluent.
You can freely start these services before running the tests however you wish, but for convenience
you can use the provided Docker Compose file to start ZooKeeper, Kafka, Kafka REST and Schema Registry::

docker-compose -f tests/integration/confluent-docker-compose.yml up -d

There are several coding style checks in `GitHub Actions <https://github.com/aiven/karapace/actions>`_.
Your code changes need to pass these tests. To run the checks locally,
you can run them manually::
Expand Down
51 changes: 51 additions & 0 deletions tests/integration/confluent-docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
hostname: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000

kafka:
image: confluentinc/cp-kafka:latest
hostname: kafka
depends_on:
- zookeeper
ports:
- "9092:9092"
- "9101:9101"
environment:
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081

schema-registry:
image: confluentinc/cp-schema-registry:6.1.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image: confluentinc/cp-schema-registry:6.1.1
image: confluentinc/cp-schema-registry:latest

Maybe this one should be also latest tag?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Maybe we should fix the version we test against and aim for compatibility against that specific version?

Here are instructions from Confluent https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html#ce-docker-quickstart, and here https://github.com/confluentinc/cp-all-in-one/blob/6.1.1-post/cp-all-in-one/docker-compose.yml the compose file referred to in the instructions. In that file the versions are

  • confluentinc/cp-schema-registry:6.1.1
  • confluentinc/cp-kafka-rest:6.1.1

So, I think 6.1.1 for both would make sense.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either way, having latest or 6.1.1 everywhere is fine as long as those are in sync. I'm a slighly favour of latest actually as this is not automatically ran, so we might not want to spend effort to keep these in sync with CP's versioning until there is some automated test run, which we guarantee that tests will pass word-by-word?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to 6.1.1 for both.

Also added mentioning that Karapace aims to be compatible with 6.1.1 and listing of known incompatibilities.


hostname: schema-registry
depends_on:
- kafka
ports:
- "8081:8081"
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'PLAINTEXT://kafka:29092'
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

rest:
image: confluentinc/cp-kafka-rest:latest
depends_on:
- kafka
ports:
- "8082:8082"
environment:
KAFKA_REST_HOST_NAME: confluent-rest
KAFKA_REST_BOOTSTRAP_SERVERS: 'kafka:29092'
KAFKA_REST_LISTENERS: "http://rest:8082"
KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'