Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use min.insync.replicas for standard tests #121

Merged
merged 14 commits into from
Nov 28, 2018
Merged

Use min.insync.replicas for standard tests #121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
7a58284
Removes explicit topic creation from standard tests
solsson Jan 9, 2018
e50b8ea
Displays topic ReplicationFactor etc at producer start
solsson Jan 9, 2018
e059690
Makes producer acks configurable by env
solsson Jan 9, 2018
2f33b8e
Revert "Removes explicit topic creation from standard tests"
solsson Nov 28, 2018
3e6c64e
Prepares tests for v5.0
solsson Nov 28, 2018
5a338f2
Uses kafkacat with librdkafka 1.0.0-RC3 in tests
solsson Nov 28, 2018
09aec3b
Uses min.insync.replicas for the kafkacat producer as well
solsson Nov 28, 2018
aaf0d35
Resource limits for topic creation, based on #221
solsson Nov 28, 2018
7dd078d
Caps resource usage for tests
solsson Nov 28, 2018
03b1e62
Increases the test consumer memory limit slightly due to OOMKilled
solsson Nov 28, 2018
f5524f2
The producer too got OOMKilled at 100Mi
solsson Nov 28, 2018
c8ef8d5
Partition the test topics, a bit
solsson Nov 28, 2018
e420fe2
The kafkacat test now uses the same acks env ad produce-consume
solsson Nov 28, 2018
5fa1078
Merge branch 'master' into test-producer-acks-config
solsson Nov 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions kafka/test/kafkacat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ spec:
spec:
containers:
- name: topic-create
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.1.0@sha256:ac3f06d87d45c7be727863f31e79fbfdcb9c610b51ba9cf03c75a95d602f15e1
command:
- ./bin/kafka-topics.sh
- --zookeeper
Expand All @@ -82,9 +82,13 @@ spec:
- --topic
- test-kafkacat
- --partitions
- "1"
- --replication-factor
- "3"
- --replication-factor
- "2"
resources:
limits:
cpu: 200m
memory: 100Mi
restartPolicy: Never
---
apiVersion: apps/v1beta2
Expand All @@ -107,26 +111,29 @@ spec:
spec:
containers:
- name: producer
image: solsson/kafkacat@sha256:b32eedf936f3cde44cd164ddc77dfcf7565a8af4e357ff6de1abe4389ca530c9
image: solsson/kafkacat@sha256:7a5961f861ddad53c25227511b9f36a5c13bc43296470b383be2f3e8e3a506f4
env:
- name: BOOTSTRAP
value: bootstrap.kafka:9092
- name: ACKS
# -1 means use brokers' min.insync.replicas
value: "-1"
command:
- /bin/bash
- -cex
- >
echo "--- start $HOSTNAME $(date --iso-8601='ns' -u) ---" >> /shared/produce.tmp
;
tail -f /shared/produce.tmp |
kafkacat -P -b $BOOTSTRAP -t test-kafkacat -v -T -d broker -K ':'
kafkacat -P -b $BOOTSTRAP -t test-kafkacat -v -T -d broker -K ':' -X request.required.acks=$ACKS
;
volumeMounts:
- name: config
mountPath: /test
- name: shared
mountPath: /shared
- name: consumer
image: solsson/kafkacat@sha256:b32eedf936f3cde44cd164ddc77dfcf7565a8af4e357ff6de1abe4389ca530c9
image: solsson/kafkacat@sha256:7a5961f861ddad53c25227511b9f36a5c13bc43296470b383be2f3e8e3a506f4
env:
- name: BOOTSTRAP
value: bootstrap.kafka:9092
Expand All @@ -145,7 +152,7 @@ spec:
- name: shared
mountPath: /shared
- name: testcase
image: solsson/kafkacat@sha256:b32eedf936f3cde44cd164ddc77dfcf7565a8af4e357ff6de1abe4389ca530c9
image: solsson/kafkacat@sha256:7a5961f861ddad53c25227511b9f36a5c13bc43296470b383be2f3e8e3a506f4
env:
- name: BOOTSTRAP
value: bootstrap.kafka:9092
Expand Down
35 changes: 28 additions & 7 deletions kafka/test/produce-consume.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ spec:
spec:
containers:
- name: topic-create
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.1.0@sha256:ac3f06d87d45c7be727863f31e79fbfdcb9c610b51ba9cf03c75a95d602f15e1
command:
- ./bin/kafka-topics.sh
- --zookeeper
Expand All @@ -65,9 +65,13 @@ spec:
- --topic
- test-produce-consume
- --partitions
- "1"
- --replication-factor
- "3"
- --replication-factor
- "2"
resources:
limits:
cpu: 200m
memory: 100Mi
restartPolicy: Never
---
apiVersion: apps/v1beta2
Expand Down Expand Up @@ -95,26 +99,35 @@ spec:
spec:
containers:
- name: producer
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.1.0@sha256:ac3f06d87d45c7be727863f31e79fbfdcb9c610b51ba9cf03c75a95d602f15e1
env:
- name: BOOTSTRAP
value: bootstrap.kafka:9092
- name: ACKS
# -1 means use brokers' min.insync.replicas
value: "-1"
command:
- /bin/bash
- -cex
- >
echo "--- start $HOSTNAME $(date --iso-8601='ns' -u) ---" >> /shared/produce.tmp
;
./bin/kafka-topics.sh --zookeeper zookeeper.kafka:2181 --describe --topic test-produce-consume
;
tail -f /shared/produce.tmp |
./bin/kafka-console-producer.sh --broker-list $BOOTSTRAP --topic test-produce-consume
./bin/kafka-console-producer.sh --broker-list $BOOTSTRAP --topic test-produce-consume --producer-property "acks=$ACKS"
;
resources:
limits:
cpu: 100m
memory: 120Mi
volumeMounts:
- name: config
mountPath: /test
- name: shared
mountPath: /shared
- name: consumer
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.1.0@sha256:ac3f06d87d45c7be727863f31e79fbfdcb9c610b51ba9cf03c75a95d602f15e1
env:
- name: BOOTSTRAP
value: bootstrap.kafka:9092
Expand All @@ -125,13 +138,17 @@ spec:
./bin/kafka-console-consumer.sh --bootstrap-server $BOOTSTRAP --topic test-produce-consume |
tee /shared/consumed.tmp
;
resources:
limits:
cpu: 100m
memory: 120Mi
volumeMounts:
- name: config
mountPath: /test
- name: shared
mountPath: /shared
- name: testcase
image: solsson/kafkacat@sha256:ebebf47061300b14a4b4c2e1e4303ab29f65e4b95d34af1b14bb8f7ec6da7cef
image: solsson/kafkacat@sha256:7a5961f861ddad53c25227511b9f36a5c13bc43296470b383be2f3e8e3a506f4
env:
- name: BOOTSTRAP
value: bootstrap.kafka:9092
Expand All @@ -153,6 +170,10 @@ spec:
- /bin/bash
- -e
- /test/quit-on-nonzero-exit.sh
resources:
limits:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /test
Expand Down