Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

addon: Initially scaled to minimize resource use #44

Closed
wants to merge 5 commits into from
Closed

addon: Initially scaled to minimize resource use #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
80f2903
Kafka and importantly Zookeeper upgraded to 2.0.0
solsson Aug 14, 2018
44da767
Scales to 2 brokers + 3 zookeeper instances
solsson Jan 8, 2018
0018eff
Same as default.replication.factor/min.insync.replicas
solsson Feb 3, 2018
620ee42
Minimizes the cluster for use with for example Minikube
solsson Jul 25, 2017
10e74ce
Configures internal topics for single broker,
solsson Feb 3, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion avro-tools/test/rest-curl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ spec:
spec:
containers:
- name: topic-create
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
command:
- ./bin/kafka-topics.sh
- --zookeeper
Expand Down
10 changes: 5 additions & 5 deletions kafka/10broker-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,9 @@ data:
# the brokers.
num.partitions=1

default.replication.factor=3
default.replication.factor=1

min.insync.replicas=2
min.insync.replicas=1

auto.create.topics.enable=true

Expand Down Expand Up @@ -115,9 +115,9 @@ data:
############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
#offsets.topic.replication.factor=1
#transaction.state.log.replication.factor=1
#transaction.state.log.min.isr=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

Expand Down
4 changes: 2 additions & 2 deletions kafka/50kafka.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ spec:
matchLabels:
app: kafka
serviceName: "broker"
replicas: 3
replicas: 1
updateStrategy:
type: OnDelete
template:
Expand Down Expand Up @@ -42,7 +42,7 @@ spec:
mountPath: /etc/kafka
containers:
- name: broker
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
env:
- name: KAFKA_LOG4J_OPTS
value: -Dlog4j.configuration=file:/etc/kafka/log4j.properties
Expand Down
2 changes: 1 addition & 1 deletion kafka/test/kafkacat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ spec:
spec:
containers:
- name: topic-create
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
command:
- ./bin/kafka-topics.sh
- --zookeeper
Expand Down
6 changes: 3 additions & 3 deletions kafka/test/produce-consume.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ spec:
spec:
containers:
- name: topic-create
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
command:
- ./bin/kafka-topics.sh
- --zookeeper
Expand Down Expand Up @@ -95,7 +95,7 @@ spec:
spec:
containers:
- name: producer
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
env:
- name: BOOTSTRAP
value: bootstrap.kafka:9092
Expand All @@ -114,7 +114,7 @@ spec:
- name: shared
mountPath: /shared
- name: consumer
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
env:
- name: BOOTSTRAP
value: bootstrap.kafka:9092
Expand Down
2 changes: 1 addition & 1 deletion maintenance/preferred-replica-election-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
spec:
containers:
- name: kafka
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
command:
- ./bin/kafka-preferred-replica-election.sh
- --zookeeper
Expand Down
2 changes: 1 addition & 1 deletion maintenance/reassign-paritions-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
spec:
containers:
- name: kafka
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
env:
- name: ZOOKEEPER
value: zookeeper.kafka:2181
Expand Down
2 changes: 1 addition & 1 deletion maintenance/replication-factor-increase-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
spec:
containers:
- name: kafka
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
env:
- name: ZOOKEEPER
value: zookeeper.kafka:2181
Expand Down
4 changes: 0 additions & 4 deletions zookeeper/10zookeeper-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,6 @@ data:
initLimit=5
syncLimit=2
server.1=pzoo-0.pzoo:2888:3888:participant
server.2=pzoo-1.pzoo:2888:3888:participant
server.3=pzoo-2.pzoo:2888:3888:participant
server.4=zoo-0.zoo:2888:3888:participant
server.5=zoo-1.zoo:2888:3888:participant

log4j.properties: |-
log4j.rootLogger=INFO, stdout
Expand Down
4 changes: 2 additions & 2 deletions zookeeper/50pzoo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ spec:
app: zookeeper
storage: persistent
serviceName: "pzoo"
replicas: 3
replicas: 1
updateStrategy:
type: OnDelete
template:
Expand All @@ -33,7 +33,7 @@ spec:
mountPath: /var/lib/zookeeper/data
containers:
- name: zookeeper
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
env:
- name: KAFKA_LOG4J_OPTS
value: -Dlog4j.configuration=file:/etc/kafka/log4j.properties
Expand Down
4 changes: 2 additions & 2 deletions zookeeper/51zoo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ spec:
app: zookeeper
storage: ephemeral
serviceName: "zoo"
replicas: 2
replicas: 0
updateStrategy:
type: OnDelete
template:
Expand All @@ -36,7 +36,7 @@ spec:
mountPath: /var/lib/zookeeper/data
containers:
- name: zookeeper
image: solsson/kafka:1.0.2@sha256:7fdb326994bcde133c777d888d06863b7c1a0e80f043582816715d76643ab789
image: solsson/kafka:2.0.0@sha256:8bc5ccb5a63fdfb977c1e207292b72b34370d2c9fe023bdc0f8ce0d8e0da1670
env:
- name: KAFKA_LOG4J_OPTS
value: -Dlog4j.configuration=file:/etc/kafka/log4j.properties
Expand Down