Make kafka version configurable #5046

ningyougang · 2021-01-18T07:39:42Z

Make kafka version configurable
Make kafka version using latest version: 2.7.0

kafka latest version: 2.7.0 fixed a bug: https://issues.apache.org/jira/browse/KAFKA-8334

It happened in our env, after deploy latest kafka version in PM, the problem gone.

Description

Related issue and scope

I opened an issue to propose and discuss this change (#????)

My changes affect the following components

Types of changes

Bug fix (generally a non-breaking change which closes an issue).
Enhancement or new feature (adds new functionality).
Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

I signed an Apache CLA.
I reviewed the style guides and followed the recommendations (Travis CI will check :).
I added tests to cover my changes.
My changes require further changes to the documentation.
I updated the documentation where necessary.

style95 · 2021-01-18T08:04:26Z

ansible/group_vars/all

@@ -150,7 +150,7 @@ kafka:
    protocols:
    - TLSv1.2
  protocol: "{{ kafka_protocol_for_setup }}"
-  version: 2.12-2.3.1
+  version: "{{ kafka_version | default('2.13-2.7.0') }}"


Need to see if it does not introduce any regression.

style95 · 2021-01-18T08:06:56Z

Just for those who visit this PR, that issue makes committing offests take more than 5 seconds, accordingly it would dramatically increase the wait time of certain activations.

codecov-io · 2021-01-18T09:40:57Z

Codecov Report

Merging #5046 (5cde872) into master (6686820) will decrease coverage by 7.34%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5046      +/-   ##
==========================================
- Coverage   82.52%   75.17%   -7.35%     
==========================================
  Files         206      211       +5     
  Lines       10006    10454     +448     
  Branches      445      471      +26     
==========================================
- Hits         8257     7859     -398     
- Misses       1749     2595     +846

Impacted Files	Coverage Δ
...core/database/cosmosdb/RxObservableImplicits.scala	`0.00% <0.00%> (-100.00%)`	⬇️
...ore/database/cosmosdb/cache/CacheInvalidator.scala	`0.00% <0.00%> (-100.00%)`	⬇️
...e/database/cosmosdb/cache/ChangeFeedConsumer.scala	`0.00% <0.00%> (-100.00%)`	⬇️
...core/database/cosmosdb/CosmosDBArtifactStore.scala	`0.00% <0.00%> (-95.85%)`	⬇️
...sk/core/database/cosmosdb/CosmosDBViewMapper.scala	`0.00% <0.00%> (-93.90%)`	⬇️
...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala	`0.00% <0.00%> (-92.31%)`	⬇️
...enwhisk/connector/kafka/KamonMetricsReporter.scala	`0.00% <0.00%> (-83.34%)`	⬇️
...e/database/cosmosdb/cache/KafkaEventProducer.scala	`0.00% <0.00%> (-78.58%)`	⬇️
...whisk/core/database/cosmosdb/CosmosDBSupport.scala	`0.00% <0.00%> (-74.08%)`	⬇️
...ore/database/azblob/AzureBlobAttachmentStore.scala	`11.53% <0.00%> (-60.58%)`	⬇️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6686820...5cde872. Read the comment docs.

rabbah

lgtm

ningyougang · 2021-01-28T04:52:58Z

Have any comment?

I benchmarked against the wurstmeister/kafka:2.13-2.7.0 , worked well. result as below

Env

Env: 3 PM nodes（128G memory; 40 core cpu;）
kafka version: wurstmeister/kafka:2.13-2.7.0

Benchmark test

I run producer and consumer side's script at the same time in kafka0 container.

Producer side

# send 50,000,000 records
/opt/kafka_2.13-2.7.0/bin/kafka-producer-perf-test.sh --topic scheduler0 --num-records 50000000  --record-size 100  --throughput -1 --producer-props acks=0 bootstrap.servers=10.105.65.214:9093,10.105.65.215:9094,10.105.65.216:9095
4077415 records sent, 815483.0 records/sec (77.77 MB/sec), 0.5 ms avg latency, 407.0 ms max latency.
4763549 records sent, 952709.8 records/sec (90.86 MB/sec), 0.3 ms avg latency, 15.0 ms max latency.
4392536 records sent, 878507.2 records/sec (83.78 MB/sec), 0.3 ms avg latency, 17.0 ms max latency.
4739980 records sent, 947996.0 records/sec (90.41 MB/sec), 0.3 ms avg latency, 14.0 ms max latency.
4678736 records sent, 935747.2 records/sec (89.24 MB/sec), 0.3 ms avg latency, 10.0 ms max latency.
4289005 records sent, 857801.0 records/sec (81.81 MB/sec), 0.4 ms avg latency, 6.0 ms max latency.
4657720 records sent, 931544.0 records/sec (88.84 MB/sec), 0.3 ms avg latency, 13.0 ms max latency.
4393393 records sent, 878678.6 records/sec (83.80 MB/sec), 0.3 ms avg latency, 11.0 ms max latency.
4286190 records sent, 857238.0 records/sec (81.75 MB/sec), 0.3 ms avg latency, 5.0 ms max latency.
4440910 records sent, 888182.0 records/sec (84.70 MB/sec), 0.3 ms avg latency, 11.0 ms max latency.
4519094 records sent, 903818.8 records/sec (86.19 MB/sec), 0.3 ms avg latency, 13.0 ms max latency.
50000000 records sent, 897311.654284 records/sec (85.57 MB/sec), 0.32 ms avg latency, 407.00 ms max latency, 0 ms 50th, 1 ms 95th, 2 ms 99th, 10 ms 99.9th.

I configured --throughput -1 to disable throttling, we can see, the tps is about 897,311

Consumer side

# Fetch 5,000,000  record (running for about 3 seconds)
/opt/kafka_2.13-2.7.0/bin/kafka-consumer-perf-test.sh  --topic scheduler0 --messages 5000000 --bootstrap-server=xxx.xxx.xxx.xxx:9093,xxx.xxx.xxx.xxx:9094,xxx.xxx.xxx.xxx:9095
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2021-01-28 11:46:58:466, 2021-01-28 11:47:01:299, 476.8650, 168.3251, 5000292, 1765016.5902, 1611802018934, -1611802016101, -0.0000, -0.0031

# Fetch 10,000,000   records  (running for about 5 seconds)
/opt/kafka_2.13-2.7.0/bin/kafka-consumer-perf-test.sh  --topic scheduler0 --messages 10000000 --bootstrap-server=xxx.xxx.xxx.xxx:9093,xxx.xxx.xxx.xxx:9094,xxx.xxx.xxx.xxx:9095
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2021-01-28 11:45:58:096, 2021-01-28 11:46:03:224, 953.6926, 185.9775, 10000192, 1950115.4446, 1611801958563, -1611801953435, -0.0000, -0.0062

# Fetch 1,000,000,000 records (running for about 20 minutes)
/opt/kafka_2.13-2.7.0/bin/kafka-consumer-perf-test.sh  --topic scheduler0 --messages 1000000000 --bootstrap-server=xxx.xxx.xxx.xxx:9093,xxx.xxx.xxx.xxx:9094,xxx.xxx.xxx.xxx:9095
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2021-01-28 11:55:33:186, 2021-01-28 12:16:03:855, 95367.4615, 77.4924, 1000000313, 812566.4277, 1611802533652, -1611801302983, -0.0001, -0.6204

We can saw the the tps is very high as well, when fetch 5,000,000, the tps is 1,765,016, when fetch 10,000,000 the tps is 1,950,115, when fetch 1,000,000,000, the tps is 81,2566

I also tested below scenes
when running kafka-producer-perf-test.sh, i adjusted the --num-records to huge value to make the producer script running for long time, when running kafka-consumer-perf-test.sh, i ajusted the --messages to huge value as well to make the consumer script running for long time

During benchmarking, i observed the cpu/memory doesn't consume too much for all kafka nodes

FYI, on our production env, the consumer.commit offset timeout(>=5.seconds) issue didn't reappeared after we used the latest kafka version.

style95 · 2021-01-28T07:06:01Z

What I mentioned was the benchmarking of OpenWhisk with this version of Kafka.
But anyway it looks good to me as is.

rabbah · 2021-01-29T13:31:23Z

Thanks @ningyougang and @style95!

codecov-commenter · 2024-10-15T06:55:48Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.79%. Comparing base (6686820) to head (5cde872).
Report is 236 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #5046       +/-   ##
===========================================
- Coverage   82.52%   71.79%   -10.73%     
===========================================
  Files         206      219       +13     
  Lines       10006    11256     +1250     
  Branches      445      503       +58     
===========================================
- Hits         8257     8081      -176     
- Misses       1749     3175     +1426

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

style95 reviewed Jan 18, 2021

View reviewed changes

rabbah reviewed Jan 18, 2021

View reviewed changes

ningyougang closed this Jan 21, 2021

ningyougang reopened this Jan 21, 2021

ningyougang closed this Jan 25, 2021

ningyougang reopened this Jan 25, 2021

upgle approved these changes Jan 27, 2021

View reviewed changes

Make kafka version configurable

5cde872

ningyougang force-pushed the kafka-version-configurable branch from 862df50 to 5cde872 Compare January 28, 2021 04:46

style95 approved these changes Jan 28, 2021

View reviewed changes

ningyougang closed this Jan 29, 2021

ningyougang reopened this Jan 29, 2021

ningyougang closed this Feb 1, 2021

ningyougang reopened this Feb 1, 2021

ningyougang closed this Feb 1, 2021

ningyougang reopened this Feb 1, 2021

ningyougang closed this Feb 2, 2021

ningyougang reopened this Feb 2, 2021

ningyougang closed this Feb 3, 2021

ningyougang reopened this Feb 3, 2021

ningyougang closed this Feb 22, 2021

ningyougang reopened this Feb 22, 2021

jiangpengcheng approved these changes Apr 29, 2021

View reviewed changes

jiangpengcheng merged commit df1970b into apache:master Apr 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make kafka version configurable #5046

Make kafka version configurable #5046

ningyougang commented Jan 18, 2021 •

edited

Loading

style95 Jan 18, 2021

style95 commented Jan 18, 2021

codecov-io commented Jan 18, 2021 •

edited

Loading

rabbah left a comment

ningyougang commented Jan 28, 2021 •

edited

Loading

style95 commented Jan 28, 2021 •

edited

Loading

rabbah commented Jan 29, 2021

codecov-commenter commented Oct 15, 2024

Make kafka version configurable #5046

Make kafka version configurable #5046

Conversation

ningyougang commented Jan 18, 2021 • edited Loading

Description

Related issue and scope

My changes affect the following components

Types of changes

Checklist:

style95 Jan 18, 2021

Choose a reason for hiding this comment

style95 commented Jan 18, 2021

codecov-io commented Jan 18, 2021 • edited Loading

Codecov Report

rabbah left a comment

Choose a reason for hiding this comment

ningyougang commented Jan 28, 2021 • edited Loading

Env

Benchmark test

style95 commented Jan 28, 2021 • edited Loading

rabbah commented Jan 29, 2021

codecov-commenter commented Oct 15, 2024

Codecov Report

ningyougang commented Jan 18, 2021 •

edited

Loading

codecov-io commented Jan 18, 2021 •

edited

Loading

ningyougang commented Jan 28, 2021 •

edited

Loading

style95 commented Jan 28, 2021 •

edited

Loading