Unclear max-message-bytes warning

[Rustin170506]

What did you do?

1. start a kafka server with this config

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    networks:
      - flink-test
    environment:
      DOCKER_API_VERSION: 1.22
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: 172.23.29.101
      KAFKA_CREATE_TOPICS: "ticdc-test:1:1"
      KAFKA_MESSAGE_MAX_BYTES: 3145728
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper

2. start the tidb and ticdc cluster with tiup: tiup playground nightly --ticdc 1
3. create a changefeed: tiup cdc:nightly cli changefeed create --sink-uri="kafka://localhost:9092/ticdc-test?protocol=canal-json"

What did you expect to see?

Create changefeed normally.

What did you see instead?

I didn't set the parameter this time, but ticdc warned it.

[2022/01/30 14:35:11.368 +08:00] [WARN] [kafka.go:383] ["topic's `max.message.bytes` less than the user set `max-message-bytes`,use topic's `max.message.bytes` to initialize the Kafka producer"] [max.message.bytes=3145728] [max-message-bytes=10485760]
[2022/01/30 14:35:11.368 +08:00] [WARN] [kafka.go:393] ["topic already exist, TiCDC will not create the topic"] [topic=ticdc-test] [detail="{\"NumPartitions\":1,\"ReplicationFactor\":1,\"ReplicaAssignment\":{\"0\":[1001]},\"ConfigEntries\":{\"max.message.bytes\":\"3145728\",\"segment.bytes\":\"1073741824\"}}"]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

mysql> SELECT tidb_version();
+-----------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                                                                                   |
+-----------------------------------------------------------------------------------------------------------------------+
| Release Version: v5.5.0-alpha
Edition: Community
Git Commit Hash: 23f7e51ae01287fa7f811e1462d8987e4d7727a4
Git Branch: heads/refs/tags/v5.5.0-alpha
UTC Build Time: 2022-01-27 14:58:42
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false |
+-----------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

TiCDC version (execute cdc version):

ticdc-test-compose git:(main) tiup cdc:nightly version
The component `cdc` version v5.5.0-nightly-20220127 is not installed; downloading from repository.
component cdc version v5.5.0-nightly-20220127 is already installed
Starting component `cdc`: /Users/rustin/.tiup/components/cdc/v5.5.0-nightly-20220127/cdc version
Release Version: v5.5.0-alpha
Git Commit Hash: 757decd8aa030febe8536c9fdea15d35c6bfb224
Git Branch: heads/refs/tags/v5.5.0-alpha
UTC Build Time: 2022-01-27 14:53:36
Go Version: go version go1.16.4 darwin/amd64
Failpoint Build: false

[Rustin170506]

This problem was discovered while I was writing the blog post, for a more complete process check out: https://zhuanlan.zhihu.com/p/463201547