Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ticdc: Use open-protocol instead of default protocol and add missing params (#7407) #7453

Merged
merged 2 commits into from
Feb 14, 2022
Merged

ticdc: Use open-protocol instead of default protocol and add missing params (#7407) #7453

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
08d4850
Resolve conflicts
Rustin170506 Feb 14, 2022
da4a393
add new line
Rustin170506 Feb 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions ticdc/manage-ticdc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,10 +182,11 @@ The following are descriptions of parameters and parameter values that can be co
| `kafka-version` | The version of the downstream Kafka (optional, `2.4.0` by default. Currently, the earliest supported Kafka version is `0.11.0.2` and the latest one is `2.7.0`. This value needs to be consistent with the actual version of the downstream Kafka) |
| `kafka-client-id` | Specifies the Kafka client ID of the replication task (optional. `TiCDC_sarama_producer_replication ID` by default) |
| `partition-num` | The number of the downstream Kafka partitions (optional. The value must be **no greater than** the actual number of partitions; otherwise, the replication task cannot be created successfully. `3` by default). Note that from v4.0.16, the default value has changed from `4` to `3`. |
| `max-message-bytes` | The maximum size of data that is sent to Kafka broker each time (optional, `1MB` by default). Note that from v4.0.16, the default value has changed from `512MB` to `1MB`. |
| `max-message-bytes` | The maximum size of data that is sent to Kafka broker each time (optional, `1MB` by default). Note that from v4.0.16, the default value has changed from `512MB` to `1MB`. Since v4.0.17, the default value is `10MB`. |
| `replication-factor` | The number of Kafka message replicas that can be saved (optional, `1` by default) |
| `protocol` | The protocol with which messages are output to Kafka. The value options are `default`, `canal`, `avro`, and `maxwell` (`default` by default) |
| `max-batch-size` | New in v4.0.9. If the message protocol supports outputting multiple data changes to one Kafka message, this parameter specifies the maximum number of data changes in one Kafka message. It currently takes effect only when Kafka's `protocol` is `default`. (optional, `4096` by default) |
| `auto-create-topic` | Determines whether TiCDC creates the topic automatically when the `topic-name` passed in does not exist in the Kafka cluster (optional, `true` by default) |
| `max-batch-size` | New in v4.0.9. If the message protocol supports outputting multiple data changes to one Kafka message, this parameter specifies the maximum number of data changes in one Kafka message. It currently takes effect only when Kafka's `protocol` is `default`. (optional, `16` by default) |
| `ca` | The path of the CA certificate file needed to connect to the downstream Kafka instance (optional) |
| `cert` | The path of the certificate file needed to connect to the downstream Kafka instance (optional) |
| `key` | The path of the certificate key file needed to connect to the downstream Kafka instance (optional) |
Expand Down Expand Up @@ -244,7 +245,7 @@ The following are descriptions of parameters that can be configured for the sink
| `maxPendingMessages` | Sets the maximum size of the pending message queue, which is optional and defaults to 1000. For example, pending for the confirmation message from Pulsar. |
| `disableBatching` | Disables automatically sending messages in batches (optional) |
| `batchingMaxPublishDelay` | Sets the duration within which the messages sent are batched (default: 10ms) |
| `compressionType` | Sets the compression algorithm used for sending messages (optional). The value options are `LZ4`, `ZLIB`, and `ZSTD` (default). |
| `compressionType` | Sets the compression algorithm used for sending messages (optional). The value options are `NONE`,`LZ4`, `ZLIB`, and `ZSTD`. (`NONE` by default) |
| `hashingScheme` | The hash algorithm used for choosing the partition to which a message is sent (optional). The value options are `JavaStringHash` (default) and `Murmur3`. |
| `properties.*` | The customized properties added to the Pulsar producer in TiCDC (optional). For example, `properties.location=Hangzhou`. |

Expand Down
6 changes: 3 additions & 3 deletions ticdc/troubleshoot-ticdc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,8 +281,7 @@ For more information, refer to [Replication task configuration file](/ticdc/mana

## When TiCDC replicates data to Kafka, can I control the maximum size of a single message in TiDB?

+ Since v4.0.9, you can set the `max-message-bytes` parameter to control the maximum size of data sent to the Kafka broker each time (optional, `64MB` by default). You can also set `max-batch-size` to specify the maximum number of change records in each Kafka message. Currently, the setting only takes effect when Kafka's protocol is `default` (optional, `4096` by default).
+ Before v4.0.9, TiCDC sets the maximum size of batch messages to 512 MB, and that of a single message to 4 MB.
Yes. You can set the `max-message-bytes` parameter to control the maximum size of data sent to the Kafka broker each time (optional, `10MB` by default). You can also set `max-batch-size` to specify the maximum number of change records in each Kafka message. Currently, the setting only takes effect when Kafka's `protocol` is `default` (optional, `16` by default).

## When TiCDC replicates data to Kafka, does a message contain multiple types of data changes?

Expand Down Expand Up @@ -360,8 +359,9 @@ If you encounter an error above, it is recommended to use BR to restore the incr

## After I upgrade the TiCDC cluster to v4.0.8, the `[CDC:ErrKafkaInvalidConfig]Canal requires old value to be enabled` error is reported when I execute a changefeed

Since v4.0.8, if the `canal` or `maxwell` protocol is used for output in a changefeed, TiCDC enables the old value feature automatically. However, if you have upgraded TiCDC from an earlier version to v4.0.8 or later, when the changefeed uses the `canal` or `maxwell` protocol and the old value feature is disabled, this error is reported. To fix the error, take the following steps:
Since v4.0.8, if the `canal` or `maxwell` protocol is used for output in a changefeed, TiCDC enables the old value feature automatically. However, if you have upgraded TiCDC from an earlier version to v4.0.8 or later, when the changefeed uses the `canal` or `maxwell` protocol and the old value feature is disabled, this error is reported.

To fix the error, take the following steps:
1. Set the value of `enable-old-value` in the changefeed configuration file to `true`.
2. Execute `cdc cli changefeed pause` to pause the replication task.

Expand Down