Workloads not distributed to all CDC nodes

[fubinzh]

What did you do?

1. TiDB cluster with 3 CDC nodes
2. create kafka changefeed

[root@maincluster-ticdc-0 /]# cat /tmp/changefeed.toml
[filter]
rules = ['bigc.*']
[sink.kafka-config.large-message-handle]
large-message-handle-option = "claim-check"
claim-check-storage-uri = "s3://qe-testing-kafka-claim-check/0926?access-key=xxx&secret-access-key=xxxx&endpoint=http%3a%2f%2fks3-cn-beijing-internal.ksyuncs.com&force-path-style=false&region=Beijing&provider=ks"
large-message-handle-compression = "snappy"

4. Run workload with 6 tables

 nohup /workload --tps=2000 --row-count=3000000000 --table-count=6 --thread=30 --workload-type=large_row --database-host=maincluster-tidb --database-port=4000 --database-user=root --database-db-name=bigc --skip-create-table=true --large-row-size=1048576 --large-ratio=0.005 > large_row_tps2k_0.005_t30_1007.log 2>&1 &

What did you expect to see?

1. Workload of 6 tables should be distributed to 3 CDC node.

What did you see instead?

At first 6 table are distributed to 3 cdc nodes (each CDC nodes take care of 2 tables), then only 2 CDC nodes have workload, finally all workloads goes to 1 CDC nodes.

Versions of the cluster

[release-version=v7.4.0] [git-hash=43848f2fb5646822903baee35b62c580498284e2]

[fubinzh]

/label affects-7.4

[fubinzh]

/severity major

[3AceShowHand]

This happens if the basic scheduler generate remove table task, but which is empty and no-op.