Dumpling may get blocked when taskChan is full and cause connection timeout

[lance6716]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

dump a lot of tables

2. What did you expect to see? (Required)

no error

3. What did you see instead (Required)

[2022/07/25 14:29:11.058 +08:00] [INFO] [dump.go:103] ["begin to run Dump"]
...
[2022/07/25 14:43:22.932 +08:00] [INFO] [conn.go:70] ["cannot execute query"] [task=some_task] [unit=dump] [retryTime=1] [sql="SHOW COLUMNS FROM `some_db`.`some_tables`"] [args=null] [error="invalid connection"]

4. What is your TiDB version? (Required)

root cause

tidb/dumpling/export/dump.go

Line 777 in 965454f

case taskChan <- task:

When taskChan is full dumpling will be blocked at sendTaskToChan. And metaConn has a 15-minute read timeout, it will be closed.

[lance6716]

should be closed by #30673

[lance6716]

retry does not take effects https://asktug.com/t/topic/993973/9?u=lance6716

/cc @lichunzhu

[lance6716]

When dumping from MySQL, the default configuration will not retry

tidb/dumpling/export/dump.go

Line 278 in d2295ca

baseConn := newBaseConn(metaConn, canRebuildConn(conf.Consistency, conf.TransactionalConsistency), rebuildConn)