The etcd client to the registry of Pump/Drainer info does not use auto-sync and fails when the PD cluster address changed

[kennytm]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Start a TiDB cluster with 3 PDs ① ② ③ and a Pump connected
2. Scale-out 3 more PDs ④ ⑤ ⑥
3. Wait 31 seconds
4. Scale-in the original PDs ① ② ③
5. Wait 31 seconds
6. Run SHOW PUMP STATUS;

2. What did you expect to see? (Required)

We see the status of the pump at step 6

3. What did you see instead (Required)

Context deadline exceeded in etcd.(*Client).List

4. What is your TiDB version? (Required)

v4.0.14

[kennytm]

In #9575 we added AutoSyncInterval so the etcd client can automatically refresh the PD member list. But this change is never carried into the tidb-tools/tidb-binlog package (later merged into this repository):

tidb/util/etcd/etcd.go

Lines 81 to 85 in 10f0093

	cli, err := clientv3.New(clientv3.Config{
	Endpoints: endpoints,
	DialTimeout: dialTimeout,
	TLS: security,
	})

(This might also manifest in the "MySQL-compatible AUTO_INCREMENT" feature #38809)

tidb/meta/autoid/autoid.go

Lines 588 to 591 in 00d48f9

	etcdCli, err := clientv3.New(clientv3.Config{
	Endpoints: addrs,
	TLS: ebd.TLSConfig(),
	})

While it is easy to just add back the option AutoSyncInterval: 30 * time.Second in, I think a bigger issue is why do we need to have 4 different etcd clients rather than sharing the same one with all the correct config 🤷

---

For now the lack of AutoSyncInterval means that any change in PD membership requires all tidb-server to be restarted.

[jackysp]

similar issue in other repos,

https://github.com/pingcap/tiflow/blob/b20e1cac3e8142f00665e2bdd4e513f4b2514c3c/dm/pkg/etcdutil/etcdutil.go#L62

https://github.com/pingcap/dm/blob/da80bda17dce4d44c2bd830b58d822dac544b08b/dm/worker/server.go#L107

https://github.com/pingcap/tiflow/blob/b20e1cac3e8142f00665e2bdd4e513f4b2514c3c/dm/ctl/common/util.go#L167

https://github.com/pingcap/dm/blob/da80bda17dce4d44c2bd830b58d822dac544b08b/pkg/etcdutil/etcdutil.go#L74

[jackysp]

Although the pump drainer has been maintained, it seems that other tool components (dm, cdc) may also have similar issues. Should we fix them all at once? @nongfushanquan @overvenus @lance6716

[hawkingrei]

I think we can create a linter to found this problem.