etcd, pd (ticdc): refine pdClient and etcdClient initialization (#9661)

[ti-chi-bot]

This is an automated cherry-pick of #9661

What problem does this PR solve?

Issue Number: close #9565 #9532
ref #9054

What is changed and how it works?

This PR can decrease the probability of cdc capture restarting during pd network isolation and increase the success rate of restarts, resulting in reduced delay for changefeed in this scenario. I tested it on tisapce and found that with this PR, network isolation no longer causes cdc capture and server to exit. (In theory, it is still possible for cdc to exit, but I did not observe it in my testing. Moreover, if cdc exits, the success rate of restarts will be greatly improved.)

The specific approach is:

1. Reduce the number of pdClients that need to be created when cdc server starts from 4 to 2. This can decrease the probability of cdc server failing to start due to pdClient creation failure.
2. After successfully creating the first pdClient, use it to obtain all online pd addresses and update the pd addresses passed to server startup parameters. After this, new addresses will be used for creating other pdClients and etcdClients.
3. When creating etcdClient, a goroutine will be started to maintain its internal address update logic, which is basically copied from pd. Reference: https://github.com/tikv/pd/blob/master/pkg/utils/etcdutil/etcdutil.go#L249

Check List

Tests

• Manual test (add detailed scripts or steps below)

1. Deploy a TiDB cluster with 3 PD nodes and 2 TiCDC nodes, create a synchronization task, and continuously write to the upstream TiDB.
2. Use the following iptables command to inject network isolation into any PD node.

sudo iptables -A OUTPUT -p tcp --dport 2379 -j DROP

3. During the test, observe the server status and synchronization delay of CDC, both of which are normal.

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a bug that may cause cdc restart many time when there is a PD network isolation in the upstream PD cluster.

[asddongmen]

/test all

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CharlesCheung96, nongfushanquan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CharlesCheung96]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@ti-chi-bot: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-dm-integration-test	3c1af5c	link	unknown	/test dm-integration-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[asddongmen]

/retest