TiDB label information maybe stale in INFORMATION_SCHEMA.TIDB_SERVERS_INFO system table

[crazycs520]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Deploy TiDB cluster with labels, such topology as following:

tidb_servers:
  - host: 10.0.1.9
    port: 4000
    status_port: 10080
    config:
      labels: { zone: "z1", host: "tidb1" }
  - host: 10.0.1.9
    port: 4002
    status_port: 10082
    config:
      labels: { zone: "z2", host: "tidb2" }

2. Use HTTP API to modify tidb2 labels:

curl -X POST -d '{"zone": "z22"}' http://10.0.1.9:10082/labels

3. Query tidb server label information from INFORMATION_SCHEMA.TIDB_SERVERS_INFO system table, you can see the stale label information of tidb2.

> select * from INFORMATION_SCHEMA.TIDB_SERVERS_INFO;
+--------------------------------------+----------+------+-------------+-------+--------------------+------------------------------------------+---------------+--------------------+
| DDL_ID                               | IP       | PORT | STATUS_PORT | LEASE | VERSION            | GIT_HASH                                 | BINLOG_STATUS | LABELS             |
+--------------------------------------+----------+------+-------------+-------+--------------------+------------------------------------------+---------------+--------------------+
| 7c1b228d-f898-4b47-ae53-f47997584f48 | 10.0.1.9 | 4002 | 10082       | 45s   | 5.7.25-TiDB-v6.5.4 | d7ce2f2faa1da3177a0f0a7e825f6e8fccd13ec8 | Off           | host=tidb2,zone=z2 |
| 9061e6f0-fa93-4c1f-b1f3-30d96720d9c6 | 10.0.1.9 | 4000 | 10080       | 45s   | 5.7.25-TiDB-v6.5.4 | d7ce2f2faa1da3177a0f0a7e825f6e8fccd13ec8 | Off           | host=tidb1,zone=z1 |
+--------------------------------------+----------+------+-------------+-------+--------------------+------------------------------------------+---------------+--------------------+

2. What did you expect to see? (Required)

Latest label information in INFORMATION_SCHEMA.TIDB_SERVERS_INFO.

3. What did you see instead (Required)

Stale label information in INFORMATION_SCHEMA.TIDB_SERVERS_INFO.

4. What is your TiDB version? (Required)

v6.5.4

[crazycs520]

This issue already fixed by #51451 , and release-6.5 already fixed after #51566