schedule: improve the leader distribution after region scatter (#2659)

[ti-srebot]

cherry-pick #2659 to release-4.0

---

Signed-off-by: nolouch [email protected]

What problem does this PR solve?

Fix #2655

What is changed and how it works?

• add leader store counter
• picker the minimal store

Check List

Tests

• Unit test
• Manual test (add detailed scripts or steps below)
  the new result with this PR.

MySQL [test]> select count(s.region_id) cnt, s.index_name, p.store_id from INFORMATION_SCHEMA.TIKV_REGION_STATUS s join INFORMATION_SCHEMA.tikv_region_peers p on s.region_id = p.region_id where s.table_name = 'ss' group by index_name, p.store_id order by index_name,cnt desc;
+-----+------------+----------+
| cnt | index_name | store_id |
+-----+------------+----------+
| 172 | NULL       |        6 |
| 172 | NULL       |       11 |
| 172 | NULL       |        1 |
| 170 | NULL       |       93 |
| 170 | NULL       |  3670905 |
| 170 | NULL       |       10 |
| 170 | NULL       |        4 |
| 170 | NULL       |        5 |
| 170 | NULL       |        8 |
|   1 | idx1       |       11 |
|   1 | idx1       |        1 |
|   1 | idx1       |        6 |
+-----+------------+----------+
12 rows in set (0.43 sec)

MySQL [test]> select count(s.region_id) cnt, s.index_name, p.store_id from INFORMATION_SCHEMA.TIKV_REGION_STATUS s join INFORMATION_SCHEMA.tikv_region_peers p on s.region_id = p.region_id where s.table_name = 'ss' and p.is_leader = 1 group by index_name, p.store_id order by index_name,cnt desc;
+-----+------------+----------+
| cnt | index_name | store_id |
+-----+------------+----------+
|  59 | NULL       |        1 |
|  58 | NULL       |        6 |
|  57 | NULL       |       93 |
|  57 | NULL       |       10 |
|  57 | NULL       |        8 |
|  57 | NULL       |        4 |
|  56 | NULL       |        5 |
|  56 | NULL       |  3670905 |
|  55 | NULL       |       11 |
|   1 | idx1       |        6 |
+-----+------------+----------+
10 rows in set (0.47 sec)

Release note

• improve the leader distribution after region scatter

[ti-srebot]

@nolouch please accept the invitation then you can push to the cherry-pick pull requests.
https://github.com/ti-srebot/pd/invitations

[rleungx]

Please resolve the conflicts @nolouch

[codecov-commenter]

Codecov Report

Merging #2684 into release-4.0 will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@               Coverage Diff               @@
##           release-4.0    #2684      +/-   ##
===============================================
+ Coverage        77.47%   77.48%   +0.01%     
===============================================
  Files              206      206              
  Lines            22446    22473      +27     
===============================================
+ Hits             17390    17414      +24     
- Misses            3750     3754       +4     
+ Partials          1306     1305       -1     

Impacted Files	Coverage Δ
pkg/mock/mockcluster/mockcluster.go	91.95% <100.00%> (ø)
server/schedule/operator/create_operator.go	96.03% <100.00%> (+0.07%)	⬆️
server/schedule/region_scatterer.go	90.57% <100.00%> (+2.08%)	⬆️
server/region_syncer/client.go	79.38% <0.00%> (-5.35%)	⬇️
server/tso/tso.go	81.57% <0.00%> (-5.27%)	⬇️
server/grpc_service.go	60.23% <0.00%> (-0.97%)	⬇️
pkg/btree/btree.go	86.84% <0.00%> (-0.81%)	⬇️
server/server.go	76.22% <0.00%> (-0.75%)	⬇️
client/client.go	71.95% <0.00%> (-0.49%)	⬇️
server/config/persist_options.go	89.76% <0.00%> (-0.40%)	⬇️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 238b9c1...341454e. Read the comment docs.

[rleungx]

/merge

[ti-srebot]

/run-all-tests