evict-leader-scheduler lost config after PD leader transfer

[kennytm]

Bug Report

What did you do?

$ tiup playground v7.1.1 --mode tikv-slim --pd 3 --kv 3 --without-monitor

$ # first check what stores are available
$ tiup ctl:v7.1.1 pd store --jq .stores[].store.id
1
3
2

$ # add the evict-leader-scheduler
$ tiup ctl:v7.1.1 pd scheduler add evict-leader-scheduler 1
$ tiup ctl:v7.1.1 pd scheduler add evict-leader-scheduler 2
$ tiup ctl:v7.1.1 pd scheduler add evict-leader-scheduler 3
$ tiup ctl:v7.1.1 pd scheduler config evict-leader-scheduler
{"store-id-ranges":{"1":[{"start-key":"","end-key":""}],"2":[{"start-key":"","end-key":""}],"3":[{"start-key":"","end-key":""}]}}

$ # transfer leader 
$ tiup ctl:v7.1.1 pd member leader show | jq .name
"pd-2"
$ tiup ctl:v7.1.1 pd member leader transfer pd-0

$ # wait until the schedulers are ready...
$ tiup ctl:v7.1.1 pd scheduler config evict-leader-scheduler
[404] scheduler not found
...
$ tiup ctl:v7.1.1 pd scheduler config evict-leader-scheduler
{"store-id-ranges":{"1":[{"start-key":"","end-key":""}],"2":[{"start-key":"","end-key":""}],"3":[{"start-key":"","end-key":""}]}}

$ # transfer leader again
$ tiup ctl:v7.1.1 pd member leader transfer pd-1
$ tiup ctl:v7.1.1 pd scheduler config evict-leader-scheduler
[404] scheduler not found
...
$ tiup ctl:v7.1.1 pd scheduler config evict-leader-scheduler # ?!
{"store-id-ranges":{"1":[{"start-key":"","end-key":""}]}}

What did you expect to see?

The evict-leader-scheduler retains the config showing all stores 1, 2, 3.

What did you see instead?

After the second transfer the scheduler can only see the first added store 1.

What version of PD are you using (pd-server -V)?

v6.5.3, v7.1.1

[kennytm]

After the first PD leader transfer (pd-2 → pd-0 in the example), (*coordinator).run() is executed for the first time, in particular executing these two lines:

pd/server/cluster/coordinator.go

Line 376 in dfe7c08

s, err := schedule.CreateScheduler(cfg.Type, c.opController, c.cluster.storage, schedule.ConfigJSONDecoder([]byte(data)))

pd/server/cluster/coordinator.go

Line 397 in dfe7c08

s, err := schedule.CreateScheduler(schedulerCfg.Type, c.opController, c.cluster.storage, schedule.ConfigSliceDecoder(schedulerCfg.Type, schedulerCfg.Args))

The first (line 376) is the new way to create scheduler with the independent configuration. It has correctly added the scheduler with config store = 1,2,3

The second (line 397) is the old way to create the scheduler, which computes the config from the initial args, which is just [1] (stores 2,3 are included into the scheduler through UpdateConfig).

The second "old way" CreateScheduler call persisted the incomplete config, which is why after the second PD leader transfer (pd-0 → pd-1) we are left with only 1 store for evict-leader-scheduler.