tiup playground scale-out --pd 2 fails to scale out 2nd node

[yahonda]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

• Run tiup playground

% tiup playground
tiup is checking updates for component playground ...
Starting component `playground`: /Users/yahonda/.tiup/components/playground/v1.10.3/tiup-playground
Using the version v6.2.0 for version constraint "".

If you'd like to use a TiDB version other than v6.2.0, cancel and retry with the following arguments:
	Specify version manually:   tiup playground <version>
	Specify version range:      tiup playground ^5
	The nightly version:        tiup playground nightly

Playground Bootstrapping...
Start pd instance:v6.2.0
Start tikv instance:v6.2.0
Start tidb instance:v6.2.0
Waiting for tidb instances ready
127.0.0.1:4000 ... Done
Start tiflash instance:v6.2.0
Waiting for tiflash instances ready
127.0.0.1:3930 ... Done
CLUSTER START SUCCESSFULLY, Enjoy it ^-^
To connect TiDB: mysql --comments --host 127.0.0.1 --port 4000 -u root -p (no password)
To view the dashboard: http://127.0.0.1:2379/dashboard
PD client endpoints: [127.0.0.1:2379]
To view the Prometheus: http://127.0.0.1:9090
To view the Grafana: http://127.0.0.1:3000

• Open another terminal to run tiup playground scale-out --pd 2

% tiup playground display
tiup is checking updates for component playground ...
Starting component `playground`: /Users/yahonda/.tiup/components/playground/v1.10.3/tiup-playground display
Pid   Role     Uptime
---   ----     ------
9647  pd       1m4.6123809s
9648  tikv     1m3.968211579s
9649  tidb     1m3.947861541s
9661  tiflash  50.962811757s
yahonda@yahondambp ~ % tiup playground scale-out --pd 2
tiup is checking updates for component playground ...
Starting component `playground`: /Users/yahonda/.tiup/components/playground/v1.10.3/tiup-playground scale-out --pd 2
% tiup playground display
tiup is checking updates for component playground ...
Starting component `playground`: /Users/yahonda/.tiup/components/playground/v1.10.3/tiup-playground display
Pid   Role     Uptime
---   ----     ------
9647  pd       1m34.025902239s
9964  pd       13.006071861s
9965  pd       exit status 1
9648  tikv     1m33.381763418s
9649  tidb     1m33.361408913s
9661  tiflash  1m20.376356258s
%

3. What did you expect to see?
   tiup playground display should show all of processes uptime.

4. What did you see instead?
   Newly added pd exit status 1

9965  pd       exit status 1

• This is the output of the terminal that kick started tiup playground.

receive command: display
receive command: scale-out
Start pd instance:v6.2.0
receive command: scale-out
Start pd instance:v6.2.0
pd quit: exit status 1
[2022/09/05 13:39:48.750 +09:00] [INFO] [util.go:41] ["Welcome to Placement Driver (PD)"]
[2022/09/05 13:39:48.750 +09:00] [INFO] [util.go:42] [PD] [release-version=v6.2.0]
[2022/09/05 13:39:48.751 +09:00] [INFO] [util.go:43] [PD] [edition=Community]
[2022/09/05 13:39:48.751 +09:00] [INFO] [util.go:44] [PD] [git-hash=db465fee5548bf747b0a82b821d01c4c72b34b4f]
[2022/09/05 13:39:48.751 +09:00] [INFO] [util.go:45] [PD] [git-branch=heads/refs/tags/v6.2.0]
[2022/09/05 13:39:48.751 +09:00] [INFO] [util.go:46] [PD] [utc-build-time="2022-08-11 01:47:41"]
[2022/09/05 13:39:48.751 +09:00] [INFO] [metricutil.go:83] ["disable Prometheus push client"]
[2022/09/05 13:39:48.751 +09:00] [INFO] [join.go:217] ["failed to open directory, maybe start for the first time"] [error="open /Users/yahonda/.tiup/data/TGX0XkL/pd-2/data/member: no such file or directory"]
[2022/09/05 13:39:48.752 +09:00] [FATAL] [main.go:91] ["join meet error"] [error="there is a member that has not joined successfully"] [stack="main.main\n\t/Users/pingcap/workspace/build-common@3/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:91\nruntime.main\n\t/usr/local/go1.18.5/src/runtime/proc.go:250"]
...
check detail log from: /Users/yahonda/.tiup/data/TGX0XkL/pd-2/pd.log
receive command: display
tiflash quit: signal: illegal instruction
Logging debug to /Users/yahonda/.tiup/data/TGX0XkL/tiflash-0/log/tiflash.log
Logging errors to /Users/yahonda/.tiup/data/TGX0XkL/tiflash-0/log/tiflash_error.log
deprecated configuration, log-file has been moved to log.file.filename
override log.file.filename with log-file, "/Users/yahonda/.tiup/data/TGX0XkL/tiflash-0/log/tiflash_tikv.log"
...
check detail log from: /Users/yahonda/.tiup/data/TGX0XkL/tiflash-0/tiflash.log

6. What version of TiUP are you using (tiup --version)?

% tiup --version
1.10.3 tiup
Go Version: go1.18.5
Git Ref: v1.10.3
GitHash: e198ac54996fa5a29f1961a460d6634ee9e75d26
%

[yahonda]

Running tiup playground scale-out --pd 1 twice works as expected.

% tiup playground display
tiup is checking updates for component playground ...
Starting component `playground`: /Users/yahonda/.tiup/components/playground/v1.10.3/tiup-playground display
Pid    Role     Uptime
---    ----     ------
10593  pd       51.32483642s
10594  tikv     51.301775395s
10596  tidb     51.277764663s
10610  tiflash  38.855468717s
% tiup playground scale-out --pd 1
tiup is checking updates for component playground ...
Starting component `playground`: /Users/yahonda/.tiup/components/playground/v1.10.3/tiup-playground scale-out --pd 1
% tiup playground display
tiup is checking updates for component playground ...
Starting component `playground`: /Users/yahonda/.tiup/components/playground/v1.10.3/tiup-playground display
Pid    Role     Uptime
---    ----     ------
10593  pd       1m0.127888062s
10642  pd       3.556573207s
10594  tikv     1m0.104844772s
10596  tidb     1m0.080833441s
10610  tiflash  47.658541931s
% tiup playground scale-out --pd 1
tiup is checking updates for component playground ...
Starting component `playground`: /Users/yahonda/.tiup/components/playground/v1.10.3/tiup-playground scale-out --pd 1
% tiup playground display
tiup is checking updates for component playground ...
Starting component `playground`: /Users/yahonda/.tiup/components/playground/v1.10.3/tiup-playground display
Pid    Role     Uptime
---    ----     ------
10593  pd       1m7.850879116s
10642  pd       11.279566001s
10652  pd       3.801777831s
10594  tikv     1m7.827834168s
10596  tidb     1m7.80382388s
10610  tiflash  55.381528124s
%

[HunDunDM]

It looks like the reason is the same as #1187 . The playground requires the same logic as the cluster.
Not only PDs, but DMs also need to perform scaling linearly.

[yahonda]

Thanks for the info. Reading #1187 and there is a pull request #1262 which "implement #1187". It looks like #1187 is included in v1.4.1. I am using v1.10.3

[yahonda]

I might be misunderstanding. #1262 is for cluster not for playground. So the reason is same but the implementation for cluster and playground are different, only cluster has implemented sequential scale out for PD.

[HunDunDM]

This PR is the implementation of tiup cluster, and the playground at that time did not support scaling. The current playground implementation may not consider this.

[yahonda]

Let me confirm what "the playground at that time did not support scaling." means.

1.tiup playground does not support scale-in and scale-out
2.tiup playground scale-out does not consider sequential scale out as tiup cluster does.

My expected answer is 1 because tiup playground -h shows scale-in and scale-out as "Available Commands:`

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  display
  help        Help about any command
  scale-in
  scale-outAvailable Commands:
  completion  Generate the autocompletion script for the specified shell
  display
  help        Help about any command
  scale-in
  scale-out

Please advise.

[HunDunDM]

Sorry, what I want to say is that and the playground at that time did not support sequential scaling for PD.

[HunDunDM]

I think this issue will be fixed in a while. At present for PD, the playground still needs to perform scale-out --pd 1 multiple times.