cli: cockroach quit does not effectively quit

[cockroach-teamcity]

The following tests appear to have failed:

#513287:

--- FAIL: acceptance/TestDockerCLI (172.770s)
test_log_scope.go:81: test logs captured to: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/logTestDockerCLI666789360
test_log_scope.go:62: use -show-logs to present logs inline
--- FAIL: acceptance/TestDockerCLI: TestDockerCLI/test_missing_log_output.tcl (31.760s)
None
--- FAIL: acceptance/TestDockerCLI (172.770s)
test_log_scope.go:81: test logs captured to: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/logTestDockerCLI666789360
test_log_scope.go:62: use -show-logs to present logs inline
--- FAIL: acceptance/TestDockerCLI: TestDockerCLI/test_missing_log_output.tcl (31.760s)
None

Please assign, take a look and update the issue accordingly.

[knz]

The error is that cockroach quit fails to actually terminates the server process.

FWIW the cockroach quit command says:

E180210 21:41:51.762099 22 rpc/context.go:516  removing connection to localhost:26257 due to error: redialing

[knz]

I'm going to temporarily disable the error on quit until this issue is fixed

[knz]

@bdarnell the problem disappears if I rewind beyond #22518 merging.

[bdarnell]

The error is that cockroach quit fails to actually terminates the server process.

Are you sure? I can't tell from the logs whether the server stopped or not. The immediate failure is here:

expect: does "echo marker; /cockroach/cockroach quit 2>&1 | grep -vE '^[IWEF][0-9]+ .+ vend\rdor/google.golang.org/grpc/' | grep -vE '^(ok|Error)'\r\nmarker\r\nE180209 02:37:53.344521 50 rpc/context.go:516  removing connection to localhost:26257 due to error: redialing\r\n:/# " (spawn_id exp3) match glob pattern "marker\r\n:/# "? no
expect: timed out
:/# 
.180209 02:38:13.348859079 EXPECT TEST: TIMEOUT WAITING FOR "marker
:/# "

An error is logged, and the test is trying to verify that no errors are logged (but the way the test is written this turns into a "timeout"). This could just be an issue with the error whitelist in doShutdown (we don't log grpc connection closed errors, but #22518 introduced a new error type). I'm still not entirely sure why we're seeing that error here, though - I don't think quit should be trying to open multiple connections.