testutils: TestRestart flaked on master

[msbutler]

See flake here

Jira issue: CRDB-28702

[blathers-crl]

cc @cockroachdb/test-eng

[msbutler]

just hit this on a seperate branch. going to skip https://teamcity.cockroachdb.com/viewLog.html?buildId=10511008&buildTypeId=Cockroach_BazelEssentialCi

[renatolabs]

In both occurrences, we have this goroutine in the logs:

goroutine 59323 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Wait(0xc0021ac000)
        github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:602 +0x125
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*RemoteFlowRunner).RunFlow.func1.3({0xc0031ba800?, 0x1?})
        github.com/cockroachdb/cockroach/pkg/sql/flowinfra/remote_flow_runner.go:123 +0x69
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
        github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
        github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b

Seems like we're stuck here:

cockroach/pkg/sql/flowinfra/remote_flow_runner.go

Lines 119 to 125 in 8dc422f

	if err := r.stopper.RunAsyncTask(ctx, "flowinfra.RemoteFlowRunner: waiting for flow to finish", func(ctx context.Context) {
	if shouldExit := <-waiterShouldExit; shouldExit {
	return
	}
	f.Wait()
	cleanup()
	}); err != nil {

Maybe the race is that:

• f.Start() returns, and closes waiterShouldExit:
  

  cockroach/pkg/sql/flowinfra/remote_flow_runner.go

  Lines 131 to 137 in 8dc422f

  	if err := f.Start(ctx); err != nil {
  	cleanup()
  	cleanedUp = true
  	waiterShouldExit <- true
  	return err
  	}
  	close(waiterShouldExit)

• The server starts quiescing

• We continue waiting for the flow to finish:
  

  cockroach/pkg/sql/flowinfra/remote_flow_runner.go

  Lines 120 to 123 in 8dc422f

  	if shouldExit := <-waiterShouldExit; shouldExit {
  	return
  	}
  	f.Wait()

Not familiar with this code, so the above might not make sense. Tagging @yuzefovich as the original author.

[yuzefovich]

The root cause for the hang is different - I think this goroutine is the culprit:

goroutine 59324 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Connection).waitOrDefault(0xc003f92c40, {0x60a67e0, 0xc00b5a6db0}, {0x0, 0x0}, {0x607e018, 0x8fc8500})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/connection.go:79 +0x305
github.com/cockroachdb/cockroach/pkg/rpc.(*Connection).ConnectNoBreaker(0xc003f92c40, {0x60a67e0, 0xc00b5a6db0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/connection.go:145 +0x5e
github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*Dialer).dial(0xc0078a18a8, {0x60a67e0, 0xc00b5a6db0}, 0x33333333?, {0x60802a0?, 0xc00b5d20f8}, 0x0, 0x0?)
	github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:175 +0x10f
github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*Dialer).DialNoBreaker(0xc0078a18a8, {0x60a67e0, 0xc00b5a6db0}, 0x3b9aca00?, 0x0?)
	github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:119 +0x8d
github.com/cockroachdb/cockroach/pkg/sql/execinfra.GetConnForOutbox({0x60a67e0, 0xc00b5a6db0}, {0x6069ee0, 0xc0078a18a8}, 0x4a82520?, 0x2540be400)
	github.com/cockroachdb/cockroach/pkg/sql/execinfra/outboxbase.go:47 +0x215
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).Run.func1({0x60a67e0, 0xc00b5a6db0}, {0x6069ee0?, 0xc0078a18a8?}, 0x1?, 0x0?, {0x60a67e0, 0xc00b5a6bd0}, 0xc00e0d7ea0, 0xc003da6750, ...)
	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:187 +0x65
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).Run(0xc003da6750, {0x60a67e0, 0xc00b5a6bd0}, {0x6069ee0, 0xc0078a18a8}, 0x4d14f60?, 0x1, 0xc00652e170, 0xc004d14fb0?)
	github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:228 +0x336
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRemoteOutputStream.func1({0x60a67e0, 0xc00b5a6bd0}, 0xc004d14f98?)
	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:768 +0xac
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).accumulateAsyncComponent.func1.1()
	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:730 +0x6b
created by github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).accumulateAsyncComponent.func1
	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:728 +0xdd

I believe the regression was introduced in #99191 and should have been fixed in #104945. I'll stress the test to make sure it's not flaky anymore (so far 5 minutes with no failures).