sql/rowexec: TestIndexJoiner failed

[cockroach-teamcity]

sql/rowexec.TestIndexJoiner failed with artifacts on master @ c3d71ac887844bef174abb6dab2a4e1ce9270ab7:

../../kv/kvclient/kvstreamer/streamer.go:393 kvstreamer.(*Streamer).Enqueue { s.mu.Lock() } <<<<<
../../kv/kvclient/kvstreamer/streamer.go:392 kvstreamer.(*Streamer).Enqueue { // acquisitions once pipelining is implemented. }
../row/kv_batch_streamer.go:77 row.NewTxnKVStreamer { reqs := spansToRequests(spans, false /* reverse */, keyLocking) }
joinreader.go:896 rowexec.(*joinReader).readInput { kvBatchFetcher, err = row.NewTxnKVStreamer(jr.Ctx, jr.streamerInfo.Streamer, spans, jr.keyLocking) }
joinreader.go:697 rowexec.(*joinReader).Next { case jrReadingInput: }
../execinfra/base.go:187 execinfra.Run { // Emit the row; stop if no more rows are needed. }
../execinfra/processorsbase.go:733 execinfra.(*ProcessorBaseNoHelper).Run { Run(pb.Ctx, pb.self, pb.Output) }
utils_test.go:71 rowexec.runProcessorTest { p.Run(context.Background()) }
joinreader_test.go:1320 rowexec.TestIndexJoiner.func4 { ) }

happened after
../../kv/kvclient/kvstreamer/budget.go:91 kvstreamer.(*budget).consume { b.mu.Lock() } <<<<<
../../kv/kvclient/kvstreamer/budget.go:90 kvstreamer.(*budget).consume { func (b *budget) consume(ctx context.Context, bytes int64, allowDebt bool) error { }
../../kv/kvclient/kvstreamer/streamer.go:510 kvstreamer.(*Streamer).Enqueue { allowDebt := len(reqs) == 1 }
../row/kv_batch_streamer.go:77 row.NewTxnKVStreamer { reqs := spansToRequests(spans, false /* reverse */, keyLocking) }
joinreader.go:896 rowexec.(*joinReader).readInput { kvBatchFetcher, err = row.NewTxnKVStreamer(jr.Ctx, jr.streamerInfo.Streamer, spans, jr.keyLocking) }
joinreader.go:697 rowexec.(*joinReader).Next { case jrReadingInput: }
../execinfra/base.go:187 execinfra.Run { // Emit the row; stop if no more rows are needed. }
../execinfra/processorsbase.go:733 execinfra.(*ProcessorBaseNoHelper).Run { Run(pb.Ctx, pb.self, pb.Output) }
utils_test.go:71 rowexec.runProcessorTest { p.Run(context.Background()) }
joinreader_test.go:1320 rowexec.TestIndexJoiner.func4 { ) }

in another goroutine: happened before
../../kv/kvclient/kvstreamer/streamer.go:758 kvstreamer.(*workerCoordinator).issueRequestsForAsyncProcessing { w.s.budget.mu.Lock() } <<<<<
../../kv/kvclient/kvstreamer/streamer.go:757 kvstreamer.(*workerCoordinator).issueRequestsForAsyncProcessing { }() }
../../kv/kvclient/kvstreamer/streamer.go:713 kvstreamer.(*workerCoordinator).mainLoop { err := w.issueRequestsForAsyncProcessing(ctx, requestsToServe, avgResponseSize) }
../../util/stop/stopper.go:488 stop.(*Stopper).RunAsyncTaskEx.func2 { f(ctx) }

happened after
../../kv/kvclient/kvstreamer/streamer.go:602 kvstreamer.(*Streamer).getNumRequestsInFlight { s.mu.Lock() } <<<<<
../../kv/kvclient/kvstreamer/streamer.go:601 kvstreamer.(*Streamer).getNumRequestsInFlight { func (s *Streamer) getNumRequestsInFlight() int { }
../../kv/kvclient/kvstreamer/streamer.go:760 kvstreamer.(*workerCoordinator).issueRequestsForAsyncProcessing {  }
../../kv/kvclient/kvstreamer/streamer.go:713 kvstreamer.(*workerCoordinator).mainLoop { err := w.issueRequestsForAsyncProcessing(ctx, requestsToServe, avgResponseSize) }
../../util/stop/stopper.go:488 stop.(*Stopper).RunAsyncTaskEx.func2 { f(ctx) }

Other goroutines holding locks:
goroutine 603434 lock 0xc006c93180
../../kv/kvclient/kvstreamer/streamer.go:758 kvstreamer.(*workerCoordinator).issueRequestsForAsyncProcessing { w.s.budget.mu.Lock() } <<<<<
../../kv/kvclient/kvstreamer/streamer.go:757 kvstreamer.(*workerCoordinator).issueRequestsForAsyncProcessing { }() }
../../kv/kvclient/kvstreamer/streamer.go:713 kvstreamer.(*workerCoordinator).mainLoop { err := w.issueRequestsForAsyncProcessing(ctx, requestsToServe, avgResponseSize) }
../../util/stop/stopper.go:488 stop.(*Stopper).RunAsyncTaskEx.func2 { f(ctx) }





ERROR: exit status 2

1 runs completed, 1 failures, over 8s
context canceled

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• TAGS=deadlock

• GOFLAGS=-parallel=4

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

[rytaft]

This was in the log right before the above output:

POTENTIAL DEADLOCK: Inconsistent locking. saw this ordering in one goroutine:

@yuzefovich looks related to the new streamer code