roachtest: sqlsmith: tracking issue for inbox communication errors

[cockroach-teamcity]

roachtest.sqlsmith/setup=seed-vec/setting=vec failed with artifacts on master @ cef0bd947590218ea6ac94f4f85dffc25e16fcd0:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/sqlsmith/setup=seed-vec/setting=vec/run_1
	sqlsmith.go:221,sqlsmith.go:251,test_runner.go:801: error: pq: inbox communication error: rpc error: code = Canceled desc = context canceled
		stmt:
		SELECT
			tab_7694._bool AS col_28205,
			tab_7694._timestamptz AS col_28206,
			tab_7695._bool AS col_28207,
			tab_7696._int4 AS col_28208,
			tab_7694._float8 AS col_28209,
			tab_7694._interval AS col_28210,
			tab_7693._timestamp AS col_28211,
			tab_7696._bool AS col_28212,
			tab_7694._string AS col_28213
		FROM
			defaultdb.public.seed_vec@seed_vec__int8__float8__date_idx AS tab_7693,
			defaultdb.public.seed_vec@[0] AS tab_7694,
			defaultdb.public.seed_vec@seed_vec__int8__float8__date_idx AS tab_7695
			JOIN defaultdb.public.seed_vec@seed_vec__int8__float8__date_idx AS tab_7696 ON
					(tab_7695._int2) = (tab_7696._int4)
					AND (tab_7695._uuid) = (tab_7696._uuid)
					AND (tab_7695._timestamp) = (tab_7696._timestamp)
					AND (tab_7695._int8) = (tab_7696._int8)
		WHERE
			tab_7696._bool
		ORDER BY
			tab_7693._int2
		LIMIT
			8:::INT8;

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh sqlsmith/setup=seed-vec/setting=vec

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

[yuzefovich]

With regards to these "inbox communication errors" my guess is that it is actually the test harness problem. For sqlsmith roachtests we use SET testing_vectorize_inject_panics=true;, and my hypothesis is that when we inject a panic, this will trigger the flow context cancellation, which in turn triggers the gRPC streams shutdown. At the moment I'm not certain why we would see the remote cancellation errors in such scenario, but there might be a race between the injected error (which we swallow in the sqlsmith) and the inbox communication error.

To test out this hypothesis I'll adjust the test harness to using the setting in 50%.

[yuzefovich]

#69188 - vectorized panic injection is enabled.
#69228 - vectorized panic injection is enabled.
68702 - vectorized panic injection is enabled.
#69736 - vectorized panic injection is enabled.
#69831 - vectorized panic injection is enabled.
#70073 - vectorized panic injection is enabled.
#70074 - vectorized panic injection is enabled.
#70105 - vectorized panic injection is enabled.
#70246 - vectorized panic injection is enabled.

[yuzefovich]

Alright, so it does look like the vectorized panic injection is the cause of these "inbox communication errors". My understanding is as follows: we have a distributed plan, one of the flows get the injected panic (most likely in Init) which is propagated either to the flow coordinator on the gateway or to the outbox on any of the nodes; next, that component triggers the flow cancellation of its flow which shuts down the gRPC stream which will be represented as an "inbox communication error" on one of the inboxes. There is likely a race between propagating the original injected error and this "inbox communication error" to the DistSQLReceiver.

I have an idea of how to make things better.