roachtest: sqlsmith stream ingestion calls UpdateDeadlineMaybe on leaf txn

[cockroach-teamcity]

(roachtest).sqlsmith/setup=seed/setting=no-mutations failed on master@ec011620c7cf299fdbb898db692b36454defc4a2:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/sqlsmith/setup=seed/setting=no-mutations/run_1
	sqlsmith.go:198,sqlsmith.go:228,test_runner.go:767: error: pq: internal error: crdb_internal.complete_stream_ingestion_job(): get-stream-ingestion-job-metadata: UpdateDeadlineMaybe() called on leaf txn
		stmt:
		WITH
			with_4711 (col_27818)
				AS (
					SELECT
						*
					FROM
						(
							VALUES
								((-8311897396447942706):::INT8),
								(8370906667990257790:::INT8),
								((-4911692604465806628):::INT8),
								(4868999840479281033:::INT8),
								((-7636973884350956732):::INT8)
						)
							AS tab_11473 (col_27818)
				)
		SELECT
			(('13:18:19.26659+12:49:00':::TIMETZ::TIMETZ + '1979-05-28':::DATE::DATE)::TIMESTAMPTZ::TIMESTAMPTZ - tab_11474._interval::INTERVAL)::TIMESTAMPTZ
				AS col_27819,
			crdb_internal.complete_stream_ingestion_job(tab_11474._int2::INT8, tab_11474._timestamptz::TIMESTAMPTZ)::INT8
				AS col_27820,
			e'\U00002603':::STRING AS col_27821,
			tab_11474._date AS col_27822
		FROM
			defaultdb.public.seed@seed__int8__float8__date_idx AS tab_11474;

More

Artifacts: /sqlsmith/setup=seed/setting=no-mutations

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[asubiotto]

Looks like bulk io:

pq: internal error: crdb_internal.complete_stream_ingestion_job(): get-stream-ingestion-job-metadata: UpdateDeadlineMaybe() called on leaf txn

cc @adityamaru. It's possible that you're distributing a plan and the leaf txns (running on remote nodes) try to do something only the root txn (on the gateway) is allowed to do.

[cockroach-teamcity]

(roachtest).sqlsmith/setup=seed/setting=no-mutations failed on master@6601d827b814d4e85a1081b03bf2562d8ac2a4ab:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/sqlsmith/setup=seed/setting=no-mutations/run_1
	cluster.go:1667,context.go:140,cluster.go:1656,test_runner.go:848: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2721381-1614322964-05-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: 6959
		4: 6603
		1: 7918
		2: dead
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1147
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:271
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1852
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /sqlsmith/setup=seed/setting=no-mutations

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[asubiotto]

Latest failure is tracked already in #60994. Will change the title to make this more obvious and avoid future sqlsmith failures posting here.