changefeedccl: TestSchemachangeDoesNotBreakSinklessFeed/sinkless is flaky

[nicktrav]

Describe the problem

The test TestSchemachangeDoesNotBreakSinklessFeed/sinkless is flaky.

Example failure during CI.

To Reproduce

Run the following locally to observe the same error from the TC build:

$ ./dev test ./pkg/ccl/changefeedccl --filter TestSchemachangeDoesNotBreakSinklessFeed/sinkless --stress
...
W220824 16:12:43.379186 588 kv/txn.go:704  [n1] 142  failure aborting transaction: node unavailable; try another peer; abort caused by: context canceled
E220824 16:12:43.391362 588 jobs/registry.go:763  [n1] 146  error claiming jobs: context canceled
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143  job 790649776371138561: running execution encountered retriable error: non-cancelable: job 790649776371138561: select-job: node unavailable; try another peer
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +(1) attached stack trace
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  -- stack trace:
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  |       github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1271
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  |       github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:410
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  |       github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:330
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  |       github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:489
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +Wraps: (2) non-cancelable
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +Wraps: (3) forced error mark
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  | "retriable job error"
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +Wraps: (4) attached stack trace
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  -- stack trace:
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  | github.com/cockroachdb/cockroach/pkg/jobs.(*Job).update
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  |       github.com/cockroachdb/cockroach/pkg/jobs/update.go:296
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  | [...repeated from below...]
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +Wraps: (5) job 790649776371138561
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +Wraps: (6) attached stack trace
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  -- stack trace:
I220824 16:12:43.380590 1640 jobs/registry.go:1273  [n1] 143 +  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
...
ERROR: exit status 1

17 runs completed, 1 failures, over 9s
...

Environment:

Observed on master @ bdadde2.

Jira issue: CRDB-18926

Epic CRDB-11732

[blathers-crl]

cc @cockroachdb/cdc

[amruss]

Closing due to: #86794