roachtest: import/mixed-versions failed

[cockroach-teamcity]

roachtest.import/mixed-versions failed with artifacts on release-24.2 @ ad78dc73db84c9749fd5d8da982bab8030376251:

(mixedversion.go:694).Run: unexpected node event: n1: cockroach process for system interface died (exit code 134)
test artifacts and logs in: /artifacts/import/mixed-versions/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: import/mixed-versions failed #129023 roachtest: import/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-queries branch-master release-blocker]

This test on roachdash | Improve this report!

Jira issue: CRDB-41449

[rytaft]

Dup of #127582

[rytaft]

Just kidding, not a dup of that issue.

[rytaft]

Looks like there was a panic on node 1:

I240819 05:54:58.089234 1 util/log/flags.go:213  [-] 1  stderr capture started
panic: unknown retry reason 8

goroutine 1763 [running]:
panic({0x5ae6b00?, 0xc012a61e30?})
	GOROOT/src/runtime/panic.go:1017 +0x3ac fp=0xc00d3218c0 sp=0xc00d321810 pc=0x49c36c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.rangeFeedErrorCounters.GetRangeFeedRetryCounter({0xc003ce5920, 0xc003ce5980, {0xc004007c80, 0x8, 0x8}, 0xc003ce59e0, 0xc003ce5a40, 0xc003ce5aa0, 0xc003ce5b00, 0xc003ce5b60, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:427 +0x85 fp=0xc00d321910 sp=0xc00d3218c0 pc=0x1b92005
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.handleRangefeedError({0x7778b48, 0xc0022fb090}, 0xc004aa03e8, {0x7729480, 0xc01738cfb4})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:681 +0x398 fp=0xc00d321a20 sp=0xc00d321910 pc=0x1ba7398
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc004aa0000, {0x7778b48, 0xc0022fb090}, 0xc002445520, {{0xc0012890e8, 0x3, 0x8}, {0xc0012890f0, 0x3, 0x8}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:605 +0x968 fp=0xc00d321e58 sp=0xc00d321a20 pc=0x1ba6be8
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1.1({0x7778b48?, 0xc0022fb090?})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:281 +0xf4 fp=0xc00d321f58 sp=0xc00d321e58 pc=0x1ba4a94
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1.Group.GoCtx.func2()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x22 fp=0xc00d321f78 sp=0xc00d321f58 pc=0x1ba4982
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x56 fp=0xc00d321fe0 sp=0xc00d321f78 pc=0x1ad7716
runtime.goexit()
	src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00d321fe8 sp=0xc00d321fe0 pc=0x4d2fc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1679
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x96

[rytaft]

@msbutler looks related to #127029.

[renatolabs]

This is a dupe of #128057. The fix for this bug is not on any public release yet. This test seems much more prone than the others to hit this bug, so I'd recommend passing mixedversion.AlwaysUseLatestPredecessors to this test so that we stop seeing this bug when v24.1.4 is out.

[blathers-crl]

Based on the specified backports for linked PR #129775, I applied the following new label(s) to this issue: branch-release-23.2, branch-release-24.1. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[rytaft]

This is a dupe of #128057. The fix for this bug is not on any public release yet. This test seems much more prone than the others to hit this bug, so I'd recommend passing mixedversion.AlwaysUseLatestPredecessors to this test so that we stop seeing this bug when v24.1.4 is out.

Thanks @renatolabs!