sql/catalog/lease: TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces failed

[cockroach-teamcity]

sql/catalog/lease.TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces failed with artifacts on master @ 36fb1b122380b273f58ac9e13c83e5a6c498454d:

Fatal error:

panic: test timed out after 15m0s
running tests:
	TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces (14m7s)

Stack:

goroutine 587984 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38

Log preceding fatal error

=== RUN   TestRangefeedUpdatesHandledProperlyInTheFaceOfRaces
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/1e31bde6494190748168afe7a747530a/logTestRangefeedUpdatesHandledProperlyInTheFaceOfRaces317038366
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:152: automatically injected an external process virtual cluster under test; see comment at top of test_server_shim.go for details.

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-44703

[spilchen]

We seem to hit this timeout every so often. See #131415 and #129097.

[spilchen]

Additional logs can be found here: https://teamcity.cockroachdb.com/repository/download/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/17849825:id/bazel-testlogs/pkg/sql/catalog/lease/lease_test/shard_1_of_4/test.log

This needs more investigation.

[spilchen]

The test involves running a concurrent query and an ALTER operation on the same table. The query starts first and acquires the table descriptor lease, at which point the test suspends it. Then, the ALTER operation begins creating a new version of the descriptor. It suspends at the end of its execution, waiting for only one version of the descriptor to exist. If the new version of descriptor change is detected by the test, the query resumes, allowing the ALTER to complete.

In the failed run, the descriptor did change, as evidenced by the ALTER operation reaching the waitForOneVersion step. However, the test failed to recognize this, leaving everything suspended until it timed out. While I couldn’t pinpoint the root cause, we could enhance the debug logic to capture additional information if this issue recurs.