sql/catalog/lease: TestLeaseDescriptorRangeFeedFailure failed

[cockroach-teamcity]

sql/catalog/lease.TestLeaseDescriptorRangeFeedFailure failed with artifacts on release-24.2 @ 4d2d0bc5b392163dfce5f721cd146f7706605e6c:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestLeaseDescriptorRangeFeedFailure (14m28s)

Stack:

goroutine 210578 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x30c
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x38

Log preceding fatal error

=== RUN   TestLeaseDescriptorRangeFeedFailure
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/5632b52c1c67b61307e1d8a1ca2dd18d/logTestLeaseDescriptorRangeFeedFailure3947378376
    test_log_scope.go:81: use -show-logs to present logs inline

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-41471

[rafiss]

goroutine 57758 [select, 13 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).RefreshLeases.func1({0x6388d50, 0x400b2075f0})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:1367 +0xb8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 52950
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c

cockroach/pkg/sql/catalog/lease/lease.go

Lines 1357 to 1367 in 9b56fdc

	// RefreshLeases starts a goroutine that refreshes the lease manager
	// leases for descriptors received in the latest system configuration via gossip or
	// rangefeeds. This function must be passed a non-nil gossip if
	// RangefeedLeases is not active.
	func (m *Manager) RefreshLeases(ctx context.Context, s *stop.Stopper, db *kv.DB) {
	m.mu.Lock()
	defer m.mu.Unlock()
	m.watchForUpdates(ctx)
	_ = s.RunAsyncTask(ctx, "refresh-leases", func(ctx context.Context) {
	for {
	select {

goroutine 56652 [select]:
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).RunBackgroundLeasingTask.func1({0x6388d50, 0x400880fe30})
	github.com/cockroachdb/cockroach/pkg/sql/catalog/lease/lease.go:1609 +0x1e4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x128
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 52950
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x31c

cockroach/pkg/sql/catalog/lease/lease.go

Lines 1584 to 1609 in 9b56fdc

	// RunBackgroundLeasingTask runs background leasing tasks which are
	// responsible for expiring old descriptor versions, monitoring
	// range feed progress / recovery, and supporting legacy expiry
	// based leases.
	func (m *Manager) RunBackgroundLeasingTask(ctx context.Context) {
	_ = m.stopper.RunAsyncTask(ctx, "lease-refresher", func(ctx context.Context) {
	refreshTimerDuration := LeaseDuration.Get(&m.storage.settings.SV)
	renewalsDisabled := false
	if refreshTimerDuration <= 0 {
	// Session based leasing still needs a refresh loop to expire
	// leases, so we will execute that without any renewals.
	refreshTimerDuration = time.Millisecond * 200
	renewalsDisabled = true
	} else {
	refreshTimerDuration = m.storage.jitteredLeaseDuration()
	}
	var refreshTimer timeutil.Timer
	defer refreshTimer.Stop()
	refreshTimer.Reset(refreshTimerDuration / 2)
	// Used to make sure that the system.descriptor lease is active.
	var rangeFeedProgressWatchDog timeutil.Timer
	rangeFeedProgressWatchDogTimeout,
	rangeFeedProgressWatchDogEnabled := m.getRangeFeedMonitorSettings()
	rangeFeedProgressWatchDog.Reset(rangeFeedProgressWatchDogTimeout)
	for {
	select {

[blathers-crl]

Based on the specified backports for linked PR #129302, I applied the following new label(s) to this issue: branch-release-23.2, branch-release-24.1. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

Based on the specified backports for linked PR #129302, I applied the following new label(s) to this issue: branch-release-24.1.4-rc. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}