ccl/multiregionccl: TestTenantStartupWithMultiRegionEnum failed [kvserver.(*StoreMetrics).updateEngineMetrics attempted to decrement a counter]

[cockroach-teamcity]

ccl/multiregionccl.TestTenantStartupWithMultiRegionEnum failed on master @ 8a97a5edd98336e2dd04ef12f08628fba84b17dd:

Fatal error:

panic: Counters should not decrease [recovered]
	panic: Counters should not decrease

Stack:

goroutine 1789 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00bd8eaf8?, {0x800d3d8, 0xc00936f9e0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x65
panic({0x5acb260?, 0x7fa2a00?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/metric.(*Counter).Update(...)
	github.com/cockroachdb/cockroach/pkg/util/metric/pkg/util/metric/metric.go:755
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*StoreMetrics).updateEngineMetrics(_, {0xc00469a908, {0x2b34c0b, 0x9f785, 0x3b501e, 0x105e, 0x7c, 0x105e, 0xd4}, {0x14c5, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/metrics.go:3765 +0xede
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).computeMetrics(_, {_, _})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3478 +0x138
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).ComputeMetricsPeriodically(_, {_, _}, _, _)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3503 +0x98
github.com/cockroachdb/cockroach/pkg/server.(*Node).computeMetricsPeriodically.func1(0xc008369508)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1035 +0xad
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1(0x0?, 0xc00936f860?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:150 +0x26
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*IntMap).Range(0x2?, 0xc00bd8fd18)
	github.com/cockroachdb/cockroach/pkg/util/syncutil/int_map.go:385 +0xd6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores(0xc0095b7140?, 0xc0095b7200?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:149 +0x4e
github.com/cockroachdb/cockroach/pkg/server.(*Node).computeMetricsPeriodically(0xc003f1d408, {0x800d3d8?, 0xc00936f9e0?}, 0x0?, 0x0?)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1034 +0x67
github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func2({0x800d3d8, 0xc00936f9e0})
	github.com/cockroachdb/cockroach/pkg/server/node.go:1007 +0x190
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe

Log preceding fatal error

=== RUN   TestTenantStartupWithMultiRegionEnum
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestTenantStartupWithMultiRegionEnum4054949013
    test_log_scope.go:81: use -show-logs to present logs inline
*
* ERROR: a panic has occurred!
* panic: Counters should not decrease
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:770
*   | github.com/cockroachdb/cockroach/pkg/util/metric.(*Counter).Update
*   | 	github.com/cockroachdb/cockroach/pkg/util/metric/pkg/util/metric/metric.go:755
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*StoreMetrics).updateEngineMetrics
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/metrics.go:3765
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).computeMetrics
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3478
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).ComputeMetricsPeriodically
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3503
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).computeMetricsPeriodically.func1
*   | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1035
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:150
*   | github.com/cockroachdb/cockroach/pkg/util/syncutil.(*IntMap).Range
*   | 	github.com/cockroachdb/cockroach/pkg/util/syncutil/int_map.go:385
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:149
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).computeMetricsPeriodically
*   | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1034
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func2
*   | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1007
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (2) panic: Counters should not decrease
* Error types: (1) *withstack.withStack (2) *errutil.leafError
*

Parameters:

• attempt=1
• run=18
• shard=5

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-39739

[yuzefovich]

dup of #125736