Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cli: TestDebugCheckStore failed #122328

Closed
cockroach-teamcity opened this issue Apr 13, 2024 · 7 comments · Fixed by #126690
Closed

cli: TestDebugCheckStore failed #122328

cockroach-teamcity opened this issue Apr 13, 2024 · 7 comments · Fixed by #126690
Assignees
@arulajmani
Labels
branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 13, 2024 •
edited by cockroach-jira-scripts
Loading

cli.TestDebugCheckStore failed on master @ f117eea22dd7be380c7141cdf6cd7aba92dd9c70:

=== RUN   TestDebugCheckStore
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestDebugCheckStore1090274282
    test_log_scope.go:81: use -show-logs to present logs inline
    debug_check_store_test.go:90: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/cli/debug_check_store_test.go:90
        	Error:      	Received unexpected error:
        	            	check-store found problems
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/cli.init
        	            	  | 	github.com/cockroachdb/cockroach/pkg/cli/debug_check_store.go:55
        	            	  | runtime.doInit1
        	            	  | 	GOROOT/src/runtime/proc.go:7206
        	            	  | runtime.doInit
        	            	  | 	GOROOT/src/runtime/proc.go:7173
        	            	  | runtime.main
        	            	  | 	GOROOT/src/runtime/proc.go:253
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) check-store found problems
        	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
        	Test:       	TestDebugCheckStore
        	Messages:   	dir=/var/lib/engflow/worker/work/2/exec/_tmp/10a3dedc16573dd48c5adba69877df01/TestDebugCheckStore1258554494/s1
        	            	out=r18:/Table/1{5-6} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: stats inconsistency:
        	            	- stored:
        	            	{ContainsEstimates:0 LastUpdateNanos:1712989308216988618 LockAge:0 GCBytesAge:89851 LiveBytes:2897 LiveCount:45 KeyBytes:2601 KeyCount:55 ValBytes:2323 ValCount:90 IntentBytes:0 IntentCount:0 LockBytes:0 LockCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:1466 SysCount:8 AbortSpanBytes:68}
        	            	- recomputed:
        	            	{ContainsEstimates:0 LastUpdateNanos:1712989308216988618 LockAge:0 GCBytesAge:89851 LiveBytes:2897 LiveCount:45 KeyBytes:2601 KeyCount:55 ValBytes:2323 ValCount:90 IntentBytes:0 IntentCount:0 LockBytes:0 LockCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:1319 SysCount:8 AbortSpanBytes:68}
        	            	- diff:
        	            	SysBytes: 1466 != 1319
        	            	scanned 68 ranges (1 with estimates), total stats contains_estimates:0 last_update_nanos:1712989308687825374 lock_age:0 gc_bytes_age:1492250 live_bytes:1783393 live_count:9169 key_bytes:503798 key_count:9463 val_bytes:1314297 val_count:9854 intent_bytes:68 intent_count:1 lock_bytes:0 lock_count:1 range_key_count:0 range_key_bytes:0 range_val_count:0 range_val_bytes:0 sys_bytes:37944 sys_count:416 abort_span_bytes:68 
        	            	
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestDebugCheckStore1090274282
--- FAIL: TestDebugCheckStore (61.52s)

Parameters:

  • attempt=1
  • run=22
  • shard=6
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-37803
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Apr 13, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Apr 13, 2024
@miraradeva
Copy link
Contributor

This is the same issue as #112219, which we haven't gotten around to doing.

I'm removing the release-blocker label and assigning P3 to match the original issue.

@miraradeva miraradeva added P-3 Issues/test failures with no fix SLA and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 19, 2024
@miraradeva miraradeva removed their assignment Apr 19, 2024
@miraradeva miraradeva added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Apr 19, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue May 2, 2024
Closed
@msbutler
Copy link
Collaborator

flaked again here https://mesolite.cluster.engflow.com/invocations/default/090b7489-3aae-44d1-be20-2b36a31149fb

@cockroach-teamcity
Copy link
Member Author

cli.TestDebugCheckStore failed on master @ d8073b58467ddb4c5b2ebe4a485fc42fb98f7200:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestDebugCheckStore (14m57s)

Stack:

goroutine 10955309 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

=== RUN   TestDebugCheckStore
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestDebugCheckStore3141802188
    test_log_scope.go:81: use -show-logs to present logs inline

Parameters:

  • attempt=1
  • run=8
  • shard=6
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

cli.TestDebugCheckStore failed on master @ 3d9eaa7939e4fb5aa0518ba7e19750a55c93cf11:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestDebugCheckStore (14m57s)

Stack:

goroutine 14373569 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

=== RUN   TestDebugCheckStore
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestDebugCheckStore3856725973
    test_log_scope.go:81: use -show-logs to present logs inline

Parameters:

  • attempt=1
  • run=11
  • shard=6
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jun 8, 2024
Closed
@nicktrav
Copy link
Collaborator

Seen again here: https://mesolite.cluster.engflow.com/invocations/default/989bf940-e487-484e-b0e7-d7015636b4d0

@github-actions GitHub Actions
Copy link

cli.TestDebugCheckStore failed on master @ b5ece5bd70bc5d82662d055ca4b2dd5d34a2316d:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestDebugCheckStore (14m57s)

Stack:

goroutine 12000204 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

=== RUN   TestDebugCheckStore
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestDebugCheckStore1277519348
    test_log_scope.go:81: use -show-logs to present logs inline

Parameters:

  • attempt=1
  • run=1
  • shard=6
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 1, 2024

cli.TestDebugCheckStore failed on master @ f96e19035560e88d705fd1b29190dbea5b5bcda6:

E240701 15:54:30.490953 5487992 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r57/1:/Table/5{5-6}] 3106  [n1,s1,r57/1:/Table/5{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r57:/Table/5{5-6} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n3,s3):2,(n2,s2):3); closed timestamp: 1719848905.151807289,0 (2024-07-01 15:48:25); raft status: {"id":"1","term":7,"vote":"2","commit":74,"lead":"0","raftState":"StatePreCandidate","applied":74,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/55]
E240701 15:54:30.491708 5487993 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r17/1:/Table/1{4-5}] 3107  [n1,s1,r17/1:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n2,s2):2,(n3,s3):3); closed timestamp: 1719848902.011087481,0 (2024-07-01 15:48:22); raft status: {"id":"1","term":7,"vote":"2","commit":77,"lead":"0","raftState":"StatePreCandidate","applied":77,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/14]
E240701 15:54:30.499041 5488094 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r68/3:/{Table/66-Max}] 3108  [n2,s2,r68/3:/{Table/66-Max}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r68:/{Table/66-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"3","term":7,"vote":"1","commit":76,"lead":"0","raftState":"StatePreCandidate","applied":76,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/66]
E240701 15:54:30.502645 5487960 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,lease,s2,r33/2:/NamespaceTable/{30-Max}] 3109  [n2,s2,r33/2:/NamespaceTable/{30-Max}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r33:/NamespaceTable/{30-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848900.193119688,0 (2024-07-01 15:48:20); raft status: {"id":"2","term":10,"vote":"2","commit":82,"lead":"0","raftState":"StatePreCandidate","applied":82,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/NamespaceTable/30]
E240701 15:54:30.509786 5488115 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicaGC,s2,r3/3:/System/{NodeLive…-tsd}] 3110  replica unavailable: (n2,s2):2 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848904.895442203,0 (2024-07-01 15:48:24); raft status: {"id":"2","term":9,"vote":"2","commit":609,"lead":"0","raftState":"StatePreCandidate","applied":609,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Min]
W240701 15:54:30.536856 15049 kv/kvserver/replica_gossip.go:158  [T1,Vsystem,n2,s2,r1/2:/{Min-System/NodeL…}] 3111  could not acquire lease for range gossip: replica unavailable: (n2,s2):2 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848904.895442203,0 (2024-07-01 15:48:24); raft status: {"id":"2","term":9,"vote":"2","commit":609,"lead":"0","raftState":"StatePreCandidate","applied":609,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Min]
E240701 15:54:30.542966 5488119 kv/kvserver/queue.go:1202  [T1,Vsystem,n3,replicaGC,s3,r33/3:/NamespaceTable/{30-Max}] 3112  replica unavailable: (n3,s3):3 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n2,s2):2); closed timestamp: 1719848904.895442203,0 (2024-07-01 15:48:24); raft status: {"id":"3","term":9,"vote":"2","commit":609,"lead":"0","raftState":"StatePreCandidate","applied":609,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Min]
E240701 15:54:30.555078 5488120 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicaGC,s1,r50/1:/Table/4{7-8}] 3113  replica unavailable: (n1,s1):1 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n2,s2):2,(n3,s3):3); closed timestamp: 1719848904.895442203,0 (2024-07-01 15:48:24); raft status: {"id":"1","term":9,"vote":"2","commit":609,"lead":"0","raftState":"StatePreCandidate","applied":609,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Min]
E240701 15:54:30.569163 5488121 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r59/2:/Table/5{7-8}] 3114  [n2,s2,r59/2:/Table/5{7-8}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r59:/Table/5{7-8} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848903.506670679,0 (2024-07-01 15:48:23); raft status: {"id":"2","term":7,"vote":"1","commit":74,"lead":"0","raftState":"StatePreCandidate","applied":74,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/57]
E240701 15:54:30.570060 5488122 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r22/2:/Table/{19-20}] 3115  [n2,s2,r22/2:/Table/{19-20}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r22:/Table/{19-20} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848905.495711572,0 (2024-07-01 15:48:25); raft status: {"id":"2","term":7,"vote":"2","commit":77,"lead":"0","raftState":"StatePreCandidate","applied":77,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/19]
W240701 15:54:30.574518 18406 kv/kvserver/replica_gossip.go:158  [T1,Vsystem,n3,s3,r1/3:/{Min-System/NodeL…}] 3116  could not acquire lease for range gossip: replica unavailable: (n3,s3):3 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n2,s2):2); closed timestamp: 1719848904.895442203,0 (2024-07-01 15:48:24); raft status: {"id":"3","term":9,"vote":"2","commit":609,"lead":"0","raftState":"StatePreCandidate","applied":609,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Min]
E240701 15:54:30.575911 5488143 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,lease,s2,r17/2:/Table/1{4-5}] 3117  [n2,s2,r17/2:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848902.011087481,0 (2024-07-01 15:48:22); raft status: {"id":"2","term":7,"vote":"2","commit":78,"lead":"0","raftState":"StatePreCandidate","applied":78,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/14]
E240701 15:54:30.576589 5488144 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,lease,s2,r43/3:/Table/4{0-1}] 3118  [n2,s2,r43/3:/Table/4{0-1}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r43:/Table/4{0-1} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"3","term":6,"vote":"0","commit":76,"lead":"0","raftState":"StatePreCandidate","applied":76,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/40]
W240701 15:54:30.577451 5488116 gossip/client.go:121  [T1,Vsystem,n3] 3119  failed to start gossip client to 127.0.0.1:44215: initial connection heartbeat failed: grpc: connection error: desc = "transport: authentication handshake failed: context deadline exceeded" [code 14/Unavailable]
E240701 15:54:30.578288 5488118 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r26/1:/Table/2{3-4}] 3120  [n1,s1,r26/1:/Table/2{3-4}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r26:/Table/2{3-4} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n3,s3):2,(n2,s2):3); closed timestamp: 1719848904.936319431,0 (2024-07-01 15:48:24); raft status: {"id":"1","term":7,"vote":"2","commit":76,"lead":"0","raftState":"StatePreCandidate","applied":76,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/23]
E240701 15:54:30.580807 5488175 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r65/2:/Table/6{3-4}] 3121  [n2,s2,r65/2:/Table/6{3-4}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r65:/Table/6{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"2","term":7,"vote":"1","commit":75,"lead":"0","raftState":"StatePreCandidate","applied":75,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/63]
E240701 15:54:30.584766 5488217 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,lease,s2,r13/2:/Table/{9-11}] 3122  [n2,s2,r13/2:/Table/{9-11}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r13:/Table/{9-11} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848911.995208623,0 (2024-07-01 15:48:31); raft status: {"id":"2","term":7,"vote":"2","commit":76,"lead":"0","raftState":"StatePreCandidate","applied":76,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/9]
E240701 15:54:30.585339 5488218 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,lease,s2,r35/3:/Table/3{2-3}] 3123  [n2,s2,r35/3:/Table/3{2-3}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r35:/Table/3{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848904.936319431,0 (2024-07-01 15:48:24); raft status: {"id":"3","term":8,"vote":"1","commit":75,"lead":"0","raftState":"StatePreCandidate","applied":75,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/32]
E240701 15:54:30.587092 5488239 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r37/1:/Table/3{4-5}] 3124  [n1,s1,r37/1:/Table/3{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r37:/Table/3{4-5} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n2,s2):2,(n3,s3):3); closed timestamp: 1719848900.193119688,0 (2024-07-01 15:48:20); raft status: {"id":"1","term":8,"vote":"2","commit":74,"lead":"0","raftState":"StatePreCandidate","applied":74,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/34]
E240701 15:54:30.587575 5488240 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r1/1:/{Min-System/NodeL…}] 3125  [n1,s1,r1/1:/{Min-System/NodeL…}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n2,s2):2,(n3,s3):3); closed timestamp: 1719848904.895442203,0 (2024-07-01 15:48:24); raft status: {"id":"1","term":9,"vote":"2","commit":609,"lead":"0","raftState":"StatePreCandidate","applied":609,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Min]
E240701 15:54:30.590632 5488271 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r16/3:/Table/1{3-4}] 3126  [n2,s2,r16/3:/Table/1{3-4}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r16:/Table/1{3-4} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848902.077377614,0 (2024-07-01 15:48:22); raft status: {"id":"3","term":7,"vote":"3","commit":346,"lead":"0","raftState":"StatePreCandidate","applied":346,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/13]
E240701 15:54:30.591176 5488272 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r52/3:/Table/5{0-1}] 3127  [n2,s2,r52/3:/Table/5{0-1}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r52:/Table/5{0-1} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"3","term":7,"vote":"1","commit":75,"lead":"0","raftState":"StatePreCandidate","applied":75,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/50]
E240701 15:54:30.607903 5488303 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,lease,s2,r9/2:/Table/{5-6}] 3128  [n2,s2,r9/2:/Table/{5-6}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r9:/Table/{5-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848904.482820611,0 (2024-07-01 15:48:24); raft status: {"id":"2","term":7,"vote":"2","commit":93,"lead":"0","raftState":"StatePreCandidate","applied":93,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/5]
E240701 15:54:30.609519 5488304 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicaGC,s2,r63/3:/Table/6{1-2}] 3129  replica unavailable: (n2,s2):2 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848904.895442203,0 (2024-07-01 15:48:24); raft status: {"id":"2","term":9,"vote":"2","commit":609,"lead":"0","raftState":"StatePreCandidate","applied":609,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Min]
E240701 15:54:30.610668 5488305 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r11/1:/Table/{7-8}] 3130  [n1,s1,r11/1:/Table/{7-8}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r11:/Table/{7-8} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"1","term":8,"vote":"1","commit":71,"lead":"0","raftState":"StatePreCandidate","applied":71,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/7]
E240701 15:54:30.611294 5488306 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicate,s1,r47/1:/Table/4{4-5}] 3131  [n1,s1,r47/1:/Table/4{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r47:/Table/4{4-5} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n2,s2):2,(n3,s3):3); closed timestamp: 1719848902.011087481,0 (2024-07-01 15:48:22); raft status: {"id":"1","term":7,"vote":"2","commit":78,"lead":"0","raftState":"StatePreCandidate","applied":78,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/44]
E240701 15:54:30.614486 5488307 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r32/2:/{Table/29-NamespaceTab…}] 3132  [n2,s2,r32/2:/{Table/29-NamespaceTab…}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r32:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"2","term":7,"vote":"1","commit":73,"lead":"0","raftState":"StatePreCandidate","applied":73,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/29]
E240701 15:54:30.615023 5488308 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r55/2:/Table/5{3-4}] 3133  [n2,s2,r55/2:/Table/5{3-4}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r55:/Table/5{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848905.495711572,0 (2024-07-01 15:48:25); raft status: {"id":"2","term":7,"vote":"2","commit":77,"lead":"0","raftState":"StatePreCandidate","applied":77,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/53]
E240701 15:54:30.615517 5488309 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r38/2:/Table/3{5-6}] 3134  [n2,s2,r38/2:/Table/3{5-6}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r38:/Table/3{5-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848900.193119688,0 (2024-07-01 15:48:20); raft status: {"id":"2","term":9,"vote":"3","commit":76,"lead":"0","raftState":"StatePreCandidate","applied":76,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/35]
E240701 15:54:30.616007 5488310 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r49/2:/Table/4{6-7}] 3135  [n2,s2,r49/2:/Table/4{6-7}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r49:/Table/4{6-7} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"2","term":6,"vote":"0","commit":79,"lead":"0","raftState":"StatePreCandidate","applied":79,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/46]
E240701 15:54:30.616519 5488311 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r51/2:/Table/{48-50}] 3136  [n2,s2,r51/2:/Table/{48-50}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r51:/Table/{48-50} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848906.098546590,0 (2024-07-01 15:48:26); raft status: {"id":"2","term":7,"vote":"2","commit":76,"lead":"0","raftState":"StatePreCandidate","applied":76,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/48]
E240701 15:54:30.617007 5488312 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r45/3:/Table/4{2-3}] 3137  [n2,s2,r45/3:/Table/4{2-3}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r45:/Table/4{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848904.936319431,0 (2024-07-01 15:48:24); raft status: {"id":"3","term":8,"vote":"1","commit":73,"lead":"0","raftState":"StatePreCandidate","applied":73,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/42]
E240701 15:54:30.617576 5488313 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r3/3:/System/{NodeLive…-tsd}] 3138  [n2,s2,r3/3:/System/{NodeLive…-tsd}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848901.430074218,0 (2024-07-01 15:48:21); raft status: {"id":"3","term":6,"vote":"0","commit":225,"lead":"0","raftState":"StatePreCandidate","applied":225,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/System/NodeLivenessMax]
E240701 15:54:30.618056 5488314 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r47/2:/Table/4{4-5}] 3139  [n2,s2,r47/2:/Table/4{4-5}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r47:/Table/4{4-5} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848902.011087481,0 (2024-07-01 15:48:22); raft status: {"id":"2","term":7,"vote":"2","commit":79,"lead":"0","raftState":"StatePreCandidate","applied":79,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/44]
E240701 15:54:30.618714 5488315 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r35/3:/Table/3{2-3}] 3140  [n2,s2,r35/3:/Table/3{2-3}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r35:/Table/3{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848904.936319431,0 (2024-07-01 15:48:24); raft status: {"id":"3","term":8,"vote":"1","commit":75,"lead":"0","raftState":"StatePreCandidate","applied":75,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/32]
E240701 15:54:30.619300 5488316 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r30/2:/Table/2{7-8}] 3141  [n2,s2,r30/2:/Table/2{7-8}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r30:/Table/2{7-8} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848906.098546590,0 (2024-07-01 15:48:26); raft status: {"id":"2","term":8,"vote":"2","commit":81,"lead":"0","raftState":"StatePreCandidate","applied":81,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/27]
E240701 15:54:30.619917 5488317 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r6/3:/Table/{0-3}] 3142  [n2,s2,r6/3:/Table/{0-3}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r6:/Table/{0-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848905.151807289,0 (2024-07-01 15:48:25); raft status: {"id":"3","term":8,"vote":"3","commit":74,"lead":"0","raftState":"StatePreCandidate","applied":74,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/0]
E240701 15:54:30.620473 5488318 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r46/3:/Table/4{3-4}] 3143  [n2,s2,r46/3:/Table/4{3-4}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r46:/Table/4{3-4} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"3","term":6,"vote":"0","commit":73,"lead":"0","raftState":"StatePreCandidate","applied":73,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/43]
E240701 15:54:30.620991 5488319 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r25/2:/Table/2{2-3}] 3144  [n2,s2,r25/2:/Table/2{2-3}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r25:/Table/2{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848900.193119688,0 (2024-07-01 15:48:20); raft status: {"id":"2","term":9,"vote":"2","commit":74,"lead":"0","raftState":"StatePreCandidate","applied":74,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/22]
E240701 15:54:30.621527 5488320 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r31/2:/Table/2{8-9}] 3145  [n2,s2,r31/2:/Table/2{8-9}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r31:/Table/2{8-9} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"2","term":7,"vote":"1","commit":93,"lead":"0","raftState":"StatePreCandidate","applied":93,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/28]
E240701 15:54:30.622046 5488321 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r67/3:/Table/6{5-6}] 3146  [n2,s2,r67/3:/Table/6{5-6}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r67:/Table/6{5-6} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"3","term":6,"vote":"0","commit":73,"lead":"0","raftState":"StatePreCandidate","applied":73,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/65]
E240701 15:54:30.622531 5488322 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r37/2:/Table/3{4-5}] 3147  [n2,s2,r37/2:/Table/3{4-5}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r37:/Table/3{4-5} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848900.193119688,0 (2024-07-01 15:48:20); raft status: {"id":"2","term":9,"vote":"2","commit":74,"lead":"0","raftState":"StatePreCandidate","applied":74,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/34]
E240701 15:54:30.623017 5488323 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r33/2:/NamespaceTable/{30-Max}] 3148  [n2,s2,r33/2:/NamespaceTable/{30-Max}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r33:/NamespaceTable/{30-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848900.193119688,0 (2024-07-01 15:48:20); raft status: {"id":"2","term":10,"vote":"2","commit":82,"lead":"0","raftState":"StatePreCandidate","applied":82,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/NamespaceTable/30]
E240701 15:54:30.623485 5488324 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r21/2:/Table/1{8-9}] 3149  [n2,s2,r21/2:/Table/1{8-9}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r21:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"2","term":7,"vote":"1","commit":74,"lead":"0","raftState":"StatePreCandidate","applied":74,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/18]
E240701 15:54:30.623968 5488325 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,replicate,s2,r36/3:/Table/3{3-4}] 3150  [n2,s2,r36/3:/Table/3{3-4}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r36:/Table/3{3-4} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848905.151807289,0 (2024-07-01 15:48:25); raft status: {"id":"3","term":7,"vote":"0","commit":75,"lead":"0","raftState":"StatePreCandidate","applied":75,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/33]
E240701 15:54:30.639824 5488336 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,lease,s2,r66/3:/Table/6{4-5}] 3151  [n2,s2,r66/3:/Table/6{4-5}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r66:/Table/6{4-5} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848907.475846250,0 (2024-07-01 15:48:27); raft status: {"id":"3","term":7,"vote":"1","commit":74,"lead":"0","raftState":"StatePreCandidate","applied":74,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/64]
E240701 15:54:30.640322 5488337 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,lease,s2,r19/3:/Table/1{6-7}] 3152  [n2,s2,r19/3:/Table/1{6-7}]: could not obtain lease: replica unavailable: (n2,s2):3 unable to serve request to r19:/Table/1{6-7} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1719848904.936319431,0 (2024-07-01 15:48:24); raft status: {"id":"3","term":8,"vote":"1","commit":75,"lead":"0","raftState":"StatePreCandidate","applied":75,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/16]
E240701 15:54:30.640742 5488338 kv/kvserver/queue.go:1202  [T1,Vsystem,n2,lease,s2,r54/2:/Table/5{2-3}] 3153  [n2,s2,r54/2:/Table/5{2-3}]: could not obtain lease: replica unavailable: (n2,s2):2 unable to serve request to r54:/Table/5{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):3); closed timestamp: 1719848903.661861728,0 (2024-07-01 15:48:23); raft status: {"id":"2","term":7,"vote":"2","commit":77,"lead":"0","raftState":"StatePreCandidate","applied":77,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/52]
E240701 15:54:30.658149 5488339 kv/kvserver/queue.go:1202  [T1,Vsystem,n1,replicaGC,s1,r45/1:/Table/4{2-3}] 3154  replica unavailable: (n1,s1):1 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n2,s2):2,(n3,s3):3); closed timestamp: 1719848904.895442203,0 (2024-07-01 15:48:24); raft status: {"id":"1","term":9,"vote":"2","commit":609,"lead":"0","raftState":"StatePreCandidate","applied":609,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Min]
--- FAIL: TestDebugCheckStore (528.51s)

Parameters:

  • attempt=1
  • run=1
  • shard=6
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@arulajmani arulajmani self-assigned this Jul 3, 2024
@arulajmani arulajmani mentioned this issue Jul 3, 2024
Merged
craig bot pushed a commit that referenced this issue Jul 8, 2024
@arulajmani
Merge #126690
d51907a 
126690: cli: bump default worker pool to heavy r=arulajmani a=arulajmani

For the benefit of TestDebugCheckStore.

Closes #122328
Closes #126593

Epic: none

Release note: None

Co-authored-by: Arul Ajmani <[email protected]>
@craig craig bot closed this as completed in f012fe4 Jul 8, 2024
blathers-crl bot pushed a commit that referenced this issue Jul 8, 2024
@arulajmani
cli: bump default worker pool to heavy
683ec53 
For the benefit of TestDebugCheckStore.

Closes #122328
Closes #126593

Epic: none

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Jul 8, 2024
Merged
asg0451 pushed a commit to asg0451/cockroach that referenced this issue Jul 10, 2024
@arulajmani @asg0451
cli: bump default worker pool to heavy
73de754 
For the benefit of TestDebugCheckStore.

Closes cockroachdb#122328
Closes cockroachdb#126593

Epic: none

Release note: None
@github-project-automation github-project-automation bot moved this to roachtest/unit test backlog in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arulajmani arulajmani

Labels
branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Projects
No open projects
KV
Status: roachtest/unit test backlog
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

cli: bump default worker pool to heavy arulajmani/cockroach
5 participants
@nicktrav @arulajmani @cockroach-teamcity @msbutler @miraradeva