Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cli: TestRemoveDeadReplicas failed #80956

Closed
cockroach-teamcity opened this issue May 4, 2022 · 7 comments
Closed

cli: TestRemoveDeadReplicas failed #80956

cockroach-teamcity opened this issue May 4, 2022 · 7 comments
Assignees
@aliher1911
Labels
A-kv-recovery branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 4, 2022 •
edited by cockroach-jira-scripts
Loading

cli.TestRemoveDeadReplicas failed with artifacts on release-22.1 @ 6c4e59fd534de518de1042b3635052029a404ae0:

not designated survivor, skipping: r41:/Table/4{5-6} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6]
not designated survivor, skipping: r42:/Table/4{6-7} [(n2,s2):4, (n4,s4):2, (n1,s1):5, (n3,s3):6, next=7, gen=14]
not designated survivor, skipping: r43:/Table/{47-50} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6]
not designated survivor, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6]
Scanning replicas on store cluster_id:02d7a502-9c1f-4cf0-8d4b-114b7475dadf node_id:2 store_id:2  for dead peers []
replica has lost quorum, recovering: r1:/{Min-System/NodeLiveness} [(n1,s1):5, (n2,s2):2, (n4,s4):3, (n3,s3):6, next=7, gen=14] -> r1:/{Min-System/NodeLiveness} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r2:/System/NodeLiveness{-Max} [(n2,s2):4, (n4,s4):2, (n3,s3):6, (n1,s1):7, next=8, gen=18] -> r2:/System/NodeLiveness{-Max} [(n2,s2):8, next=9, gen=18]
replica has lost quorum, recovering: r3:/System/{NodeLivenessMax-tsd} [(n2,s2):4, (n3,s3):2, (n4,s4):8, (n1,s1):9, next=10, gen=26] -> r3:/System/{NodeLivenessMax-tsd} [(n2,s2):10, next=11, gen=26]
replica has lost quorum, recovering: r4:/System{/tsd-tse} [(n2,s2):4, (n3,s3):2, (n4,s4):3, (n1,s1):5, next=6, gen=10] -> r4:/System{/tsd-tse} [(n2,s2):6, next=7, gen=10]
replica has lost quorum, recovering: r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r6:/Table/{SystemConfigSpan/Start-11} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r7:/Table/1{1-2} [(n1,s1):5, (n2,s2):2, (n4,s4):3, (n3,s3):6, next=7, gen=14] -> r7:/Table/1{1-2} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r8:/Table/1{2-3} [(n2,s2):4, (n3,s3):2, (n4,s4):6, (n1,s1):7, next=8, gen=18] -> r8:/Table/1{2-3} [(n2,s2):8, next=9, gen=18]
replica has lost quorum, recovering: r9:/Table/1{3-4} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r9:/Table/1{3-4} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r10:/Table/1{4-5} [(n3,s3):4, (n2,s2):2, (n1,s1):5, (n4,s4):6, next=7, gen=14] -> r10:/Table/1{4-5} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r11:/Table/1{5-6} [(n2,s2):4, (n3,s3):2, (n4,s4):6, (n1,s1):7, next=8, gen=18] -> r11:/Table/1{5-6} [(n2,s2):8, next=9, gen=18]
replica has lost quorum, recovering: r12:/Table/1{6-7} [(n2,s2):4, (n3,s3):8, (n1,s1):9, (n4,s4):10, next=11, gen=30] -> r12:/Table/1{6-7} [(n2,s2):11, next=12, gen=30]
replica has lost quorum, recovering: r13:/Table/1{7-8} [(n2,s2):4, (n4,s4):10, (n3,s3):3, (n1,s1):11, next=12, gen=34] -> r13:/Table/1{7-8} [(n2,s2):12, next=13, gen=34]
replica has lost quorum, recovering: r14:/Table/1{8-9} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r14:/Table/1{8-9} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r15:/Table/{19-20} [(n1,s1):5, (n4,s4):2, (n3,s3):3, (n2,s2):6, next=7, gen=14] -> r15:/Table/{19-20} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r16:/Table/2{0-1} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r16:/Table/2{0-1} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r17:/Table/2{1-2} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r17:/Table/2{1-2} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r18:/Table/2{2-3} [(n1,s1):1, (n2,s2):2, (n4,s4):3, (n3,s3):4, next=5, gen=6] -> r18:/Table/2{2-3} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r19:/Table/2{3-4} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r19:/Table/2{3-4} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r20:/Table/2{4-5} [(n1,s1):5, (n3,s3):2, (n2,s2):3, (n4,s4):6, next=7, gen=14] -> r20:/Table/2{4-5} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r21:/Table/2{5-6} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6] -> r21:/Table/2{5-6} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r22:/Table/2{6-7} [(n2,s2):4, (n3,s3):2, (n4,s4):3, (n1,s1):5, next=6, gen=10] -> r22:/Table/2{6-7} [(n2,s2):6, next=7, gen=10]
replica has lost quorum, recovering: r23:/Table/2{7-8} [(n1,s1):5, (n2,s2):2, (n3,s3):3, (n4,s4):6, next=7, gen=14] -> r23:/Table/2{7-8} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r24:/Table/2{8-9} [(n3,s3):4, (n1,s1):5, (n2,s2):3, (n4,s4):6, next=7, gen=14] -> r24:/Table/2{8-9} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r25:/{Table/29-NamespaceTable/30} [(n2,s2):4, (n3,s3):2, (n4,s4):6, (n1,s1):7, next=8, gen=18] -> r25:/{Table/29-NamespaceTable/30} [(n2,s2):8, next=9, gen=18]
replica has lost quorum, recovering: r26:/NamespaceTable/{30-Max} [(n2,s2):4, (n3,s3):2, (n4,s4):3, (n1,s1):5, next=6, gen=10] -> r26:/NamespaceTable/{30-Max} [(n2,s2):6, next=7, gen=10]
replica has lost quorum, recovering: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r27:/{NamespaceTable/Max-Table/32} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r28:/Table/3{2-3} [(n1,s1):1, (n4,s4):4, (n2,s2):3, (n3,s3):5, next=6, gen=10] -> r28:/Table/3{2-3} [(n2,s2):6, next=7, gen=10]
replica has lost quorum, recovering: r29:/Table/3{3-4} [(n3,s3):4, (n4,s4):2, (n1,s1):5, (n2,s2):6, next=7, gen=14] -> r29:/Table/3{3-4} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r30:/Table/3{4-5} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r30:/Table/3{4-5} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r31:/Table/3{5-6} [(n2,s2):4, (n3,s3):2, (n1,s1):9, (n4,s4):10, next=11, gen=30] -> r31:/Table/3{5-6} [(n2,s2):11, next=12, gen=30]
replica has lost quorum, recovering: r32:/Table/3{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, (n3,s3):4, next=5, gen=6] -> r32:/Table/3{6-7} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r33:/Table/3{7-8} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r33:/Table/3{7-8} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r35:/Table/{39-40} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6] -> r35:/Table/{39-40} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r36:/Table/4{0-1} [(n4,s4):6, (n3,s3):2, (n2,s2):3, (n1,s1):7, next=8, gen=18] -> r36:/Table/4{0-1} [(n2,s2):8, next=9, gen=18]
replica has lost quorum, recovering: r37:/Table/4{1-2} [(n3,s3):4, (n1,s1):17, (n2,s2):3, (n4,s4):18, next=19, gen=62] -> r37:/Table/4{1-2} [(n2,s2):19, next=20, gen=62]
replica has lost quorum, recovering: r38:/Table/4{2-3} [(n1,s1):5, (n2,s2):2, (n3,s3):3, (n4,s4):6, next=7, gen=14] -> r38:/Table/4{2-3} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r39:/Table/4{3-4} [(n2,s2):4, (n4,s4):14, (n3,s3):3, (n1,s1):15, next=16, gen=50] -> r39:/Table/4{3-4} [(n2,s2):16, next=17, gen=50]
replica has lost quorum, recovering: r40:/Table/4{4-5} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r40:/Table/4{4-5} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r41:/Table/4{5-6} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4, next=5, gen=6] -> r41:/Table/4{5-6} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r42:/Table/4{6-7} [(n2,s2):4, (n4,s4):2, (n1,s1):5, (n3,s3):6, next=7, gen=14] -> r42:/Table/4{6-7} [(n2,s2):7, next=8, gen=14]
replica has lost quorum, recovering: r43:/Table/{47-50} [(n1,s1):1, (n4,s4):2, (n3,s3):3, (n2,s2):4, next=5, gen=6] -> r43:/Table/{47-50} [(n2,s2):5, next=6, gen=6]
replica has lost quorum, recovering: r44:/{Table/50-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, (n4,s4):4, next=5, gen=6] -> r44:/{Table/50-Max} [(n2,s2):5, next=6, gen=6]
aborting intent: /Local/Range/System/tsd/RangeDescriptor (txn ac492bd6-6762-46e1-85ae-06a356a7bbab)
Scanning replicas on store cluster_id:02d7a502-9c1f-4cf0-8d4b-114b7475dadf node_id:2 store_id:2  for dead peers []
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

Same failure on other branches

/cc @cockroachdb/server @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-15432
@cockroach-teamcity cockroach-teamcity added branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 4, 2022
@blathers-crl blathers-crl bot added the T-server-and-security DB Server & Security label May 4, 2022
@cockroach-teamcity
Copy link
Member Author

cli.TestRemoveDeadReplicas failed with artifacts on release-22.1 @ 31c42eb6bd8eb5e57c29abc02e664b18c278166d:

replica has not lost quorum, skipping: r7:/Table/1{1-2} [(n1,s1):5, (n3,s3):2, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r8:/Table/1{2-3} [(n1,s1):5, (n2,s2):2, (n4,s4):3, next=6, gen=12]
replica has not lost quorum, skipping: r11:/Table/1{5-6} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r12:/Table/1{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r13:/Table/1{7-8} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r15:/Table/{19-20} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n4,s4):4, (n1,s1):5, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r17:/Table/2{1-2} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r18:/Table/2{2-3} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r18:/Table/2{2-3} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r19:/Table/2{3-4} [(n1,s1):5, (n2,s2):2, (n4,s4):3, next=6, gen=12]
replica has lost quorum, recovering: r21:/Table/2{5-6} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r21:/Table/2{5-6} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r22:/Table/2{6-7} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r22:/Table/2{6-7} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r23:/Table/2{7-8} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r23:/Table/2{7-8} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n4,s4):4, (n1,s1):5, (n2,s2):3, next=6, gen=12]
replica has lost quorum, recovering: r25:/{Table/29-NamespaceTable/30} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r25:/{Table/29-NamespaceTable/30} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r26:/NamespaceTable/{30-Max} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r26:/NamespaceTable/{30-Max} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r28:/Table/3{2-3} [(n3,s3):4, (n2,s2):2, (n1,s1):5, next=6, gen=12]
replica has lost quorum, recovering: r29:/Table/3{3-4} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r29:/Table/3{3-4} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r33:/Table/3{7-8} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r33:/Table/3{7-8} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r35:/Table/{39-40} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r35:/Table/{39-40} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r36:/Table/4{0-1} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r37:/Table/4{1-2} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r37:/Table/4{1-2} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r39:/Table/4{3-4} [(n4,s4):4, (n2,s2):2, (n1,s1):5, next=6, gen=12]
replica has lost quorum, recovering: r40:/Table/4{4-5} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r40:/Table/4{4-5} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n1,s1):5, (n2,s2):2, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
aborting intent: /Local/Range/Table/29/RangeDescriptor (txn 0749d194-8af7-46e8-898c-c6dbacc1a69e)
Scanning replicas on store cluster_id:1626391f-57a8-4b9d-9aeb-a02d78a21752 node_id:2 store_id:2  for dead peers []
replica has not lost quorum, skipping: r4:/System{/tsd-tse} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r7:/Table/1{1-2} [(n1,s1):5, (n3,s3):2, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r8:/Table/1{2-3} [(n1,s1):5, (n2,s2):2, (n4,s4):3, next=6, gen=12]
replica has not lost quorum, skipping: r11:/Table/1{5-6} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r12:/Table/1{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r13:/Table/1{7-8} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r15:/Table/{19-20} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n4,s4):4, (n1,s1):5, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r17:/Table/2{1-2} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r19:/Table/2{3-4} [(n1,s1):5, (n2,s2):2, (n4,s4):3, next=6, gen=12]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n4,s4):4, (n1,s1):5, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r28:/Table/3{2-3} [(n3,s3):4, (n2,s2):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r36:/Table/4{0-1} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r39:/Table/4{3-4} [(n4,s4):4, (n2,s2):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n1,s1):5, (n2,s2):2, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 10, 2022
Closed
@jlinder jlinder added sync-me and removed sync-me labels May 20, 2022
@cockroach-teamcity
Copy link
Member Author

cli.TestRemoveDeadReplicas failed with artifacts on release-22.1 @ 4e5b7aef63f7594f7d56eb24362b9bf78b39e9a2:

replica has not lost quorum, skipping: r9:/Table/1{3-4} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r12:/Table/1{6-7} [(n3,s3):4, (n4,s4):2, (n2,s2):3, next=5, gen=8] -> r12:/Table/1{6-7} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r13:/Table/1{7-8} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n2,s2):4, (n4,s4):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r15:/Table/{19-20} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has lost quorum, recovering: r16:/Table/2{0-1} [(n3,s3):4, (n4,s4):2, (n2,s2):3, next=5, gen=8] -> r16:/Table/2{0-1} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r17:/Table/2{1-2} [(n3,s3):4, (n4,s4):2, (n2,s2):3, next=5, gen=8] -> r17:/Table/2{1-2} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r19:/Table/2{3-4} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r21:/Table/2{5-6} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r22:/Table/2{6-7} [(n1,s1):5, (n4,s4):2, (n2,s2):3, next=6, gen=12]
replica has lost quorum, recovering: r23:/Table/2{7-8} [(n3,s3):4, (n4,s4):2, (n2,s2):3, next=5, gen=8] -> r23:/Table/2{7-8} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r25:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r26:/NamespaceTable/{30-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has lost quorum, recovering: r30:/Table/3{4-5} [(n3,s3):4, (n4,s4):2, (n2,s2):3, next=5, gen=8] -> r30:/Table/3{4-5} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r31:/Table/3{5-6} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r32:/Table/3{6-7} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r32:/Table/3{6-7} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r33:/Table/3{7-8} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has lost quorum, recovering: r35:/Table/{39-40} [(n3,s3):4, (n2,s2):2, (n4,s4):3, next=5, gen=8] -> r35:/Table/{39-40} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r37:/Table/4{1-2} [(n3,s3):4, (n4,s4):2, (n2,s2):3, next=5, gen=8] -> r37:/Table/4{1-2} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r39:/Table/4{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has lost quorum, recovering: r40:/Table/4{4-5} [(n3,s3):4, (n4,s4):2, (n2,s2):3, next=5, gen=8] -> r40:/Table/4{4-5} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r44:/{Table/50-Max} [(n3,s3):4, (n4,s4):2, (n2,s2):3, next=5, gen=8] -> r44:/{Table/50-Max} [(n2,s2):5, next=6, gen=8]
aborting intent: /Local/Range/System/tsd/RangeDescriptor (txn cc5bbd15-662b-4a60-b392-346ea6210a23)
aborting intent: /Local/Range/Table/34/RangeDescriptor (txn cae0c45d-0ac9-4634-9f7d-fd2bef5a9ccd)
aborting intent: /Local/Range/Table/36/RangeDescriptor (txn 3c59a3a0-4ffd-4bc7-bf18-e414ee8d14c6)
Scanning replicas on store cluster_id:5f7bdd48-91fd-4d2e-a5b3-75809aaf81db node_id:2 store_id:2  for dead peers []
replica has not lost quorum, skipping: r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r3:/System/{NodeLivenessMax-tsd} [(n1,s1):5, (n2,s2):2, (n4,s4):3, next=6, gen=12]
replica has not lost quorum, skipping: r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r7:/Table/1{1-2} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r9:/Table/1{3-4} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r13:/Table/1{7-8} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n2,s2):4, (n4,s4):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r15:/Table/{19-20} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r19:/Table/2{3-4} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r21:/Table/2{5-6} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r22:/Table/2{6-7} [(n1,s1):5, (n4,s4):2, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r25:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r26:/NamespaceTable/{30-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r31:/Table/3{5-6} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r33:/Table/3{7-8} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r39:/Table/4{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

cli.TestRemoveDeadReplicas failed with artifacts on release-22.1 @ 3181b7faa2f3b41d6a15ab4b74d2c60bcfe5132d:

replica has lost quorum, recovering: r9:/Table/1{3-4} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r9:/Table/1{3-4} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r11:/Table/1{5-6} [(n3,s3):4, (n2,s2):2, (n4,s4):3, next=5, gen=8] -> r11:/Table/1{5-6} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r12:/Table/1{6-7} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r13:/Table/1{7-8} [(n1,s1):1, (n3,s3):2, (n2,s2):4, next=5, gen=8]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r17:/Table/2{1-2} [(n2,s2):4, (n3,s3):10, (n4,s4):3, next=11, gen=32] -> r17:/Table/2{1-2} [(n2,s2):11, next=12, gen=32]
replica has lost quorum, recovering: r18:/Table/2{2-3} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r18:/Table/2{2-3} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r20:/Table/2{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has lost quorum, recovering: r22:/Table/2{6-7} [(n4,s4):4, (n3,s3):8, (n2,s2):3, next=9, gen=24] -> r22:/Table/2{6-7} [(n2,s2):9, next=10, gen=24]
replica has lost quorum, recovering: r23:/Table/2{7-8} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r23:/Table/2{7-8} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r26:/NamespaceTable/{30-Max} [(n4,s4):4, (n1,s1):9, (n2,s2):3, next=10, gen=28]
replica has not lost quorum, skipping: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has lost quorum, recovering: r28:/Table/3{2-3} [(n2,s2):4, (n3,s3):8, (n4,s4):3, next=9, gen=24] -> r28:/Table/3{2-3} [(n2,s2):9, next=10, gen=24]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n4,s4):4, (n1,s1):7, (n2,s2):3, next=8, gen=20]
replica has not lost quorum, skipping: r30:/Table/3{4-5} [(n2,s2):4, (n4,s4):2, (n1,s1):15, next=16, gen=52]
replica has not lost quorum, skipping: r31:/Table/3{5-6} [(n2,s2):4, (n1,s1):7, (n4,s4):3, next=8, gen=20]
replica has not lost quorum, skipping: r32:/Table/3{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r35:/Table/{39-40} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r36:/Table/4{0-1} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r36:/Table/4{0-1} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r37:/Table/4{1-2} [(n2,s2):4, (n3,s3):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r38:/Table/4{2-3} [(n1,s1):5, (n3,s3):2, (n2,s2):3, next=6, gen=12]
replica has lost quorum, recovering: r39:/Table/4{3-4} [(n2,s2):4, (n3,s3):2, (n4,s4):3, (n1,s1):5LEARNER, next=6, gen=9] -> r39:/Table/4{3-4} [(n2,s2):6, next=7, gen=9]
replica has lost quorum, recovering: r40:/Table/4{4-5} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r40:/Table/4{4-5} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r41:/Table/4{5-6} [(n1,s1):5, (n4,s4):2, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n3,s3):4, (n2,s2):3, next=5, gen=8]
aborting intent: /Local/Range/System/"tse"/RangeDescriptor (txn e7f9613c-97a5-47f0-b0cd-b95e6896903f)
Scanning replicas on store cluster_id:64dba09e-1f44-4a2e-8939-c422beb5b059 node_id:2 store_id:2  for dead peers []
replica has not lost quorum, skipping: r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r8:/Table/1{2-3} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r12:/Table/1{6-7} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r13:/Table/1{7-8} [(n1,s1):1, (n3,s3):2, (n2,s2):4, next=5, gen=8]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r20:/Table/2{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r26:/NamespaceTable/{30-Max} [(n4,s4):4, (n1,s1):9, (n2,s2):3, next=10, gen=28]
replica has not lost quorum, skipping: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n4,s4):4, (n1,s1):7, (n2,s2):3, next=8, gen=20]
replica has not lost quorum, skipping: r30:/Table/3{4-5} [(n2,s2):4, (n4,s4):2, (n1,s1):15, next=16, gen=52]
replica has not lost quorum, skipping: r31:/Table/3{5-6} [(n2,s2):4, (n1,s1):7, (n4,s4):3, next=8, gen=20]
replica has not lost quorum, skipping: r32:/Table/3{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r35:/Table/{39-40} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r37:/Table/4{1-2} [(n2,s2):4, (n3,s3):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r38:/Table/4{2-3} [(n1,s1):5, (n3,s3):2, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r41:/Table/4{5-6} [(n1,s1):5, (n4,s4):2, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n3,s3):4, (n2,s2):3, next=5, gen=8]
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

Same failure on other branches

This test on roachdash | Improve this report!

@blathers-crl blathers-crl bot added the T-kv KV Team label Jun 16, 2022
@knz knz added A-kv-recovery and removed T-server-and-security DB Server & Security labels Jun 16, 2022
@cockroach-teamcity
Copy link
Member Author

cli.TestRemoveDeadReplicas failed with artifacts on release-22.1 @ 31c6f8ccfdc8eed04ddeb3e2d8e59646d17365ee:

replica has not lost quorum, skipping: r8:/Table/1{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r9:/Table/1{3-4} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r10:/Table/1{4-5} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r10:/Table/1{4-5} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r11:/Table/1{5-6} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r11:/Table/1{5-6} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r12:/Table/1{6-7} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r12:/Table/1{6-7} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r19:/Table/2{3-4} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r19:/Table/2{3-4} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r20:/Table/2{4-5} [(n1,s1):5, (n2,s2):2, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r21:/Table/2{5-6} [(n1,s1):5, (n3,s3):2, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r22:/Table/2{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r23:/Table/2{7-8} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r25:/{Table/29-NamespaceTable/30} [(n2,s2):4, (n3,s3):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r26:/NamespaceTable/{30-Max} [(n2,s2):4, (n3,s3):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r28:/Table/3{2-3} [(n4,s4):4, (n2,s2):2, (n3,s3):3, (n1,s1):5LEARNER, next=6, gen=9] -> r28:/Table/3{2-3} [(n2,s2):6, next=7, gen=9]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r31:/Table/3{5-6} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r31:/Table/3{5-6} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r32:/Table/3{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r33:/Table/3{7-8} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r34:/Table/3{8-9} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has lost quorum, recovering: r35:/Table/{39-40} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r35:/Table/{39-40} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r36:/Table/4{0-1} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r36:/Table/4{0-1} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r38:/Table/4{2-3} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r38:/Table/4{2-3} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r42:/Table/4{6-7} [(n4,s4):4, (n3,s3):2, (n2,s2):3, next=5, gen=8] -> r42:/Table/4{6-7} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r43:/Table/{47-50} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):5, (n3,s3):2, (n2,s2):3, next=6, gen=12]
aborting intent: /Local/Range/Table/39/RangeDescriptor (txn 7c9a4816-c284-4bd4-9f9b-6993edf5b941)
Scanning replicas on store cluster_id:61d65b97-c1c3-407e-a1a3-5834ac90ae8c node_id:2 store_id:2  for dead peers []
replica has not lost quorum, skipping: r3:/System/{NodeLivenessMax-tsd} [(n2,s2):4, (n3,s3):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r7:/Table/1{1-2} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r8:/Table/1{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r9:/Table/1{3-4} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r20:/Table/2{4-5} [(n1,s1):5, (n2,s2):2, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r21:/Table/2{5-6} [(n1,s1):5, (n3,s3):2, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r22:/Table/2{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r23:/Table/2{7-8} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r25:/{Table/29-NamespaceTable/30} [(n2,s2):4, (n3,s3):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r26:/NamespaceTable/{30-Max} [(n2,s2):4, (n3,s3):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r32:/Table/3{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r33:/Table/3{7-8} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r34:/Table/3{8-9} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r43:/Table/{47-50} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):5, (n3,s3):2, (n2,s2):3, next=6, gen=12]
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss

Same failure on other branches

This test on roachdash | Improve this report!

@blathers-crl
Copy link

blathers-crl bot commented Jun 24, 2022

cc @cockroachdb/replication

@blathers-crl blathers-crl bot added the A-kv-replication Relating to Raft, consensus, and coordination. label Jun 24, 2022
@erikgrinaker erikgrinaker removed A-kv-replication Relating to Raft, consensus, and coordination. T-kv KV Team labels Jun 24, 2022
@tbg tbg mentioned this issue Jun 27, 2022
Merged
tbg added a commit to tbg/cockroach that referenced this issue Jun 27, 2022
@tbg
cli: skip TestRemoveDeadReplicas
0a1fe89 
See cockroachdb#80956

Release note: None
@cockroach-teamcity
Copy link
Member Author

cli.TestRemoveDeadReplicas failed with artifacts on release-22.1 @ c5100d0c34cf1c56276f45ef8bfb1557ddf68009:

replica has not lost quorum, skipping: r10:/Table/1{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has lost quorum, recovering: r12:/Table/1{6-7} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r12:/Table/1{6-7} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r13:/Table/1{7-8} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r13:/Table/1{7-8} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r15:/Table/{19-20} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r18:/Table/2{2-3} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r18:/Table/2{2-3} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r19:/Table/2{3-4} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r21:/Table/2{5-6} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r23:/Table/2{7-8} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r25:/{Table/29-NamespaceTable/30} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r25:/{Table/29-NamespaceTable/30} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r26:/NamespaceTable/{30-Max} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r26:/NamespaceTable/{30-Max} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r28:/Table/3{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has lost quorum, recovering: r30:/Table/3{4-5} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r30:/Table/3{4-5} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r32:/Table/3{6-7} [(n2,s2):4, (n4,s4):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r33:/Table/3{7-8} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has lost quorum, recovering: r34:/Table/3{8-9} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r34:/Table/3{8-9} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r35:/Table/{39-40} [(n2,s2):4, (n1,s1):7, (n3,s3):6, next=8, gen=20]
replica has lost quorum, recovering: r36:/Table/4{0-1} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r36:/Table/4{0-1} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r37:/Table/4{1-2} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r37:/Table/4{1-2} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r39:/Table/4{3-4} [(n2,s2):4, (n1,s1):5, (n3,s3):3, next=6, gen=12]
replica has lost quorum, recovering: r40:/Table/4{4-5} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r40:/Table/4{4-5} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r41:/Table/4{5-6} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r41:/Table/4{5-6} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r43:/Table/{47-50} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
aborting intent: /Local/Range/System/tsd/RangeDescriptor (txn cbc450ce-b0d8-40e9-8f3d-c226605245ab)
aborting intent: /Local/Range/Table/44/RangeDescriptor (txn ae59c847-2e03-4aea-8d9f-049e10f2b631)
Scanning replicas on store cluster_id:fe388dfa-5050-4011-9097-aa4d2b2ca278 node_id:2 store_id:2  for dead peers []
replica has not lost quorum, skipping: r1:/{Min-System/NodeLiveness} [(n1,s1):5, (n4,s4):2, (n2,s2):3, next=6, gen=12]
replica has not lost quorum, skipping: r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r7:/Table/1{1-2} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r8:/Table/1{2-3} [(n4,s4):4, (n2,s2):2, (n1,s1):7, next=8, gen=20]
replica has not lost quorum, skipping: r10:/Table/1{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r15:/Table/{19-20} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r19:/Table/2{3-4} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r21:/Table/2{5-6} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r23:/Table/2{7-8} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r28:/Table/3{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r32:/Table/3{6-7} [(n2,s2):4, (n4,s4):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r33:/Table/3{7-8} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r35:/Table/{39-40} [(n2,s2):4, (n1,s1):7, (n3,s3):6, next=8, gen=20]
replica has not lost quorum, skipping: r39:/Table/4{3-4} [(n2,s2):4, (n1,s1):5, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r43:/Table/{47-50} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

Same failure on other branches

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Jun 29, 2022
@tbg
Merge #83446
dd1989b 
83446: cli: skip TestRemoveDeadReplicas r=erikgrinaker a=tbg

See #80956

Release note: None


Co-authored-by: Tobias Grieger <[email protected]>
@erikgrinaker
Copy link
Contributor

This command will be deprecated in 22.2, and removed in 23.1. The test has been skipped, which should be sufficient for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aliher1911 aliher1911

Labels
A-kv-recovery branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jlinder @knz @erikgrinaker @tbg @aliher1911 @cockroach-teamcity