Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cli: TestRemoveDeadReplicas failed #80532

Closed
cockroach-teamcity opened this issue Apr 26, 2022 · 3 comments
Closed

cli: TestRemoveDeadReplicas failed #80532

cockroach-teamcity opened this issue Apr 26, 2022 · 3 comments
Assignees
@aliher1911
Labels
C-test-failure Broken test (automatically or manually discovered). deprecated-branch-release-22.1.0 O-robot Originated from a bot. S-3 Medium-low impact: incurs increased costs for some users (incl lower avail, recoverable bad data)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 26, 2022 •
edited by cockroach-jira-scripts
Loading

cli.TestRemoveDeadReplicas failed with artifacts on release-22.1.0 @ b7681be8d92aca93546d9abd0a72a147a4b52ff2:

replica has lost quorum, recovering: r9:/Table/1{3-4} [(n2,s2):4, (n3,s3):6, (n4,s4):3, next=7, gen=16] -> r9:/Table/1{3-4} [(n2,s2):7, next=8, gen=16]
replica has not lost quorum, skipping: r10:/Table/1{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r12:/Table/1{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r15:/Table/{19-20} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n2,s2):4, (n1,s1):5, (n3,s3):3, next=6, gen=12]
replica has lost quorum, recovering: r17:/Table/2{1-2} [(n4,s4):4, (n3,s3):2, (n2,s2):6, next=7, gen=16] -> r17:/Table/2{1-2} [(n2,s2):7, next=8, gen=16]
replica has lost quorum, recovering: r18:/Table/2{2-3} [(n2,s2):4, (n4,s4):6, (n3,s3):3, next=7, gen=16] -> r18:/Table/2{2-3} [(n2,s2):7, next=8, gen=16]
replica has not lost quorum, skipping: r19:/Table/2{3-4} [(n2,s2):4, (n3,s3):2, (n1,s1):7, next=8, gen=20]
replica has not lost quorum, skipping: r20:/Table/2{4-5} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r22:/Table/2{6-7} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r23:/Table/2{7-8} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r26:/NamespaceTable/{30-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has lost quorum, recovering: r28:/Table/3{2-3} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r28:/Table/3{2-3} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n1,s1):5, (n2,s2):2, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r30:/Table/3{4-5} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r31:/Table/3{5-6} [(n3,s3):4, (n2,s2):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r33:/Table/3{7-8} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r35:/Table/{39-40} [(n4,s4):4, (n1,s1):7, (n2,s2):3, next=8, gen=20]
replica has lost quorum, recovering: r36:/Table/4{0-1} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r36:/Table/4{0-1} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r38:/Table/4{2-3} [(n4,s4):4, (n2,s2):6, (n3,s3):3, next=7, gen=16] -> r38:/Table/4{2-3} [(n2,s2):7, next=8, gen=16]
not designated survivor, skipping: r41:/Table/4{5-6} [(n1,s1):6, (n4,s4):2, (n3,s3):4, (n2,s2):7LEARNER, next=8, gen=17]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has lost quorum, recovering: r43:/Table/{47-50} [(n4,s4):4, (n2,s2):2, (n3,s3):3, next=5, gen=8] -> r43:/Table/{47-50} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
Scanning replicas on store cluster_id:71abb5dd-96fc-4e5a-9f20-615def455ee1 node_id:2 store_id:2  for dead peers []
replica has not lost quorum, skipping: r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r8:/Table/1{2-3} [(n1,s1):5, (n2,s2):2, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r10:/Table/1{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r12:/Table/1{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r15:/Table/{19-20} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n2,s2):4, (n1,s1):5, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r19:/Table/2{3-4} [(n2,s2):4, (n3,s3):2, (n1,s1):7, next=8, gen=20]
replica has not lost quorum, skipping: r20:/Table/2{4-5} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r22:/Table/2{6-7} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r23:/Table/2{7-8} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r24:/Table/2{8-9} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r26:/NamespaceTable/{30-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r29:/Table/3{3-4} [(n1,s1):5, (n2,s2):2, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r30:/Table/3{4-5} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r31:/Table/3{5-6} [(n3,s3):4, (n2,s2):2, (n1,s1):5, next=6, gen=12]
replica has not lost quorum, skipping: r33:/Table/3{7-8} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r35:/Table/{39-40} [(n4,s4):4, (n1,s1):7, (n2,s2):3, next=8, gen=20]
not designated survivor, skipping: r41:/Table/4{5-6} [(n1,s1):6, (n4,s4):2, (n3,s3):4, (n2,s2):7LEARNER, next=8, gen=17]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss,deadlock

Same failure on other branches

/cc @cockroachdb/server @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-15952
@cockroach-teamcity cockroach-teamcity added deprecated-branch-release-22.1.0 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 26, 2022
@blathers-crl blathers-crl bot added the T-server-and-security DB Server & Security label Apr 26, 2022
@erikgrinaker
Copy link
Contributor 

F220426 07:00:12.837759 18684439 kv/kvserver/pkg/kv/kvserver/replica_init.go:344  [n1,s1,r21/10:/Table/2{5-6},raft] 1  attempted to change replica's ID from 10 to 9

Same as #75133 and #79074.

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 27, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

cli.TestRemoveDeadReplicas failed with artifacts on release-22.1.0 @ 65560026c29330949a9670db267b229a46c47ed7:

replica has not lost quorum, skipping: r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r7:/Table/1{1-2} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r7:/Table/1{1-2} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r8:/Table/1{2-3} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r8:/Table/1{2-3} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r10:/Table/1{4-5} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r10:/Table/1{4-5} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r11:/Table/1{5-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r12:/Table/1{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r13:/Table/1{7-8} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r13:/Table/1{7-8} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r18:/Table/2{2-3} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has lost quorum, recovering: r19:/Table/2{3-4} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r19:/Table/2{3-4} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r20:/Table/2{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r21:/Table/2{5-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r22:/Table/2{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has lost quorum, recovering: r24:/Table/2{8-9} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r24:/Table/2{8-9} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r25:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):5, (n4,s4):2, (n2,s2):6, next=7, gen=16]
replica has lost quorum, recovering: r30:/Table/3{4-5} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r30:/Table/3{4-5} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r31:/Table/3{5-6} [(n1,s1):5, (n2,s2):2, (n3,s3):3, next=6, gen=12]
replica has lost quorum, recovering: r32:/Table/3{6-7} [(n2,s2):4, (n3,s3):2, (n4,s4):3, next=5, gen=8] -> r32:/Table/3{6-7} [(n2,s2):5, next=6, gen=8]
replica has lost quorum, recovering: r33:/Table/3{7-8} [(n2,s2):4, (n4,s4):2, (n3,s3):3, next=5, gen=8] -> r33:/Table/3{7-8} [(n2,s2):5, next=6, gen=8]
replica has not lost quorum, skipping: r35:/Table/{39-40} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r37:/Table/4{1-2} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r38:/Table/4{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n2,s2):8, (n4,s4):2, (n1,s1):7, next=9, gen=24]
replica has not lost quorum, skipping: r43:/Table/{47-50} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
Scanning replicas on store cluster_id:2deb82e0-f420-4dbd-9de7-5ff808180600 node_id:2 store_id:2  for dead peers []
replica has not lost quorum, skipping: r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r4:/System{/tsd-tse} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r11:/Table/1{5-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r12:/Table/1{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r14:/Table/1{8-9} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r16:/Table/2{0-1} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r18:/Table/2{2-3} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r20:/Table/2{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r21:/Table/2{5-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r22:/Table/2{6-7} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r25:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r27:/{NamespaceTable/Max-Table/32} [(n1,s1):5, (n4,s4):2, (n2,s2):6, next=7, gen=16]
replica has not lost quorum, skipping: r31:/Table/3{5-6} [(n1,s1):5, (n2,s2):2, (n3,s3):3, next=6, gen=12]
replica has not lost quorum, skipping: r35:/Table/{39-40} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
replica has not lost quorum, skipping: r37:/Table/4{1-2} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r38:/Table/4{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]
replica has not lost quorum, skipping: r42:/Table/4{6-7} [(n2,s2):8, (n4,s4):2, (n1,s1):7, next=9, gen=24]
replica has not lost quorum, skipping: r43:/Table/{47-50} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4, gen=4]
replica has not lost quorum, skipping: r44:/{Table/50-Max} [(n1,s1):1, (n4,s4):2, (n2,s2):3, next=4, gen=4]
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 4, 2022
Closed
@jlinder jlinder added sync-me and removed sync-me labels May 20, 2022
@tbg tbg added the S-3 Medium-low impact: incurs increased costs for some users (incl lower avail, recoverable bad data) label May 30, 2022
@tbg
Copy link
Member

tbg commented Jun 27, 2022

Closing since this branch is no longer being tested.

@tbg tbg closed this as completed Jun 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aliher1911 aliher1911

Labels
C-test-failure Broken test (automatically or manually discovered). deprecated-branch-release-22.1.0 O-robot Originated from a bot. S-3 Medium-low impact: incurs increased costs for some users (incl lower avail, recoverable bad data)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jlinder @erikgrinaker @tbg @aliher1911 @cockroach-teamcity