Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestRaftPreVote failed #108124

Closed
cockroach-teamcity opened this issue Aug 3, 2023 · 3 comments · Fixed by #108454
Closed

kv/kvserver: TestRaftPreVote failed #108124

cockroach-teamcity opened this issue Aug 3, 2023 · 3 comments · Fixed by #108454
Assignees
@erikgrinaker
Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 3, 2023 •
edited by cockroach-jira-scripts
Loading

kv/kvserver.TestRaftPreVote failed with artifacts on master @ 03b1d4c562a4f46cf002a3d95436f45e9dce6b29:

=== RUN   TestRaftPreVote
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/ec76f3007984602f26e3543a773d47f5/logTestRaftPreVote3375929530
    test_log_scope.go:81: use -show-logs to present logs inline
=== CONT  TestRaftPreVote
    client_raft_test.go:6342: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/ec76f3007984602f26e3543a773d47f5/logTestRaftPreVote3375929530
--- FAIL: TestRaftPreVote (42.43s)
=== RUN   TestRaftPreVote/partial=true/quiesce=false
        --- FAIL: TestRaftPreVote/partial=true/quiesce=false (12.63s)
=== RUN   TestRaftPreVote/partial=true/quiesce=false/expLease=true
    client_raft_test.go:6247: n1 has lease
    client_raft_test.go:6255: n3 not quiesced
    client_raft_test.go:6260: n3 partitioned
    client_raft_test.go:6265: n1 StateLeader at term=6 commit=22
    client_raft_test.go:6273: n3 not quiesced
    client_raft_test.go:6279: n3 StateFollower at term=6 commit=22
    client_raft_test.go:6177: r64 write rejected: GC [/Table/Max,"\xfa\x00"), GC [/Table/Max,"\xfa\x00"), GC [/Table/Max,"\xfa\x00"), GC [/Table/Max,"\xfa\x00")
    client_raft_test.go:6279: n3 StateFollower at term=6 commit=22
    client_raft_test.go:6279: n3 StatePreCandidate at term=6 commit=22
    client_raft_test.go:6282: n3 became pre-candidate
    client_raft_test.go:6289: n3 StatePreCandidate at term=6 commit=22
    client_raft_test.go:6289: n3 StatePreCandidate at term=6 commit=22
    client_raft_test.go:6292: n3 is still pre-candidate
    client_raft_test.go:6299: n1 StateLeader at term=6 commit=23
    client_raft_test.go:6303: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_test.go:6303
        	            				github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:23
        	Error:      	Not equal: 
        	            	expected: 0x16
        	            	actual  : 0x17
        	Test:       	TestRaftPreVote/partial=true/quiesce=false/expLease=true
            --- FAIL: TestRaftPreVote/partial=true/quiesce=false/expLease=true (6.40s)
=== RUN   TestRaftPreVote/partial=true
    --- FAIL: TestRaftPreVote/partial=true (21.14s)
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-30323
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv-replication labels Aug 3, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Aug 3, 2023
@nvanbenschoten
Copy link
Member

@erikgrinaker it looks like you just added this test in 6661c46, so I'll assign you for now.

@nvanbenschoten nvanbenschoten added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure labels Aug 6, 2023
@cockroach-teamcity
Copy link
Member Author

kv/kvserver.TestRaftPreVote failed with artifacts on master @ b87852acc3562ac10f2f957af7d806416abe2381:

=== RUN   TestRaftPreVote
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/ec76f3007984602f26e3543a773d47f5/logTestRaftPreVote1231183238
    test_log_scope.go:81: use -show-logs to present logs inline
=== CONT  TestRaftPreVote
    client_raft_test.go:6348: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/ec76f3007984602f26e3543a773d47f5/logTestRaftPreVote1231183238
--- FAIL: TestRaftPreVote (39.90s)
=== RUN   TestRaftPreVote/partial=false
    --- FAIL: TestRaftPreVote/partial=false (19.27s)
=== RUN   TestRaftPreVote/partial=false/quiesce=true
        --- FAIL: TestRaftPreVote/partial=false/quiesce=true (7.69s)
=== RUN   TestRaftPreVote/partial=false/quiesce=true/expLease=false
    client_raft_test.go:6253: n1 has lease
    client_raft_test.go:6258: n3 quiesced
    client_raft_test.go:6266: n3 partitioned
    client_raft_test.go:6271: n1 StateLeader at term=6 commit=24
    client_raft_test.go:6276: n3 unquiesced
    client_raft_test.go:6285: n3 StateFollower at term=6 commit=24
    client_raft_test.go:6285: n3 StatePreCandidate at term=6 commit=24
    client_raft_test.go:6288: n3 became pre-candidate
    client_raft_test.go:6295: n3 StatePreCandidate at term=6 commit=24
    client_raft_test.go:6295: n3 StatePreCandidate at term=6 commit=24
    client_raft_test.go:6298: n3 is still pre-candidate
    client_raft_test.go:6305: n1 StateLeader at term=6 commit=26
    client_raft_test.go:6311: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_raft_test.go:6311
        	            				github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:23
        	Error:      	"26" is not less than or equal to "25"
        	Test:       	TestRaftPreVote/partial=false/quiesce=true/expLease=false
            --- FAIL: TestRaftPreVote/partial=false/quiesce=true/expLease=false (7.69s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@erikgrinaker
Copy link
Contributor

This is probably caused by #106828.

@nvanbenschoten nvanbenschoten mentioned this issue Aug 9, 2023
Merged
craig bot pushed a commit that referenced this issue Aug 9, 2023
@fqazi @nvanbenschoten
Merge #107355 #108454
b91c919 
107355: cli: enhance diagnosing contention with redacted debug zips r=fqazi a=fqazi

Previously a redacted zip, we would exclude retries and key information for contended keys since they could contain PII data. This patch does the following:

- Adds a new builtin is_system_table_key which allows us to know if a key belongs to a system table
- Modified redacted debug zips to include data for system keys in contention tables conditionally (if they belong to system tables)
- Include retries and last_retry_reason information for queries in cluster insights to help diagnose contention

Fixes: #104593

108454: Revert "kvprober: metamorphically enable / configure kvprober" r=andrewbaptist,joshimhoff a=nvanbenschoten

This reverts (most of) commit 769ba1c.

That commit metamorphically enabled kvprober. This has been observed to be destabliziing to unit tests. When the metamorphic constant is enabled (50% of the time) and when kvprober is fast enough, random ranges will see extra requests that they aren’t expecting. This adds nondeterminism which can trip up tests in any number of different ways.

All of the following flakes have been tracked back to kvprober:

Fixes #107864.
Fixes #108242.
Fixes #108441.
Fixes #108349.
Fixes #108124.
Closes #108366.

Release note: None

Co-authored-by: Faizan Qazi <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in #108454 Aug 9, 2023
@craig craig bot closed this as completed in 1374908 Aug 9, 2023
@pav-kv pav-kv mentioned this issue Aug 29, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@erikgrinaker erikgrinaker

Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

Revert "kvprober: metamorphically enable / configure kvprober" nvanbenschoten/cockroach
3 participants
@erikgrinaker @nvanbenschoten @cockroach-teamcity