Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestCheckConsistencyInconsistent failed in CI #89400

Closed
nvanbenschoten opened this issue Oct 5, 2022 · 2 comments
Closed

kv/kvserver: TestCheckConsistencyInconsistent failed in CI #89400

nvanbenschoten opened this issue Oct 5, 2022 · 2 comments
Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-test-failure Broken test (automatically or manually discovered).

Comments

@nvanbenschoten
Copy link
Member

nvanbenschoten commented Oct 5, 2022 •
edited by cockroach-jira-scripts
Loading

Opening because I don't see an automatically opened issue.

CI in #89340 (c387857, based on fc50f10) failed here:

=== RUN   TestCheckConsistencyInconsistent
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/09357cecfdbbab5926b4b055936a9b62/logTestCheckConsistencyInconsistent405603334
    test_log_scope.go:79: use -show-logs to present logs inline
    consistency_queue_test.go:386:
          Error Trace:  /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/6062/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/consistency_queue_test.go:386
          Error:        Should be empty, but was --- leaseholder
                        +++ follower
                        -0,0 /Local/RangeID/5/r/RangeAppliedState
                        -    ts:1970-01-01 00:00:00 +0000 UTC
                        -    value:raft_applied_index:34 lease_applied_index:15 range_stats:<last_update_nanos:1664922159271520766 live_bytes:40 live_count:2 key_bytes:28 key_count:2 val_bytes:12 val_count:2 sys_bytes:484 sys_count:6 > raft_closed_timestamp:<wall_time:1664922157282965628 > raft_applied_index_term:6
                        -    raw mvcc_key/value: 01698d727261736b00 12040800100018002000280032337fb76066030822100f1a1a09fe290efae1fd1a1720282802301c3802400c480260e4036806220a08fcf09c9c98bcbf8d172806
                        +0,0 /Local/RangeID/5/r/RangeAppliedState
                        +    ts:1970-01-01 00:00:00 +0000 UTC
                        +    value:raft_applied_index:33 lease_applied_index:14 range_stats:<last_update_nanos:1664922159271520766 live_bytes:40 live_count:2 key_bytes:28 key_count:2 val_bytes:12 val_count:2 sys_bytes:484 sys_count:6 > raft_closed_timestamp:<wall_time:1664922157279764330 > raft_applied_index_term:6
                        +    raw mvcc_key/value: 01698d727261736b00 120408001000180020002800323393359e5c030821100e1a1a09fe290efae1fd1a1720282802301c3802400c480260e4036806220a08eabed99a98bcbf8d172806
          Test:         TestCheckConsistencyInconsistent
    consistency_queue_test.go:396:
          Error Trace:  /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/6062/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_test_/kvserver_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvserver/consistency_queue_test.go:396
          Error:        Not equal:
                        expected: "--- leaseholder\n+++ follower\n+0.000000123,987 \"e\"\n+    ts:1970-01-01 00:00:00.000000123 +0000 UTC\n+    value:\"\\x00\\x00\\x00\\x00\\x01T\"\n+    raw mvcc_key/value: 6500000000000000007b000003db0d 000000000154\n"
                        actual  : "--- leaseholder\n+++ follower\n-0.000000123,987 /Local/RangeID/5/r/RangeAppliedState\n-    ts:1970-01-01 00:00:00.000000123 +0000 UTC\n-    value:txn={<nil>} ts=0,0 del=false klen=0 vlen=0 rawlen=51 mergeTs=<nil> txnDidNotUpdateMeta=false\n-    raw mvcc_key/value: 01698d727261736b00000000000000007b000003db0d 12040800100018002000280032337fb76066030822100f1a1a09fe290efae1fd1a1720282802301c3802400c480260e4036806220a08fcf09c9c98bcbf8d172806\n+0,0 /Local/RangeID/5/r/RangeAppliedState\n+    ts:1970-01-01 00:00:00 +0000 UTC\n+    value:raft_applied_index:33 lease_applied_index:14 range_stats:<last_update_nanos:1664922159271520766 live_bytes:40 live_count:2 key_bytes:28 key_count:2 val_bytes:12 val_count:2 sys_bytes:484 sys_count:6 > raft_closed_timestamp:<wall_time:1664922157279764330 > raft_applied_index_term:6\n+    raw mvcc_key/value: 01698d727261736b00 120408001000180020002800323393359e5c030821100e1a1a09fe290efae1fd1a1720282802301c3802400c480260e4036806220a08eabed99a98bcbf8d172806\n+1664922160.279193320,0 \"e\"\n+    ts:2022-10-04 22:22:40.27919332 +0000 UTC\n+    value:\"\\x00\\x00\\x00\\x00\\x01T\"\n+    raw mvcc_key/value: 6500171afde2361e06e809 000000000154\n"
                        Diff:
                        --- Expected
                        +++ Actual
                        @@ -2,6 +2,14 @@
                         +++ follower
                        -+0.000000123,987 "e"
                        -+    ts:1970-01-01 00:00:00.000000123 +0000 UTC
                        +-0.000000123,987 /Local/RangeID/5/r/RangeAppliedState
                        +-    ts:1970-01-01 00:00:00.000000123 +0000 UTC
                        +-    value:txn={<nil>} ts=0,0 del=false klen=0 vlen=0 rawlen=51 mergeTs=<nil> txnDidNotUpdateMeta=false
                        +-    raw mvcc_key/value: 01698d727261736b00000000000000007b000003db0d 12040800100018002000280032337fb76066030822100f1a1a09fe290efae1fd1a1720282802301c3802400c480260e4036806220a08fcf09c9c98bcbf8d172806
                        ++0,0 /Local/RangeID/5/r/RangeAppliedState
                        ++    ts:1970-01-01 00:00:00 +0000 UTC
                        ++    value:raft_applied_index:33 lease_applied_index:14 range_stats:<last_update_nanos:1664922159271520766 live_bytes:40 live_count:2 key_bytes:28 key_count:2 val_bytes:12 val_count:2 sys_bytes:484 sys_count:6 > raft_closed_timestamp:<wall_time:1664922157279764330 > raft_applied_index_term:6
                        ++    raw mvcc_key/value: 01698d727261736b00 120408001000180020002800323393359e5c030821100e1a1a09fe290efae1fd1a1720282802301c3802400c480260e4036806220a08eabed99a98bcbf8d172806
                        ++1664922160.279193320,0 "e"
                        ++    ts:2022-10-04 22:22:40.27919332 +0000 UTC
                         +    value:"\x00\x00\x00\x00\x01T"
                        -+    raw mvcc_key/value: 6500000000000000007b000003db0d 000000000154
                        ++    raw mvcc_key/value: 6500171afde2361e06e809 000000000154
          Test:         TestCheckConsistencyInconsistent
    consistency_queue_test.go:406: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/09357cecfdbbab5926b4b055936a9b62/logTestCheckConsistencyInconsistent405603334
--- FAIL: TestCheckConsistencyInconsistent (11.48s)

Notice the difference in the raft_applied_index and range_applied_index in the RangeAppliedState. That should not be possible during a consistency check, which should view all replicas at the same applied index.

Jira issue: CRDB-20245
@nvanbenschoten nvanbenschoten added C-test-failure Broken test (automatically or manually discovered). A-kv-replication Relating to Raft, consensus, and coordination. T-kv-replication labels Oct 5, 2022
@nvanbenschoten
Copy link
Member Author

Fixed by #89369.

@blathers-crl
Copy link

blathers-crl bot commented Oct 5, 2022

cc @cockroachdb/replication

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv-replication Relating to Raft, consensus, and coordination. C-test-failure Broken test (automatically or manually discovered).
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nvanbenschoten