kv/kvserver: TestTruncateLog failed #79719

cockroach-teamcity · 2022-04-09T07:29:18Z

kv/kvserver.TestTruncateLog failed with artifacts on release-22.1 @ 9b76da708050aab88c24eaa74e7cae05aa5ddc70:

=== RUN   TestTruncateLog
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestTruncateLog3631979790
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTruncateLog
    raft_log_queue_test.go:810: -- test log scope end --
--- FAIL: TestTruncateLog (0.63s)
=== RUN   TestTruncateLog/loosely-coupled=false
    raft_log_queue_test.go:771: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=false (0.29s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

TAGS=bazel,gss

/cc @cockroachdb/kv _{This test on roachdash | Improve this report!

Jira issue: CRDB-14994}

The text was updated successfully, but these errors were encountered:

erikgrinaker · 2022-04-11T09:28:58Z

@sumeerbhola What's your take on this?

tbg · 2022-04-11T11:37:49Z

Sibling issue on master: #77825

sumeerbhola · 2022-04-11T13:20:31Z

This failure is in loosely-coupled=false, unlike the TestTerm issue (which I was unable to reproduce and seems to not have recurred for 2 weeks).

    --- FAIL: TestTruncateLog/loosely-coupled=false (0.29s)
    --- PASS: TestTruncateLog/loosely-coupled=true (0.30s)

Trying to reproduce.
I see the following in the logs, so maybe the problem is in the test which got unlucky with an unexpected merge.

I220409 07:14:24.781245 630552 kv/kvserver/batcheval/cmd_truncate_log.go:56 ⋮ [s1,r1/1:‹/M{in-ax}›] 1  attempting to truncate raft logs for another range: r2. Normally this is due to a merge and can be ignored.

This is contradicting the place of the failure which is the error below, which suggest more has been truncated (not less).

		// The term of the last truncated entry is still available.
		tc.repl.mu.Lock()
		term, err := tc.repl.raftTermRLocked(indexes[4])
		tc.repl.mu.Unlock()
		if err != nil {
			t.Fatal(err)
		}

sumeerbhola · 2022-04-11T15:59:27Z

640,000 runs so far with no failure :(

rail · 2022-05-25T12:38:40Z

Manually synced with Jira

cockroach-teamcity · 2022-05-27T07:24:58Z

kv/kvserver.TestTruncateLog failed with artifacts on release-22.1 @ d64792534335d20233269318e2b1b4ce694d394e:

=== RUN   TestTruncateLog
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestTruncateLog2479753635
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTruncateLog
    raft_log_queue_test.go:810: -- test log scope end --
--- FAIL: TestTruncateLog (0.75s)
=== RUN   TestTruncateLog/loosely-coupled=true
    raft_log_queue_test.go:771: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=true (0.35s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

TAGS=bazel,gss

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-16T07:11:06Z

kv/kvserver.TestTruncateLog failed with artifacts on release-22.1 @ ae7edece60a9d0d89a86a6fc6408eb253c371881:

=== RUN   TestTruncateLog
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestTruncateLog4246184072
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTruncateLog
    raft_log_queue_test.go:805: -- test log scope end --
--- FAIL: TestTruncateLog (0.73s)
=== RUN   TestTruncateLog/loosely-coupled=false
    raft_log_queue_test.go:766: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=false (0.32s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

TAGS=bazel,gss

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-10-19T07:38:26Z

kv/kvserver.TestTruncateLog failed with artifacts on release-22.1 @ 3875aa648b3aeb78aa458b3513e41bdbc52004e3:

=== RUN   TestTruncateLog
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestTruncateLog3712744608
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTruncateLog
    raft_log_queue_test.go:805: -- test log scope end --
--- FAIL: TestTruncateLog (0.67s)
=== RUN   TestTruncateLog/loosely-coupled=false
    raft_log_queue_test.go:766: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=false (0.31s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

TAGS=bazel,gss

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-13T08:07:05Z

kv/kvserver.TestTruncateLog failed with artifacts on release-22.1 @ d21a4a188562e73172c66fa9d57eb1baa6ab149c:

=== RUN   TestTruncateLog
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestTruncateLog684204320
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTruncateLog
    raft_log_queue_test.go:805: -- test log scope end --
--- FAIL: TestTruncateLog (0.68s)
=== RUN   TestTruncateLog/loosely-coupled=true
    raft_log_queue_test.go:766: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=true (0.33s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-31T09:08:27Z

kv/kvserver.TestTruncateLog failed with artifacts on release-22.1 @ 0468eb91d6fa303a4f016e99aff7bb2824c38f1c:

=== RUN   TestTruncateLog
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestTruncateLog2078562329
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTruncateLog
    raft_log_queue_test.go:805: -- test log scope end --
--- FAIL: TestTruncateLog (0.82s)
=== RUN   TestTruncateLog/loosely-coupled=false
    raft_log_queue_test.go:766: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=false (0.36s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-04T07:57:05Z

kv/kvserver.TestTruncateLog failed with artifacts on release-22.1 @ 21fc175f98866bd9b18ead1606867bdb933878e4:

=== RUN   TestTruncateLog
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestTruncateLog268465716
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTruncateLog
    raft_log_queue_test.go:805: -- test log scope end --
--- FAIL: TestTruncateLog (0.82s)
=== RUN   TestTruncateLog/loosely-coupled=false
    raft_log_queue_test.go:766: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=false (0.37s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-08T08:26:51Z

kv/kvserver.TestTruncateLog failed with artifacts on release-22.1 @ afad258bd2690d6ff8b0b8636224775dabbd6d98:

=== RUN   TestTruncateLog
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestTruncateLog2791901721
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTruncateLog
    raft_log_queue_test.go:805: -- test log scope end --
--- FAIL: TestTruncateLog (0.80s)
=== RUN   TestTruncateLog/loosely-coupled=true
    raft_log_queue_test.go:766: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=true (0.39s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-12T11:34:05Z

kv/kvserver.TestTruncateLog failed with artifacts on release-22.1 @ 9ab632f3bedfb653dacf571fb4b10e02de51ef77:

=== RUN   TestTruncateLog
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestTruncateLog2261802216
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTruncateLog
    raft_log_queue_test.go:805: -- test log scope end --
--- FAIL: TestTruncateLog (0.88s)
=== RUN   TestTruncateLog/loosely-coupled=false
    raft_log_queue_test.go:766: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=false (0.38s)
=== RUN   TestTruncateLog/loosely-coupled=true
    raft_log_queue_test.go:766: requested index is unavailable due to compaction
    --- FAIL: TestTruncateLog/loosely-coupled=true (0.45s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

irfansharif · 2023-04-12T22:13:31Z

It's started happening more frequently, also on #100449.

erikgrinaker · 2023-04-13T09:51:06Z

Yeah, these are on our slate, just have to deal with some GA blockers first.

erikgrinaker · 2023-04-13T10:06:15Z

Looks like this and TestTerm are only flaky on release-22.1, the master builds haven't flaked for months. We could consider just skipping it, since this is almost certainly a test flake and 22.1 is out of support in a month anyway.

erikgrinaker · 2023-04-13T10:45:23Z

There were some related bugfixes in this area in June 2022 that weren't backported to 22.1:

kv: remove dead code in Replica.raftTruncatedStateLocked #81879 (see kv: remove dead code in Replica.raftTruncatedStateLocked #81879 (review))
keys: resolve subtle non-bug by exporting RaftLogKeyFromPrefix #82479

However, we did see flakes on master a few days after those merged. Will see if I can find any other fixes in that area.

erikgrinaker · 2023-04-13T11:32:52Z

Haven't been able to reproduce after ~100k stress runs on release-22.1, of both TestTruncateLog and TestTerm, regular and under race (to provoke races via slowness). Adding sleeps in the test and code did not reproduce either.

This test has been flaky as far back as 2019. Since this isn't flaking in later releases, and 22.1 is out of support in a month, I'm going to cut our losses and skip this on release-22.1.

erikgrinaker · 2023-04-13T13:53:15Z

Resolved by #101435.

cockroach-teamcity added branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 9, 2022

blathers-crl bot added the T-kv KV Team label Apr 9, 2022

erikgrinaker added T-kv-replication and removed T-kv KV Team labels Apr 11, 2022

jlinder added the sync-me-3 label May 24, 2022

jlinder removed the sync-me-3 label May 27, 2022

tbg added the S-3 Medium-low impact: incurs increased costs for some users (incl lower avail, recoverable bad data) label May 30, 2022

cockroach-teamcity mentioned this issue Jun 13, 2022

kv/kvserver: TestTruncateLog failed #82806

Closed

erikgrinaker self-assigned this Apr 13, 2023

erikgrinaker mentioned this issue Apr 13, 2023

release-22.1: kvserver: skip TestTerm and TestTruncateLog #101435

Merged

erikgrinaker closed this as completed Apr 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kv/kvserver: TestTruncateLog failed #79719

kv/kvserver: TestTruncateLog failed #79719

cockroach-teamcity commented Apr 9, 2022 •

edited by cockroach-jira-scripts

Loading

erikgrinaker commented Apr 11, 2022

tbg commented Apr 11, 2022

sumeerbhola commented Apr 11, 2022

sumeerbhola commented Apr 11, 2022

rail commented May 25, 2022

cockroach-teamcity commented May 27, 2022

cockroach-teamcity commented Sep 16, 2022

cockroach-teamcity commented Oct 19, 2022

cockroach-teamcity commented Mar 13, 2023

cockroach-teamcity commented Mar 31, 2023

cockroach-teamcity commented Apr 4, 2023

cockroach-teamcity commented Apr 8, 2023

cockroach-teamcity commented Apr 12, 2023

irfansharif commented Apr 12, 2023

erikgrinaker commented Apr 13, 2023

erikgrinaker commented Apr 13, 2023

erikgrinaker commented Apr 13, 2023

erikgrinaker commented Apr 13, 2023

erikgrinaker commented Apr 13, 2023

kv/kvserver: TestTruncateLog failed #79719

kv/kvserver: TestTruncateLog failed #79719

Comments

cockroach-teamcity commented Apr 9, 2022 • edited by cockroach-jira-scripts Loading

erikgrinaker commented Apr 11, 2022

tbg commented Apr 11, 2022

sumeerbhola commented Apr 11, 2022

sumeerbhola commented Apr 11, 2022

rail commented May 25, 2022

cockroach-teamcity commented May 27, 2022

cockroach-teamcity commented Sep 16, 2022

cockroach-teamcity commented Oct 19, 2022

cockroach-teamcity commented Mar 13, 2023

cockroach-teamcity commented Mar 31, 2023

cockroach-teamcity commented Apr 4, 2023

cockroach-teamcity commented Apr 8, 2023

cockroach-teamcity commented Apr 12, 2023

irfansharif commented Apr 12, 2023

erikgrinaker commented Apr 13, 2023

erikgrinaker commented Apr 13, 2023

erikgrinaker commented Apr 13, 2023

erikgrinaker commented Apr 13, 2023

erikgrinaker commented Apr 13, 2023

cockroach-teamcity commented Apr 9, 2022 •

edited by cockroach-jira-scripts

Loading