kv/kvserver: TestRaftSSTableSideloadingTruncation failed

[cockroach-teamcity]

kv/kvserver.TestRaftSSTableSideloadingTruncation failed with artifacts on release-22.2 @ c003f83581db6f5c0fdf76e3d190539d3acb6679:

=== RUN   TestRaftSSTableSideloadingTruncation
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestRaftSSTableSideloadingTruncation3004024942
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestRaftSSTableSideloadingTruncation
    replica_sideload_test.go:859: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestRaftSSTableSideloadingTruncation3004024942
--- FAIL: TestRaftSSTableSideloadingTruncation (0.10s)
=== RUN   TestRaftSSTableSideloadingTruncation/loosely-coupled=true
    replica_sideload_test.go:856: expected all files to be cleaned up, but found [i14.t6 i15.t6 i16.t6 i17.t6 i18.t6 i19.t6 i20.t6 i21.t6 i22.t6 i23.t6]
    --- FAIL: TestRaftSSTableSideloadingTruncation/loosely-coupled=true (0.03s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-32230

[blathers-crl]

cc @cockroachdb/replication

[kvoli]

I am probably missing something obvious here, but this failed reliably for the bisect window I attempted (up to 251a721) using dev test pkg/kv/kvserver -f TestRaftSSTableSideloadingTruncation -v --stress

bisect attempt #1

git bisect start
# status: waiting for both good and bad commits
# bad: [c003f83581db6f5c0fdf76e3d190539d3acb6679] Merge pull request #112062 from cockroachdb/blathers/backport-release-22.2-112025
git bisect bad c003f83581db6f5c0fdf76e3d190539d3acb6679
# status: waiting for good commit(s), bad commit known
# good: [8a42d07084615255b18217edb0f065cbc4a8859e] roachtest: update version of nodejs used by ORM tests
git bisect good 8a42d07084615255b18217edb0f065cbc4a8859e
# bad: [79c461a7434754e3dacfd16893be7cd52c2552c4] Merge pull request #110721 from fqazi/seqeuenceByIDUpgrade
git bisect bad 79c461a7434754e3dacfd16893be7cd52c2552c4
# bad: [d211d5307c6f0400e3017be130976499c260adc8] Merge pull request #110211 from rafiss/backport22.2-110144
git bisect bad d211d5307c6f0400e3017be130976499c260adc8
# bad: [9869f346de39e6ff097e825823437daa6370527e] Merge pull request #110053 from kvoli/backport22.2-109819
git bisect bad 9869f346de39e6ff097e825823437daa6370527e
# bad: [eba3e91b9fe08273c611b26faa364b8d077fd5ee] Merge pull request #106143 from cockroachdb/blathers/backport-release-22.2-106098
git bisect bad eba3e91b9fe08273c611b26faa364b8d077fd5ee
# bad: [742a52114e8c46149bc97a5998e92f96295d001f] Merge pull request #109685 from pavelkalinnikov/backport22.2-109659
git bisect bad 742a52114e8c46149bc97a5998e92f96295d001f
# bad: [6672744eff8afc4c66a71b269afe45feb464a18e] Merge pull request #109917 from DarrylWong/backport22.2-109438
git bisect bad 6672744eff8afc4c66a71b269afe45feb464a18e
# bad: [251a721a88369b0233162abd0b02f4875948aaa9] roachprod: azure gc cleans up empty clusters
git bisect bad 251a721a88369b0233162abd0b02f4875948aaa9
# first bad commit: [251a721a88369b0233162abd0b02f4875948aaa9] roachprod: azure gc cleans up empty clusters

I then tested a further back commit (2b49a9f June 16th) which also failed.

[kvoli]

Seems like a known problem based on this slack (internal) conversation with the same issue.

When this fails, this error pops up. Wrapping the cleanup check in a SucceedsSoon doesn't prevent the test from flaking, so it doesn't appear to be a timing issue, where it would eventually pass.

while removing sideloaded files during log truncation: matching pattern "auxiliary/sideloading/r0XXXX/r1/i14.t6" on dir auxiliary/sideloading/r0XXXX /r1: invalid argument

Removing the release blocker since this isn't a regression.

cc @cockroachdb/storage
xref as possible cause cockroachdb/pebble#1236

[jbowens]

Hrm, looks like we need to backport cockroachdb/pebble#2065.

[jbowens]

Resolved in 324e8e8.