Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check conflict at output level in CompactFiles #3926

Closed
wants to merge 2 commits into from
Closed

Check conflict at output level in CompactFiles #3926

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ea5a73f
Check conflict at output level in CompactFiles
ajkr May 31, 2018
9e7cb30
address my own comment
ajkr May 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions db/compaction_picker.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -853,6 +853,11 @@ Status CompactionPicker::SanitizeCompactionInputFilesForAllLevels(
}
}
}
if (RangeOverlapWithCompaction(smallestkey, largestkey, output_level)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is it possible that there is no overlap of input files with running compactions, but there is overlap at output level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a weird edge case, where the input files' key range is in a gap between files that are pending compaction to the same output level. The pending compaction merges the input files and can produce output files that cover the gap.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification. In this edge case, we might end up with files in the output with overlapping key ranges. So checking for range overlap with pending compactions would detect that and not start the manual compaction.

return Status::Aborted(
"A running compaction is writing to the same output level in an "
"overlapping key range");
}
return Status::OK();
}

Expand Down
62 changes: 62 additions & 0 deletions db/db_compaction_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3702,6 +3702,68 @@ TEST_F(DBCompactionTest, CompactionStatsTest) {
VerifyCompactionStats(*cfd, *collector);
}

TEST_F(DBCompactionTest, CompactFilesOutputRangeConflict) {
// LSM setup:
// L1: [ba bz]
// L2: [a b] [c d]
// L3: [a b] [c d]
//
// Thread 1: Thread 2:
// Begin compacting all L2->L3
// Compact [ba bz] L1->L3
// End compacting all L2->L3
//
// The compaction operation in thread 2 should be disallowed because the range
// overlaps with the compaction in thread 1, which also covers that range in
// L3.
Options options = CurrentOptions();
FlushedFileCollector* collector = new FlushedFileCollector();
options.listeners.emplace_back(collector);
Reopen(options);

for (int level = 3; level >= 2; --level) {
ASSERT_OK(Put("a", "val"));
ASSERT_OK(Put("b", "val"));
ASSERT_OK(Flush());
ASSERT_OK(Put("c", "val"));
ASSERT_OK(Put("d", "val"));
ASSERT_OK(Flush());
MoveFilesToLevel(level);
}
ASSERT_OK(Put("ba", "val"));
ASSERT_OK(Put("bz", "val"));
ASSERT_OK(Flush());
MoveFilesToLevel(1);

SyncPoint::GetInstance()->LoadDependency({
{"CompactFilesImpl:0",
"DBCompactionTest::CompactFilesOutputRangeConflict:Thread2Begin"},
{"DBCompactionTest::CompactFilesOutputRangeConflict:Thread2End",
"CompactFilesImpl:1"},
});
SyncPoint::GetInstance()->EnableProcessing();

auto bg_thread = port::Thread([&]() {
// Thread 1
std::vector<std::string> filenames = collector->GetFlushedFiles();
filenames.pop_back();
ASSERT_OK(db_->CompactFiles(CompactionOptions(), filenames,
3 /* output_level */));
});

// Thread 2
TEST_SYNC_POINT(
"DBCompactionTest::CompactFilesOutputRangeConflict:Thread2Begin");
std::string filename = collector->GetFlushedFiles().back();
ASSERT_FALSE(
db_->CompactFiles(CompactionOptions(), {filename}, 3 /* output_level */)
.ok());
TEST_SYNC_POINT(
"DBCompactionTest::CompactFilesOutputRangeConflict:Thread2End");

bg_thread.join();
}

INSTANTIATE_TEST_CASE_P(DBCompactionTestWithParam, DBCompactionTestWithParam,
::testing::Values(std::make_tuple(1, true),
std::make_tuple(1, false),
Expand Down