assertion failure in version_set.cc

[jsteemann]

Expected behavior

Not running into an assertion failure.

Actual behavior

Running into an assertion failure in version_set.cc:4010

4000 uint64_t VersionSet::ApproximateSizeLevel0(Version* v,
4001                                            const LevelFilesBrief& files_brief,
4002                                            const Slice& key_start,
4003                                            const Slice& key_end) {
4004   // level 0 files are not in sorted order, we need to iterate through
4005   // the list to compute the total bytes that require scanning
4006   uint64_t size = 0;
4007   for (size_t i = 0; i < files_brief.num_files; i++) {
4008     const uint64_t start = ApproximateSize(v, files_brief.files[i], key_start);
4009     const uint64_t end = ApproximateSize(v, files_brief.files[i], key_end);
4010     assert(end >= start);
4011     size += end - start;
4012   }
4013   return size;
4014 }

Original call is from application layer to DBImpl::GetApproximateSizes.

Steps to reproduce the behavior

The assertion failure happens very seldomly and "randomly" (i.e. it is not easy to reproduce).
The keys provided to DBImpl::GetApproximateSize compare ok (i.e. start < end).

[riversand963]

@jsteemann Have you been able to reproduce this deterministically? Which version of RocksDB are you using?

[jsteemann]

Unfortunately not.
We have seen this assertion failure probably twice in a period of at least two months. So it may be a real edge case.

[riversand963]

@jsteemann which version of RocksDB were you using when this assertion failure happened?

[jsteemann]

That was 5.9. But I checked 5.14 as well and found that the corresponding code in version_set.cc is almost identical, so probably the assertion may still trigger under some (yet to be found) circumstances.

[riversand963]

@jsteemann , thanks for the report. One thought is that the assertion error could have been caused by CompactFiles writing files to the same range as another concurrent compaction, which was addressed in #3926. Could you upgrade your RocksDB version and see if the assertion error still exists?

[jsteemann]

Ok, will do. Thanks for the update!

[riversand963]

@jsteemann Thanks for looking into this, I will close this issue for now, but you can re-open it once you see the failure again. Steps to reproduce will be highly appreciated.