[WIP] internal/manifest: l0SubLevels data-structure and methods to #563
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
sumeerbhola
commented
Mar 4, 2020
There was a problem hiding this comment.
Only the second commit is relevant. This is very much WIP.
Reviewable status: 0 of 22 files reviewed, all discussions resolved (waiting on @itsbilal and @petermattis)
|
@sumeerbhola I probably won't be able to look at this until next week. |
sumeerbhola
force-pushed
the
sublevel
branch
4 times, most recently
from
d1f48f6 to
af6f1d5
Compare
This is a work in progress. We need to plumb knowledge of L0 sublevels into compaction.
measure stack depth, compaction score, pick L0->Lbase and intra-L0 compactions. The main construction here (beyond the sub-levels introduced in of intervals is O(N) where N is the number of files in L0, but we expect that typically each interval will have a file count that is sublinear in N. These intervals allow for precisely tracking the depth and are used as the seed for computing compactions. This code lacks any tests (see the work todos listed in l0_sublevels.go).
added preliminary testing Used a MANIFEST with > 600 files and 2GB in L0 from one of the slow imports. Results from it are summarized as a comment in the test file.
sumeerbhola
force-pushed
the
sublevel
branch
4 times, most recently
from
2a527ca to
9a982e4
Compare
and picking up files in L0 that overlapped with Lbase files that were not picked. Resulting in two files in Lbase overlapping.
- reduce intra-L0. Lbase and higher levels should be given priority only if the cost of a high score is high (measured by currentByteRatios). This mostly eliminated intra-L0. - not have huge compactions involving L0. Try to cutoff at 500MB - don't completely starve Lbase: if the score becomes very high do them even if it sacrifices L0=>Lbase.
The previous code was generating more wide files than desirable.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 16, 2020
This change introduces L0SubLevels, a data structure to encaptulate all information to compute L0 Sublevels, flush split keys, and base / intra-L0 compactions. This data structure will be generated every time there's a file addition/deletion in L0, the integration bits will come in future PRs. The methods to pick base / intra-L0 compactions will come in a separate PR. Captures some of the largest code pieces of cockroachdb#563, and effectively replaces a part of that PR. Sumeer wrote the bulk of this code as part of his prototype, so thanks Sumeer for his work. Most of these functions are dead code at the moment, only invoked by the provided datadriven and non-datadriven tests.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 20, 2020
This change introduces L0SubLevels, a data structure to encaptulate all information to compute L0 Sublevels, flush split keys, and base / intra-L0 compactions. This data structure will be generated every time there's a file addition/deletion in L0, the integration bits will come in future PRs. The methods to pick base / intra-L0 compactions will come in a separate PR. Captures some of the largest code pieces of cockroachdb#563, and effectively replaces a part of that PR. Sumeer wrote the bulk of this code as part of his prototype, so thanks Sumeer for his work. Most of these functions are dead code at the moment, only invoked by the provided datadriven and non-datadriven tests.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 21, 2020
This change introduces L0SubLevels, a data structure to encaptulate all information to compute L0 Sublevels, flush split keys, and base / intra-L0 compactions. This data structure will be generated every time there's a file addition/deletion in L0, the integration bits will come in future PRs. The methods to pick base / intra-L0 compactions will come in a separate PR. Captures some of the largest code pieces of cockroachdb#563, and effectively replaces a part of that PR. Sumeer wrote the bulk of this code as part of his prototype, so thanks Sumeer for his work. Most of these functions are dead code at the moment, only invoked by the provided datadriven and non-datadriven tests.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 21, 2020
This change introduces L0SubLevels, a data structure to encaptulate all information to compute L0 Sublevels, flush split keys, and base / intra-L0 compactions. This data structure will be generated every time there's a file addition/deletion in L0, the integration bits will come in future PRs. The methods to pick base / intra-L0 compactions will come in a separate PR. Captures some of the largest code pieces of cockroachdb#563, and effectively replaces a part of that PR. Sumeer wrote the bulk of this code as part of his prototype, so thanks Sumeer for his work. Most of these functions are dead code at the moment, only invoked by the provided datadriven and non-datadriven tests.
itsbilal
added a commit
that referenced
this pull request
Apr 21, 2020
This change introduces L0SubLevels, a data structure to encaptulate all information to compute L0 Sublevels, flush split keys, and base / intra-L0 compactions. This data structure will be generated every time there's a file addition/deletion in L0, the integration bits will come in future PRs. The methods to pick base / intra-L0 compactions will come in a separate PR. Captures some of the largest code pieces of #563, and effectively replaces a part of that PR. Sumeer wrote the bulk of this code as part of his prototype, so thanks Sumeer for his work. Most of these functions are dead code at the moment, only invoked by the provided datadriven and non-datadriven tests.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 23, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 27, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 28, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 29, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 30, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
Apr 30, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
May 1, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
May 1, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
May 4, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this pull request
May 4, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of cockroachdb#563. Thanks Sumeer for his work, most of this was written by him.
itsbilal
added a commit
that referenced
this pull request
May 5, 2020
This change adds methods to L0SubLevels to help pick, score, and generate L0 -> LBase, and L0 -> L0 compactions, based on information captured in the data structure about L0 sublevels. These functions will be called from in compaction.go and compaction_picker.go in a future change. Also adds associated datadriven unit tests, and a benchmark. Covers a large part of #563. Thanks Sumeer for his work, most of this was written by him.
This was referenced Jun 22, 2020
Closed
|
This was taken over by @itsbilal's work. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
measure stack depth, compaction score, pick L0->Lbase and intra-L0
compactions.
The main construction here (beyond the sub-levels introduced in
of intervals is O(N) where N is the number of files in L0, but
we expect that typically each interval will have a file count that
is sublinear in N. These intervals allow for precisely tracking
the depth and are used as the seed for computing compactions.