Use segmented Slice in SliceDictionaryWriter #15956
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arunthirupathi
force-pushed
the
segmented_slice_builder_2
branch
from
dac8e5c
to
668c781
Compare
arunthirupathi
changed the title
arunthirupathi
force-pushed
the
segmented_slice_builder_2
branch
from
668c781
to
1bbc8e0
Compare
arunthirupathi
force-pushed
the
segmented_slice_builder_2
branch
from
1bbc8e0
to
4762596
Compare
|
Here is the performance comparison before and after this change. Base is before this change. Segmented is with this change. Note Direct in base and direct in segmented has no change between them and they should be 100% and the result is close enough.
|
arunthirupathi
force-pushed
the
segmented_slice_builder_2
branch
from
4762596
to
942e3ca
Compare
highker
reviewed
Apr 22, 2021
presto-orc/src/main/java/com/facebook/presto/orc/writer/SegmentedSliceBlockBuilder.java
Outdated
Show resolved
Hide resolved
presto-orc/src/main/java/com/facebook/presto/orc/writer/SegmentedSliceBlockBuilder.java
Outdated
Show resolved
Hide resolved
presto-orc/src/test/java/com/facebook/presto/orc/writer/TestSegmentedSliceBlockBuilder.java
Outdated
Show resolved
Hide resolved
presto-orc/src/test/java/com/facebook/presto/orc/writer/TestSegmentedSliceBlockBuilder.java
Outdated
Show resolved
Hide resolved
Comment on lines
294
to
291
| public BlockBuilder newBlockBuilderLike(BlockBuilderStatus blockBuilderStatus, int expectedEntries) | ||
| { | ||
| if (blockBuilderStatus != null) { | ||
| throw new UnsupportedOperationException("Not yet implemented"); | ||
| } |
There was a problem hiding this comment.
blockBuilderStatus is actually fairly important. QQ: Is newBlockBuilderLike used anywhere in orc package? If not, just throw?
There was a problem hiding this comment.
This blockBuilder is only used by the SliceDictionaryBuilder and it passes in null for the blockBuilderStatus.
presto-orc/src/main/java/com/facebook/presto/orc/writer/SegmentedSliceBlockBuilder.java
Show resolved
Hide resolved
|
|
||
| private final DynamicSliceOutput openSliceOutput; | ||
|
|
||
| private int openSegment; |
There was a problem hiding this comment.
s/openSegment/openSegmentIndex, if I read the code correctly
There was a problem hiding this comment.
You are right, renamed it.
presto-orc/src/main/java/com/facebook/presto/orc/writer/SegmentedSliceBlockBuilder.java
Outdated
Show resolved
Hide resolved
| offsets[openSegment][openSegmentOffset] = openSliceOutput.size(); | ||
| if (openSegmentOffset == SegmentHelper.SEGMENT_SIZE) { | ||
| // Add the current finalized slice to closedSlices | ||
| Slice slice = openSliceOutput.copySlice(); |
There was a problem hiding this comment.
We do a copy to save space? We usually don't call this method to avoid heavy GC. Maybe slice() is good enough.
There was a problem hiding this comment.
Slice() gives the view of the object. After the segment is full, the bytes[] are copied to the copySllice and the dynamicSlliceOuptut is reset and reused for the new segment.
presto-orc/src/main/java/com/facebook/presto/orc/writer/SegmentedSliceBlockBuilder.java
Outdated
Show resolved
Hide resolved
arunthirupathi
force-pushed
the
segmented_slice_builder_2
branch
from
942e3ca
to
8c884ea
Compare
arunthirupathi
force-pushed
the
segmented_slice_builder_2
branch
2 times, most recently
from
21cd7b1
to
9bf70ba
Compare
highker
approved these changes
Apr 25, 2021
presto-orc/src/main/java/com/facebook/presto/orc/writer/SegmentedSliceBlockBuilder.java
Outdated
Show resolved
Hide resolved
presto-orc/src/test/java/com/facebook/presto/orc/writer/TestSegmentedSliceBlockBuilder.java
Outdated
Show resolved
Hide resolved
presto-orc/src/test/java/com/facebook/presto/orc/writer/TestSegmentedSliceBlockBuilder.java
Outdated
Show resolved
Hide resolved
Store elements of dictionary in Segmented Slices, instead of one contiguous segment. When the number of elements in the dictionary is less than 100,000 there is no noticeable performance degradation. When the number of elements in the dictionary reaches 10,000,000 sorting/comparing the element needs to compute segment/offset which makes it worse by 10%. But this is an unlikely case.
arunthirupathi
force-pushed
the
segmented_slice_builder_2
branch
from
9bf70ba
to
16e9b84
Compare
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Store elements of dictionary in Segmented Slices, instead of
one contiguous segment. When the number of elements
in the dictionary is less than 100,000 there is no noticeable
performance degradation. When the number of elements in the
dictionary reaches 10,000,000 sorting/comparing the element
needs to compute segment/offset which makes it worse by 10%.
But this is an unlikely case.
Test plan -
Added new test cases for the SegmentedSlices.
Dictionary is covered by existing tests.