Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

[NSE-857] Further optimizations of validity buffer split #915

Merged
merged 4 commits into from
May 11, 2022

Conversation

FelixYBW
Copy link
Collaborator

@FelixYBW FelixYBW commented May 11, 2022

What changes were proposed in this pull request?

Previous implementation of validity buffer split performance is too bad.

no validity buffer 1.55
w/ validity buffer, before opt 2.51
w/ validity buffer, after opt 2.01

From profile data, validity buffer split still takes more time than value copy. Most of the time is spending on the first touch of dst buffer

67.32%  BenchmarkShuffl  libspark_columnar_jni.so  [.] sparkcolumnarplugin::shuffle::Splitter::SplitBoolType                                                                              
25.96%  BenchmarkShuffl  libspark_columnar_jni.so  [.] sparkcolumnarplugin::shuffle::Splitter::SplitFixedWidthValueBuffer                                                                 
 1.81%  BenchmarkShuffl  libspark_columnar_jni.so  [.] sparkcolumnarplugin::shuffle::RoundRobinSplitter::ComputeAndCountPartitionId                                                       
 1.48%  BenchmarkShuffl  libc-2.17.so              [.] __memmove_ssse3_back                                                                                                               
 1.31%  BenchmarkShuffl  libspark_columnar_jni.so  [.] sparkcolumnarplugin::shuffle::Splitter::DoSplit                                                                                    
 1.02%  BenchmarkShuffl  libspark_columnar_jni.so  [.] sparkcolumnarplugin::shuffle::Splitter::SplitFixedWidthValidityBuffer    

How was this patch tested?

@github-actions
Copy link

#857

@FelixYBW FelixYBW merged commit 3e97b3d into oap-project:main May 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant