Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-40471][BUILD] Upgrade RoaringBitmap to 0.9.32 #37914

Closed
wants to merge 3 commits into from
Closed

[SPARK-40471][BUILD] Upgrade RoaringBitmap to 0.9.32 #37914

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f3ab353
upgrade version
LuciferYang Sep 16, 2022
8302dae
update bench result
LuciferYang Sep 16, 2022
2312bb2
Merge branch 'upmaster' into SPARK-40471
LuciferYang Sep 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions core/benchmarks/MapStatusesConvertBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
MapStatuses Convert Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1031-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
OpenJDK 64-Bit Server VM 11.0.16+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
MapStatuses Convert: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Num Maps: 50000 Fetch partitions:500 1324 1333 7 0.0 1324283680.0 1.0X
Num Maps: 50000 Fetch partitions:1000 2650 2670 32 0.0 2650318387.0 0.5X
Num Maps: 50000 Fetch partitions:1500 4018 4059 53 0.0 4017921009.0 0.3X
Num Maps: 50000 Fetch partitions:500 1269 1276 8 0.0 1268666001.0 1.0X
Num Maps: 50000 Fetch partitions:1000 2672 2695 39 0.0 2671542753.0 0.5X
Num Maps: 50000 Fetch partitions:1500 4034 4069 50 0.0 4033696987.0 0.3X


10 changes: 5 additions & 5 deletions core/benchmarks/MapStatusesConvertBenchmark-jdk17-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
MapStatuses Convert Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1031-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
OpenJDK 64-Bit Server VM 17.0.4+8-LTS on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
MapStatuses Convert: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Num Maps: 50000 Fetch partitions:500 1092 1104 22 0.0 1091691925.0 1.0X
Num Maps: 50000 Fetch partitions:1000 2172 2192 29 0.0 2171702137.0 0.5X
Num Maps: 50000 Fetch partitions:1500 3268 3291 27 0.0 3267904436.0 0.3X
Num Maps: 50000 Fetch partitions:500 1228 1238 17 0.0 1228191051.0 1.0X
Num Maps: 50000 Fetch partitions:1000 2380 2393 16 0.0 2379601524.0 0.5X
Num Maps: 50000 Fetch partitions:1500 3803 3857 55 0.0 3802550172.0 0.3X


8 changes: 4 additions & 4 deletions core/benchmarks/MapStatusesConvertBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
MapStatuses Convert Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1031-azure
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
MapStatuses Convert: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Num Maps: 50000 Fetch partitions:500 1001 1033 36 0.0 1000638934.0 1.0X
Num Maps: 50000 Fetch partitions:1000 1699 1705 7 0.0 1699358972.0 0.6X
Num Maps: 50000 Fetch partitions:1500 2647 2855 314 0.0 2646904255.0 0.4X
Num Maps: 50000 Fetch partitions:500 1179 1187 13 0.0 1178581948.0 1.0X
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me double check the base data

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run this bench without this pr

================================================================================================
MapStatuses Convert Benchmark
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1019-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
MapStatuses Convert:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Num Maps: 50000 Fetch partitions:500               1150           1156           5          0.0  1149981444.0       1.0X
Num Maps: 50000 Fetch partitions:1000              2304           2322          18          0.0  2304318021.0       0.5X
Num Maps: 50000 Fetch partitions:1500              3557           3792         377          0.0  3556704966.0       0.3X

Num Maps: 50000 Fetch partitions:1000 2237 2257 21 0.0 2236562602.0 0.5X
Num Maps: 50000 Fetch partitions:1500 3448 3680 367 0.0 3447793753.0 0.3X


4 changes: 2 additions & 2 deletions dev/deps/spark-deps-hadoop-2-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
HikariCP/2.5.1//HikariCP-2.5.1.jar
JLargeArrays/1.5//JLargeArrays-1.5.jar
JTransforms/3.1//JTransforms-3.1.jar
RoaringBitmap/0.9.31//RoaringBitmap-0.9.31.jar
RoaringBitmap/0.9.32//RoaringBitmap-0.9.32.jar
ST4/4.0.4//ST4-4.0.4.jar
activation/1.1.1//activation-1.1.1.jar
aircompressor/0.21//aircompressor-0.21.jar
Expand Down Expand Up @@ -242,7 +242,7 @@ scala-library/2.12.16//scala-library-2.12.16.jar
scala-parser-combinators_2.12/1.1.2//scala-parser-combinators_2.12-1.1.2.jar
scala-reflect/2.12.16//scala-reflect-2.12.16.jar
scala-xml_2.12/2.1.0//scala-xml_2.12-2.1.0.jar
shims/0.9.31//shims-0.9.31.jar
shims/0.9.32//shims-0.9.32.jar
slf4j-api/1.7.36//slf4j-api-1.7.36.jar
snakeyaml/1.31//snakeyaml-1.31.jar
snappy-java/1.1.8.4//snappy-java-1.1.8.4.jar
Expand Down
4 changes: 2 additions & 2 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
HikariCP/2.5.1//HikariCP-2.5.1.jar
JLargeArrays/1.5//JLargeArrays-1.5.jar
JTransforms/3.1//JTransforms-3.1.jar
RoaringBitmap/0.9.31//RoaringBitmap-0.9.31.jar
RoaringBitmap/0.9.32//RoaringBitmap-0.9.32.jar
ST4/4.0.4//ST4-4.0.4.jar
activation/1.1.1//activation-1.1.1.jar
aircompressor/0.21//aircompressor-0.21.jar
Expand Down Expand Up @@ -229,7 +229,7 @@ scala-library/2.12.16//scala-library-2.12.16.jar
scala-parser-combinators_2.12/1.1.2//scala-parser-combinators_2.12-1.1.2.jar
scala-reflect/2.12.16//scala-reflect-2.12.16.jar
scala-xml_2.12/2.1.0//scala-xml_2.12-2.1.0.jar
shims/0.9.31//shims-0.9.31.jar
shims/0.9.32//shims-0.9.32.jar
slf4j-api/1.7.36//slf4j-api-1.7.36.jar
snakeyaml/1.31//snakeyaml-1.31.jar
snappy-java/1.1.8.4//snappy-java-1.1.8.4.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -825,7 +825,7 @@
<dependency>
<groupId>org.roaringbitmap</groupId>
<artifactId>RoaringBitmap</artifactId>
<version>0.9.31</version>
<version>0.9.32</version>
</dependency>

<!-- Netty Begin -->
Expand Down