Skip to content

Commit

Permalink
[SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1)
Browse files Browse the repository at this point in the history 
### What changes were proposed in this pull request?

This PR regenerates the `sql/core` benchmarks in JDK8/11 to compare the result. In general, we compare the ratio instead of the time. However, in this PR, the average time is compared. This PR should be considered as a rough comparison.

**A. EXPECTED CASES(JDK11 is faster in general)**
- [x] BloomFilterBenchmark (JDK11 is faster except one case)
- [x] BuiltInDataSourceWriteBenchmark (JDK11 is faster at CSV/ORC)
- [x] CSVBenchmark (JDK11 is faster except five cases)
- [x] ColumnarBatchBenchmark (JDK11 is faster at `boolean`/`string` and some cases in `int`/`array`)
- [x] DatasetBenchmark (JDK11 is faster with `string`, but is slower for `long` type)
- [x] ExternalAppendOnlyUnsafeRowArrayBenchmark (JDK11 is faster except two cases)
- [x] ExtractBenchmark (JDK11 is faster except HOUR/MINUTE/SECOND/MILLISECONDS/MICROSECONDS)
- [x] HashedRelationMetricsBenchmark (JDK11 is faster)
- [x] JSONBenchmark (JDK11 is much faster except eight cases)
- [x] JoinBenchmark (JDK11 is faster except five cases)
- [x] OrcNestedSchemaPruningBenchmark (JDK11 is faster in nine cases)
- [x] PrimitiveArrayBenchmark (JDK11 is faster)
- [x] SortBenchmark (JDK11 is faster except `Arrays.sort` case)
- [x] UDFBenchmark (N/A, values are too small)
- [x] UnsafeArrayDataBenchmark (JDK11 is faster except one case)
- [x] WideTableBenchmark (JDK11 is faster except two cases)

**B. CASES WE NEED TO INVESTIGATE MORE LATER**
- [x] AggregateBenchmark (JDK11 is slower in general)
- [x] CompressionSchemeBenchmark (JDK11 is slower in general except `string`)
- [x] DataSourceReadBenchmark (JDK11 is slower in general)
- [x] DateTimeBenchmark (JDK11 is slightly slower in general except `parsing`)
- [x] MakeDateTimeBenchmark (JDK11 is slower except two cases)
- [x] MiscBenchmark (JDK11 is slower except ten cases)
- [x] OrcV2NestedSchemaPruningBenchmark (JDK11 is slower)
- [x] ParquetNestedSchemaPruningBenchmark (JDK11 is slower except six cases)
- [x] RangeBenchmark (JDK11 is slower except one case)

`FilterPushdownBenchmark/InExpressionBenchmark/WideSchemaBenchmark` will be compared later because it took long timer.

### Why are the changes needed?

According to the result, there are some difference between JDK8/JDK11.
This will be a baseline for the future improvement and comparison. Also, as a reproducible  environment, the following environment is used.
- Instance: `r3.xlarge`
- OS: `CentOS Linux release 7.5.1804 (Core)`
- JDK:
  - `OpenJDK Runtime Environment (build 1.8.0_222-b10)`
  - `OpenJDK Runtime Environment 18.9 (build 11.0.4+11-LTS)`

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

This is a test-only PR. We need to run benchmark.

Closes #26003 from dongjoon-hyun/SPARK-29320.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
@dongjoon-hyun
dongjoon-hyun committed Oct 3, 2019
1 parent 7aca0dd commit 854a0f7
Show file tree
Hide file tree
Showing 50 changed files with 3,276 additions and 1,362 deletions.
143 changes: 143 additions & 0 deletions sql/core/benchmarks/AggregateBenchmark-jdk11-results.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
================================================================================================
aggregate without grouping
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
agg w/o group: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
agg w/o group wholestage off 55644 59484 NaN 37.7 26.5 1.0X
agg w/o group wholestage on 896 906 8 2340.7 0.4 62.1X


================================================================================================
stat functions
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
stddev: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
stddev wholestage off 8655 9022 519 12.1 82.5 1.0X
stddev wholestage on 1306 1323 13 80.3 12.5 6.6X

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
kurtosis: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
kurtosis wholestage off 40795 41041 349 2.6 389.0 1.0X
kurtosis wholestage on 1441 1468 22 72.8 13.7 28.3X


================================================================================================
aggregate with linear keys
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Aggregate w keys: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
codegen = F 10559 10956 562 7.9 125.9 1.0X
codegen = T hashmap = F 6533 6567 34 12.8 77.9 1.6X
codegen = T hashmap = T 1362 1377 22 61.6 16.2 7.8X


================================================================================================
aggregate with randomized keys
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Aggregate w keys: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
codegen = F 12631 12634 3 6.6 150.6 1.0X
codegen = T hashmap = F 8434 8478 44 9.9 100.5 1.5X
codegen = T hashmap = T 2484 2598 117 33.8 29.6 5.1X


================================================================================================
aggregate with string key
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Aggregate w string key: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
codegen = F 4173 4204 44 5.0 199.0 1.0X
codegen = T hashmap = F 2664 2721 81 7.9 127.0 1.6X
codegen = T hashmap = T 1178 1219 59 17.8 56.2 3.5X


================================================================================================
aggregate with decimal key
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Aggregate w decimal key: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
codegen = F 3740 3747 9 5.6 178.3 1.0X
codegen = T hashmap = F 2398 2528 184 8.7 114.3 1.6X
codegen = T hashmap = T 638 644 7 32.9 30.4 5.9X


================================================================================================
aggregate with multiple key types
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Aggregate w multiple keys: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
codegen = F 6874 6896 31 3.1 327.8 1.0X
codegen = T hashmap = F 3866 3886 28 5.4 184.3 1.8X
codegen = T hashmap = T 2619 2641 31 8.0 124.9 2.6X


================================================================================================
max function bytecode size of wholestagecodegen
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
max function bytecode size: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
codegen = F 793 806 19 0.8 1209.7 1.0X
codegen = T hugeMethodLimit = 10000 401 456 61 1.6 611.2 2.0X
codegen = T hugeMethodLimit = 1500 694 715 19 0.9 1059.3 1.1X


================================================================================================
cube
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
cube: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
cube wholestage off 3616 3630 19 1.4 689.8 1.0X
cube wholestage on 1819 1866 41 2.9 347.0 2.0X


================================================================================================
hash and BytesToBytesMap
================================================================================================

OpenJDK 64-Bit Server VM 11.0.4+11-LTS on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
BytesToBytesMap: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
UnsafeRowhash 321 321 1 65.4 15.3 1.0X
murmur3 hash 145 145 0 144.5 6.9 2.2X
fast hash 70 71 1 298.9 3.3 4.6X
arrayEqual 198 199 2 105.7 9.5 1.6X
Java HashMap (Long) 132 136 3 158.6 6.3 2.4X
Java HashMap (two ints) 152 156 4 138.3 7.2 2.1X
Java HashMap (UnsafeRow) 819 839 17 25.6 39.1 0.4X
LongToUnsafeRowMap (opt=false) 459 472 25 45.7 21.9 0.7X
LongToUnsafeRowMap (opt=true) 107 108 1 195.5 5.1 3.0X
BytesToBytesMap (off Heap) 1012 1019 10 20.7 48.3 0.3X
BytesToBytesMap (on Heap) 963 974 18 21.8 45.9 0.3X
Aggregate HashMap 41 43 2 515.1 1.9 7.9X


Loading

0 comments on commit 854a0f7

Please sign in to comment.