From 7692a08b7a68efdac7abbf966471b17d3636749c Mon Sep 17 00:00:00 2001 From: Yuan Zhou Date: Thu, 10 Jun 2021 14:51:45 +0800 Subject: [PATCH] use arrow-unsafe allocator by default This patch makes arrow-unsafe allocator as default profile arrow-netty will instance netty memory pool to reduce the alloc/dealloc overhead which introduced big memory footprint. local perf tests showed the performace is not heavily impacted after switch to arrow-unsafe one more benefit is we could remove the -XX:MaxDirectMemorySize flag which also impacts the vanilla Spark memory usage Signed-off-by: Yuan Zhou --- docs/Configuration.md | 1 - pom.xml | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/Configuration.md b/docs/Configuration.md index 8b6615687..92d01aa53 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -11,7 +11,6 @@ You can add these configuration into spark-defaults.conf to enable or disable th | spark.executorEnv.CC | To set up the location of gcc | /path/to/gcc/ | | spark.executor.memory| To set up how much memory to be used for Spark Executor. | | | spark.memory.offHeap.size| To set up how much memory to be used for Java OffHeap.
Please notice Native SQL Engine will leverage this setting to allocate memory space for native usage even offHeap is disabled.
The value is based on your system and it is recommended to set it larger if you are facing Out of Memory issue in Native SQL Engine | 30G | -| spark.executor.extraJavaOptions | To set up how much Direct Memory to be used for Native SQL Engine. The value is based on your system and it is recommended to set it larger if you are facing Out of Memory issue in Native SQL Engine | -XX:MaxDirectMemorySize=30G | | spark.sql.sources.useV1SourceList | Choose to use V1 source | avro | | spark.sql.join.preferSortMergeJoin | To turn off preferSortMergeJoin in Spark | false | | spark.sql.extensions | To turn on Native SQL Engine Plugin | com.intel.oap.ColumnarPlugin | diff --git a/pom.xml b/pom.xml index 2f2cf80e6..557bcdf66 100644 --- a/pom.xml +++ b/pom.xml @@ -213,7 +213,7 @@ arrow-netty - true + false arrow-memory-netty @@ -222,7 +222,7 @@ arrow-unsafe - false + true arrow-memory-unsafe