[SPARK-2590][SQL] Added option to handle incremental collection, disabled by default #1853
Conversation
|
QA tests have started for PR 1853. This patch merges cleanly. |
|
Removed "thriftServer" from the property name, since Spark SQL core doesn't know anything about Hive Thrift server, and non-Hive JDBC/ODBC interface can also leverage this property in the future. |
|
QA results for PR 1853: |
| @@ -30,6 +30,7 @@ private[spark] object SQLConf { | |||
| val SHUFFLE_PARTITIONS = "spark.sql.shuffle.partitions" | |||
| val CODEGEN_ENABLED = "spark.sql.codegen" | |||
| val DIALECT = "spark.sql.dialect" | |||
| val INCREMENTAL_COLLECT_ENABLED = "spark.sql.incrementalCollect" | |||
There was a problem hiding this comment.
Lets put this under spark.sql.thriftServer.... Also, what do you think about putting it in with the thrift server instead? It seems kinda odd to me to be configuring things for the thrift server in a place where the code doesn't even exist. This is how we do things for Hive now (see HiveContext in #1819 ).
There was a problem hiding this comment.
OK. I put it in SQLConf because of exactly the same reason Zongheng mentioned, and also had mixed feelings about this. I'd prefer to have something like Hive ConfVars to define all configurations, but let's do it later.
|
QA tests have started for PR 1853. This patch merges cleanly. |
|
QA results for PR 1853: |
|
Thanks! I've merged this to master and 1.1 |
…bled by default JIRA issue: [SPARK-2590](https://issues.apache.org/jira/browse/SPARK-2590) Author: Cheng Lian <[email protected]> Closes #1853 from liancheng/inc-collect-option and squashes the following commits: cb3ea45 [Cheng Lian] Moved incremental collection option to Thrift server 43ce3aa [Cheng Lian] Changed incremental collect option name 623abde [Cheng Lian] Added option to handle incremental collection, disabled by default (cherry picked from commit 21a95ef) Signed-off-by: Michael Armbrust <[email protected]>
…bled by default JIRA issue: [SPARK-2590](https://issues.apache.org/jira/browse/SPARK-2590) Author: Cheng Lian <[email protected]> Closes apache#1853 from liancheng/inc-collect-option and squashes the following commits: cb3ea45 [Cheng Lian] Moved incremental collection option to Thrift server 43ce3aa [Cheng Lian] Changed incremental collect option name 623abde [Cheng Lian] Added option to handle incremental collection, disabled by default
JIRA issue: SPARK-2590