Skip to content

Commit

Permalink
Merge pull request apache#373 from zhou-jiang/pie-spark-2.3.2
Browse files Browse the repository at this point in the history 
<rdar://problem/49703974> [Apache Spark] URL.setURLStreamHandlerFactory causing incompatible with Cassandra Bulk Writer
  • Loading branch information
@jiangzho
jiangzho authored and GitHub Enterprise committed Apr 30, 2019
2 parents aabb2e4 + 5ddef27 commit 8c4374a
Show file tree
Hide file tree
Showing 39 changed files with 61 additions and 43 deletions.
2 changes: 1 addition & 1 deletion assembly/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/kvstore/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/network-common/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/network-shuffle/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/network-yarn/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/sketch/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/tags/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/unsafe/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion core/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion examples/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/docker-integration-tests/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/flume-assembly/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/flume-sink/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/flume/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kafka-0-10-assembly/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kafka-0-10-sql/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kafka-0-10/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kafka-0-8-assembly/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kafka-0-8/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kinesis-asl-assembly/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/kinesis-asl/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion external/spark-ganglia-lgpl/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion graphx/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion hadoop-cloud/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion launcher/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion mllib-local/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion mllib/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<packaging>pom</packaging>
<name>Spark Project Parent POM</name>
<url>http://spark.apache.org/</url>
Expand Down
2 changes: 1 addition & 1 deletion repl/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/core/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion resource-managers/mesos/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion resource-managers/yarn/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion sql/core/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
28 changes: 23 additions & 5 deletions sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,19 @@ import org.apache.spark.util.{MutableURLClassLoader, Utils}
*/
private[sql] class SharedState(val sparkContext: SparkContext) extends Logging {

private var initialized = false

initialize()

private def initialize() {
synchronized {
if (!initialized) {
SharedState.setUrlStreamHanderFactoryIfNeeded(sparkContext.conf)
initialized = true
}
}
}

// Load hive-site.xml into hadoopConf and determine the warehouse path we want to use, based on
// the config from both hive and Spark SQL. Finally set the warehouse config value to sparkConf.
val warehousePath: String = {
Expand Down Expand Up @@ -153,11 +166,16 @@ private[sql] class SharedState(val sparkContext: SparkContext) extends Logging {
}

object SharedState extends Logging {
try {
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
} catch {
case e: Error =>
logWarning("URL.setURLStreamHandlerFactory failed to set FsUrlStreamHandlerFactory")

private def setUrlStreamHanderFactoryIfNeeded(conf: SparkConf) = {
try {
if (conf.getBoolean("spark.FsUrlStreamHandlerFactory.enabled", true)) {
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
}
} catch {
case e: Error =>
logWarning("URL.setURLStreamHandlerFactory failed to set FsUrlStreamHandlerFactory")
}
}

private val HIVE_EXTERNAL_CATALOG_CLASS_NAME = "org.apache.spark.sql.hive.HiveExternalCatalog"
Expand Down
2 changes: 1 addition & 1 deletion sql/hive-thriftserver/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion sql/hive/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion streaming/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion tools/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>com.apple.pie.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.3.2-pie1.0.3-SNAPSHOT</version>
<version>2.3.2-pie1.0.4-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down

0 comments on commit 8c4374a

Please sign in to comment.