Skip to content

Commit

Permalink
[SPARK-20096][SPARK SUBMIT][MINOR] Expose the right queue name not nu…
Browse files Browse the repository at this point in the history
…ll if set by --conf or configure file

## What changes were proposed in this pull request?

while submit apps with -v or --verbose, we can print the right queue name, but if we set a queue name with `spark.yarn.queue` by --conf or in the spark-default.conf, we just got `null`  for the queue in Parsed arguments.
```
bin/spark-shell -v --conf spark.yarn.queue=thequeue
Using properties file: /home/hadoop/spark-2.1.0-bin-apache-hdp2.7.3/conf/spark-defaults.conf
....
Adding default property: spark.yarn.queue=default
Parsed arguments:
  master                  yarn
  deployMode              client
  ...
  queue                   null
  ....
  verbose                 true
Spark properties used, including those specified through
 --conf and those from the properties file /home/hadoop/spark-2.1.0-bin-apache-hdp2.7.3/conf/spark-defaults.conf:
  spark.yarn.queue -> thequeue
  ....
```
## How was this patch tested?

ut and local verify

Author: Kent Yao <[email protected]>

Closes #17430 from yaooqinn/SPARK-20096.
  • Loading branch information
yaooqinn authored and srowen committed Mar 30, 2017
1 parent 258bff2 commit e9d268f
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
.orNull
numExecutors = Option(numExecutors)
.getOrElse(sparkProperties.get("spark.executor.instances").orNull)
queue = Option(queue).orElse(sparkProperties.get("spark.yarn.queue")).orNull
keytab = Option(keytab).orElse(sparkProperties.get("spark.yarn.keytab")).orNull
principal = Option(principal).orElse(sparkProperties.get("spark.yarn.principal")).orNull

Expand Down
11 changes: 11 additions & 0 deletions core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,17 @@ class SparkSubmitSuite
appArgs.childArgs should be (Seq("--master", "local", "some", "--weird", "args"))
}

test("print the right queue name") {
val clArgs = Seq(
"--name", "myApp",
"--class", "Foo",
"--conf", "spark.yarn.queue=thequeue",
"userjar.jar")
val appArgs = new SparkSubmitArguments(clArgs)
appArgs.queue should be ("thequeue")
appArgs.toString should include ("thequeue")
}

test("specify deploy mode through configuration") {
val clArgs = Seq(
"--master", "yarn",
Expand Down

0 comments on commit e9d268f

Please sign in to comment.