Skip to content

Commit

Permalink
[SPARK-1896] Respect spark.master (and --master) before MASTER in spa…
Browse files Browse the repository at this point in the history
…rk-shell

The hierarchy for configuring the Spark master in the shell is as follows:
```
MASTER > --master > spark.master (spark-defaults.conf)
```
This is inconsistent with the way we run normal applications, which is:
```
--master > spark.master (spark-defaults.conf) > MASTER
```

I was trying to run a shell locally on a standalone cluster launched through the ec2 scripts, which automatically set `MASTER` in spark-env.sh. It was surprising to me that `--master` didn't take effect, considering that this is the way we tell users to set their masters [here](http://people.apache.org/~pwendell/spark-1.0.0-rc7-docs/scala-programming-guide.html#initializing-spark).

Author: Andrew Or <[email protected]>

Closes #846 from andrewor14/shell-master and squashes the following commits:

2cb81c9 [Andrew Or] Respect spark.master before MASTER in REPL

(cherry picked from commit cce7745)
Signed-off-by: Tathagata Das <[email protected]>
  • Loading branch information
andrewor14 authored and tdas committed May 23, 2014
1 parent 23cc40e commit c3b4065
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
Original file line number Diff line number Diff line change
Expand Up @@ -962,11 +962,10 @@ class SparkILoop(in0: Option[BufferedReader], protected val out: JPrintWriter,
private def getMaster(): String = {
val master = this.master match {
case Some(m) => m
case None => {
case None =>
val envMaster = sys.env.get("MASTER")
val propMaster = sys.props.get("spark.master")
envMaster.orElse(propMaster).getOrElse("local[*]")
}
propMaster.orElse(envMaster).getOrElse("local[*]")
}
master
}
Expand Down

0 comments on commit c3b4065

Please sign in to comment.