Skip to content

Commit

Permalink
[MINOR] Enable pyspark test in local mode
Browse files Browse the repository at this point in the history
### What is this PR for?
Enabling test for pyspark in local mode

### What type of PR is it?
[Improvement]

### Todos
* [x] - Add spark configuration and
* [x] - Edit a logic for finding spark home

### What is the Jira issue?
N/A

### How should this be tested?
1. Download spark under {ZEPPELIN_HOME}
1. run `testing/startSparkCluster.sh`
1. `mvn clean package -Pspark-1.6 -Ppyspark -Pyarn -DskipTests`
1. Run `ZeppelinSparkClusterTest`

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Jongyoul Lee <[email protected]>

Closes #1201 from jongyoul/minor/enable-pyspark-test-in-local-mode and squashes the following commits:

67ccc2d [Jongyoul Lee] Added spark configuration while not using CI
  • Loading branch information
jongyoul committed Jul 28, 2016
1 parent 5afd7ca commit 7cc908b
Showing 1 changed file with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,8 @@ protected static void startUp() throws Exception {

String sparkHome = getSparkHome();
if (sparkHome != null) {
sparkIntpSetting.getProperties().setProperty("master", "spark://" + getHostname() + ":7071");
sparkIntpSetting.getProperties().setProperty("spark.cores.max", "2");
// set spark home for pyspark
sparkIntpSetting.getProperties().setProperty("spark.home", sparkHome);
pySpark = true;
Expand All @@ -192,7 +194,7 @@ private static String getHostname() {
}

private static String getSparkHome() {
String sparkHome = getSparkHomeRecursively(new File(System.getProperty("user.dir")));
String sparkHome = getSparkHomeRecursively(new File(System.getProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_HOME.getVarName())));
System.out.println("SPARK HOME detected " + sparkHome);
return sparkHome;
}
Expand Down

0 comments on commit 7cc908b

Please sign in to comment.