Build and Package

In order for EclairJS Node to talk to Apache Spark, it needs a instance of Apache Toree running and Toree must be able to connect to your Spark master.

Prerequisites

• Java 8 update 70 or higher

Instructions

1. Download Apache Spark built with Hadoop 2.6 and extract it from the archive.

2. Install Jupyter (pip install jupyter for example) and the Jupyter Kernel Gateway (pip install jupyter-kernel-gateway)

3. Install Apache Toree (pip install toree===0.1.0.dev8 for example)

• Be sure to note down where Apache Toree was installed:

$ pip show toree
...
Location: /usr/local/lib/python2.7/site-packages

4. Download the EclairJS Nashorn JAR file from Maven (http://repo2.maven.org/maven2/org/eclairjs/eclairjs-nashorn/${ECLAIRJS_VERSION}/eclairjs-nashorn-${ECLAIRJS_VERSION}-jar-with-dependencies.jar) (replace ${ECLAIRJS_VERSION} with the version you are using)

5. Download kernel.json and replace the following:

• /usr/local/share/jupyter/kernels/apache_toree_scala/bin/run.sh with the location of your installed Apache Toree (/usr/local/lib/python2.7/site-packages/toree/bin/run.sh for example on OSX, see the Location in step 3)
• "SPARK_HOME" should point to the extracted Apache Spark directory (spark-1.6.0-bin-hadoop2.6)
• /opt/nashorn/lib/eclairjs.jar should point at the JAR file downloaded in step 4.

6. Figure out your Jupyter data directory by running:

$ jupyter --data
/Users/youruser/Library/Jupyter

Copy kernel.json to kernels/eclair/ in the directory you got above.

7. Start Jupyter:

jupyter notebook --no-browser

8. Test EclairJS Node

To make sure everything is working, create a simple EclairJS Node example.

Create a file called package.json:

{
  "name": "eclairjs-test",
  "version": "0.1.0",
  "dependencies": {
    "eclairjs": "*"
  }
}

And a file called test.js:

var spark = require('eclairjs');

var sc = new spark.SparkContext("local[*]", "Simple Text");

var data = sc.parallelize([1,2,3,4,5,6,7,8,9,0]);

data.collect().then(function(val) {
  console.log("Success:", val);

  sc.stop().then(process.exit);
}).catch(function(err) {
  console.log("Error:", err);
  sc.stop().then(process.exit);
});

Install the dependencies:

$ npm install

Now we are ready to actually run the example:

$ node --harmony test.js
Starting WebSocket: ws://127.0.0.1:8888/api/kernels/436e67e6-2605-4085-9c5d-ba43d828a038
got kernel
Success: [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 0 ]

To run test suite:

$ npm run integration-test