How to make BIDMat as the GPU Backend for Spark?

[databig]

@jcanny I wanna leverage the GPU Resources in Spark.
For example use GPU to do some Matrix Computation.
I am thinking about how to configure to ake BIDMat as the GPU Backend for Spark?
Likely, I use Maven, How to add sth in POM.XML (Attached)

4.0.0 org.apache apache 14 org.apache.spark spark-parent 1.1.0 pom Spark Project Parent POM http://spark.apache.org/ Apache 2.0 License http://www.apache.org/licenses/LICENSE-2.0.html repo scm:git:[email protected]:apache/spark.git scm:git:https://git-wip-us.apache.org/repos/asf/spark.git scm:git:[email protected]:apache/spark.git v1.1.0-rc4 matei Matei Zaharia [email protected] http://www.cs.berkeley.edu/~matei Apache Software Foundation http://spark.apache.org JIRA https://issues.apache.org/jira/browse/SPARK

<prerequisites>
    <maven>3.0.4</maven>
</prerequisites>

<mailingLists>
    <mailingList>
        <name>Dev Mailing List</name>
        <post>[email protected]</post>
        <subscribe>[email protected]</subscribe>
        <unsubscribe>[email protected]</unsubscribe>
    </mailingList>

    <mailingList>
        <name>User Mailing List</name>
        <post>[email protected]</post>
        <subscribe>[email protected]</subscribe>
        <unsubscribe>[email protected]</unsubscribe>
    </mailingList>

    <mailingList>
        <name>Commits Mailing List</name>
        <post>[email protected]</post>
        <subscribe>[email protected]</subscribe>
        <unsubscribe>[email protected]</unsubscribe>
    </mailingList>
</mailingLists>

<modules>
    <module>core</module>
    <module>bagel</module>
    <module>graphx</module>
    <module>mllib</module>
    <module>tools</module>
    <module>streaming</module>
    <module>sql/catalyst</module>
    <module>sql/core</module>
    <module>sql/hive</module>
    <module>repl</module>
    <module>assembly</module>
    <module>external/twitter</module>
    <module>external/kafka</module>
    <module>external/flume</module>
    <module>external/flume-sink</module>
    <module>external/zeromq</module>
    <module>external/mqtt</module>
    <module>examples</module>
</modules>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

    <java.version>1.6</java.version>
    <sbt.project.name>spark</sbt.project.name>
    <scala.version>2.10.4</scala.version>
    <scala.binary.version>2.10</scala.binary.version>
    <scala.macros.version>2.0.1</scala.macros.version>
    <mesos.version>0.18.1</mesos.version>
    <mesos.classifier>shaded-protobuf</mesos.classifier>
    <akka.group>org.spark-project.akka</akka.group>
    <akka.version>2.2.3-shaded-protobuf</akka.version>
    <slf4j.version>1.7.5</slf4j.version>
    <log4j.version>1.2.17</log4j.version>
    <hadoop.version>1.0.4</hadoop.version>
    <protobuf.version>2.4.1</protobuf.version>
    <yarn.version>${hadoop.version}</yarn.version>
    <hbase.version>0.94.6</hbase.version>
    <flume.version>1.4.0</flume.version>
    <zookeeper.version>3.4.5</zookeeper.version>
    <hive.version>0.12.0</hive.version>
    <parquet.version>1.4.3</parquet.version>
    <jblas.version>1.2.3</jblas.version>
    <jetty.version>8.1.14.v20131031</jetty.version>
    <chill.version>0.3.6</chill.version>
    <codahale.metrics.version>3.0.0</codahale.metrics.version>
    <avro.version>1.7.6</avro.version>
    <jets3t.version>0.7.1</jets3t.version>
    <aws.java.sdk.version>1.8.3</aws.java.sdk.version>
    <aws.kinesis.client.version>1.1.0</aws.kinesis.client.version>

    <PermGen>64m</PermGen>
    <MaxPermGen>512m</MaxPermGen>
</properties>

<repositories>
    <repository>
        <id>central</id>
        <!-- This should be at top, it makes maven try the central repo first and then others and hence faster dep resolution -->
        <name>Maven Repository</name>
        <url>https://repo1.maven.org/maven2</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>apache-repo</id>
        <name>Apache Repository</name>
        <url>https://repository.apache.org/content/repositories/releases</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>jboss-repo</id>
        <name>JBoss Repository</name>
        <url>https://repository.jboss.org/nexus/content/repositories/releases</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>mqtt-repo</id>
        <name>MQTT Repository</name>
        <url>https://repo.eclipse.org/content/repositories/paho-releases</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>cloudera-repo</id>
        <name>Cloudera Repository</name>
        <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>mapr-repo</id>
        <name>MapR Repository</name>
        <url>http://repository.mapr.com/maven</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>spring-releases</id>
        <name>Spring Release Repository</name>
        <url>https://repo.spring.io/libs-release</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
</repositories>
<pluginRepositories>
    <pluginRepository>
        <id>central</id>
        <url>https://repo1.maven.org/maven2</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </pluginRepository>
</pluginRepositories>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-util</artifactId>
            <version>${jetty.version}</version>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-security</artifactId>
            <version>${jetty.version}</version>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-plus</artifactId>
            <version>${jetty.version}</version>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-server</artifactId>
            <version>${jetty.version}</version>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>14.0.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.3.2</version>
        </dependency>
        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-math3</artifactId>
            <version>3.3</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.google.code.findbugs</groupId>
            <artifactId>jsr305</artifactId>
            <version>1.3.9</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jul-to-slf4j</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jcl-over-slf4j</artifactId>
            <version>${slf4j.version}</version>
            <!-- <scope>runtime</scope> --> <!-- more correct, but scalac 2.10.3 doesn't like it -->
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>${log4j.version}</version>
        </dependency>
        <dependency>
            <groupId>com.ning</groupId>
            <artifactId>compress-lzf</artifactId>
            <version>1.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.xerial.snappy</groupId>
            <artifactId>snappy-java</artifactId>
            <version>1.0.5.3</version>
        </dependency>
        <dependency>
            <groupId>net.jpountz.lz4</groupId>
            <artifactId>lz4</artifactId>
            <version>1.2.0</version>
        </dependency>
        <dependency>
            <groupId>com.clearspring.analytics</groupId>
            <artifactId>stream</artifactId>
            <version>2.7.0</version>
            <exclusions>
                <!-- Only HyperLogLogPlus is used, which doesn't depend on fastutil -->
                <exclusion>
                    <groupId>it.unimi.dsi</groupId>
                    <artifactId>fastutil</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <!-- In theory we need not directly depend on protobuf since Spark does not directly
             use it. However, when building with Hadoop/YARN 2.2 Maven doesn't correctly bump
             the protobuf version up from the one Mesos gives. For now we include this variable
             to explicitly bump the version when building with YARN. It would be nice to figure
             out why Maven can't resolve this correctly (like SBT does). -->
        <dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>${protobuf.version}</version>
        </dependency>
        <dependency>
            <groupId>com.twitter</groupId>
            <artifactId>chill_${scala.binary.version}</artifactId>
            <version>${chill.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.ow2.asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.ow2.asm</groupId>
                    <artifactId>asm-commons</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.twitter</groupId>
            <artifactId>chill-java</artifactId>
            <version>${chill.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.ow2.asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.ow2.asm</groupId>
                    <artifactId>asm-commons</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>${akka.group}</groupId>
            <artifactId>akka-actor_${scala.binary.version}</artifactId>
            <version>${akka.version}</version>
        </dependency>
        <dependency>
            <groupId>${akka.group}</groupId>
            <artifactId>akka-remote_${scala.binary.version}</artifactId>
            <version>${akka.version}</version>
        </dependency>
        <dependency>
            <groupId>${akka.group}</groupId>
            <artifactId>akka-slf4j_${scala.binary.version}</artifactId>
            <version>${akka.version}</version>
        </dependency>
        <dependency>
            <groupId>${akka.group}</groupId>
            <artifactId>akka-testkit_${scala.binary.version}</artifactId>
            <version>${akka.version}</version>
        </dependency>
        <dependency>
            <groupId>colt</groupId>
            <artifactId>colt</artifactId>
            <version>1.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.mesos</groupId>
            <artifactId>mesos</artifactId>
            <version>${mesos.version}</version>
            <classifier>${mesos.classifier}</classifier>
            <exclusions>
                <exclusion>
                    <groupId>com.google.protobuf</groupId>
                    <artifactId>protobuf-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>commons-net</groupId>
            <artifactId>commons-net</artifactId>
            <version>2.2</version>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.0.23.Final</version>
        </dependency>
        <dependency>
            <groupId>org.apache.derby</groupId>
            <artifactId>derby</artifactId>
            <version>10.4.2.0</version>
        </dependency>
        <dependency>
            <groupId>com.codahale.metrics</groupId>
            <artifactId>metrics-core</artifactId>
            <version>${codahale.metrics.version}</version>
        </dependency>
        <dependency>
            <groupId>com.codahale.metrics</groupId>
            <artifactId>metrics-jvm</artifactId>
            <version>${codahale.metrics.version}</version>
        </dependency>
        <dependency>
            <groupId>com.codahale.metrics</groupId>
            <artifactId>metrics-json</artifactId>
            <version>${codahale.metrics.version}</version>
        </dependency>
        <dependency>
            <groupId>com.codahale.metrics</groupId>
            <artifactId>metrics-ganglia</artifactId>
            <version>${codahale.metrics.version}</version>
        </dependency>
        <dependency>
            <groupId>com.codahale.metrics</groupId>
            <artifactId>metrics-graphite</artifactId>
            <version>${codahale.metrics.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-compiler</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-reflect</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>jline</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-actors</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scalap</artifactId>
            <version>${scala.version}</version>
        </dependency>
        <dependency>
            <groupId>org.scalatest</groupId>
            <artifactId>scalatest_${scala.binary.version}</artifactId>
            <version>2.1.5</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.easymock</groupId>
            <artifactId>easymockclassextension</artifactId>
            <version>3.1</version>
            <scope>test</scope>
        </dependency>
        <!-- Needed by cglib which is needed by easymock. -->
        <dependency>
            <groupId>asm</groupId>
            <artifactId>asm</artifactId>
            <version>3.3.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-all</artifactId>
            <version>1.9.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.scalacheck</groupId>
            <artifactId>scalacheck_${scala.binary.version}</artifactId>
            <version>1.11.3</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.10</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.novocode</groupId>
            <artifactId>junit-interface</artifactId>
            <version>0.10</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-recipes</artifactId>
            <version>2.4.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.jboss.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.ow2.asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.jboss.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>commons-logging</groupId>
                    <artifactId>commons-logging</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>servlet-api-2.5</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>junit</groupId>
                    <artifactId>junit</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>${avro.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-ipc</artifactId>
            <version>${avro.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jetty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jetty-util</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.velocity</groupId>
                    <artifactId>velocity</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-mapred</artifactId>
            <version>${avro.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jetty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jetty-util</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.velocity</groupId>
                    <artifactId>velocity</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <!-- See SPARK-1556 for info on this dependency: -->
        <dependency>
            <groupId>net.java.dev.jets3t</groupId>
            <artifactId>jets3t</artifactId>
            <version>${jets3t.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>commons-logging</groupId>
                    <artifactId>commons-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-api</artifactId>
            <version>${yarn.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.ow2.asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.jboss.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>commons-logging</groupId>
                    <artifactId>commons-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-common</artifactId>
            <version>${yarn.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.ow2.asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.jboss.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>commons-logging</groupId>
                    <artifactId>commons-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-server-web-proxy</artifactId>
            <version>${yarn.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.ow2.asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.jboss.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>commons-logging</groupId>
                    <artifactId>commons-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-client</artifactId>
            <version>${yarn.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.ow2.asm</groupId>
                    <artifactId>asm</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.jboss.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>commons-logging</groupId>
                    <artifactId>commons-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <!-- Matches the version of jackson-core-asl pulled in by avro -->
            <groupId>org.codehaus.jackson</groupId>
            <artifactId>jackson-mapper-asl</artifactId>
            <version>1.8.8</version>
        </dependency>
    </dependencies>
</dependencyManagement>

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-enforcer-plugin</artifactId>
                <version>1.3.1</version>
                <executions>
                    <execution>
                        <id>enforce-versions</id>
                        <goals>
                            <goal>enforce</goal>
                        </goals>
                        <configuration>
                            <rules>
                                <requireMavenVersion>
                                    <version>3.0.4</version>
                                </requireMavenVersion>
                                <requireJavaVersion>
                                    <version>${java.version}</version>
                                </requireJavaVersion>
                            </rules>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>build-helper-maven-plugin</artifactId>
                <version>1.8</version>
            </plugin>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.0</version>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile-first</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>attach-scaladocs</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>doc-jar</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>${scala.version}</scalaVersion>
                    <recompileMode>incremental</recompileMode>
                    <useZincServer>true</useZincServer>
                    <args>
                        <arg>-unchecked</arg>
                        <arg>-deprecation</arg>
                        <arg>-feature</arg>
                        <arg>-language:postfixOps</arg>
                    </args>
                    <jvmArgs>
                        <jvmArg>-Xms1024m</jvmArg>
                        <jvmArg>-Xmx1024m</jvmArg>
                        <jvmArg>-XX:PermSize=${PermGen}</jvmArg>
                        <jvmArg>-XX:MaxPermSize=${MaxPermGen}</jvmArg>
                    </jvmArgs>
                    <javacArgs>
                        <javacArg>-source</javacArg>
                        <javacArg>${java.version}</javacArg>
                        <javacArg>-target</javacArg>
                        <javacArg>${java.version}</javacArg>
                    </javacArgs>
                    <!-- The following plugin is required to use quasiquotes in Scala 2.10 and is used
                         by Spark SQL for code generation. -->
                    <compilerPlugins>
                        <compilerPlugin>
                            <groupId>org.scalamacros</groupId>
                            <artifactId>paradise_${scala.version}</artifactId>
                            <version>${scala.macros.version}</version>
                        </compilerPlugin>
                    </compilerPlugins>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                    <encoding>UTF-8</encoding>
                    <maxmem>1024m</maxmem>
                    <fork>true</fork>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.17</version>
                <configuration>
                    <!-- Uses scalatest instead -->
                    <skipTests>true</skipTests>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.scalatest</groupId>
                <artifactId>scalatest-maven-plugin</artifactId>
                <version>1.0-RC2</version>
                <configuration>
                    <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
                    <junitxml>.</junitxml>
                    <filereports>${project.build.directory}/SparkTestSuite.txt</filereports>
                    <argLine>-Xmx3g -XX:MaxPermSize=${MaxPermGen} -XX:ReservedCodeCacheSize=512m</argLine>
                    <stderr />
                    <systemProperties>
                        <java.awt.headless>true</java.awt.headless>
                        <spark.test.home>${session.executionRootDirectory}</spark.test.home>
                        <spark.testing>1</spark.testing>
                    </systemProperties>
                </configuration>
                <executions>
                    <execution>
                        <id>test</id>
                        <goals>
                            <goal>test</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>2.4</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <version>1.7</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.2</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <version>2.2.1</version>
                <configuration>
                    <attach>true</attach>
                </configuration>
                <executions>
                    <execution>
                        <id>create-source-jar</id>
                        <goals>
                            <goal>jar-no-fork</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-clean-plugin</artifactId>
                <version>2.5</version>
                <configuration>
                    <filesets>
                        <fileset>
                            <directory>work</directory>
                        </fileset>
                        <fileset>
                            <directory>checkpoint</directory>
                        </fileset>
                    </filesets>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>

    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-enforcer-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <executions>
                <execution>
                    <id>add-scala-sources</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>add-source</goal>
                    </goals>
                    <configuration>
                        <sources>
                            <source>src/main/scala</source>
                        </sources>
                    </configuration>
                </execution>
                <execution>
                    <id>add-scala-test-sources</id>
                    <phase>generate-test-sources</phase>
                    <goals>
                        <goal>add-test-source</goal>
                    </goals>
                    <configuration>
                        <sources>
                            <source>src/test/scala</source>
                        </sources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-source-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.scalastyle</groupId>
            <artifactId>scalastyle-maven-plugin</artifactId>
            <version>0.4.0</version>
            <configuration>
                <verbose>false</verbose>
                <failOnViolation>true</failOnViolation>
                <includeTestSourceDirectory>false</includeTestSourceDirectory>
                <failOnWarning>false</failOnWarning>
                <sourceDirectory>${basedir}/src/main/scala</sourceDirectory>
                <testSourceDirectory>${basedir}/src/test/scala</testSourceDirectory>
                <configLocation>scalastyle-config.xml</configLocation>
                <outputFile>scalastyle-output.xml</outputFile>
                <outputEncoding>UTF-8</outputEncoding>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>check</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

<profiles>

    <!-- Ganglia integration is not included by default due to LGPL-licensed code -->
    <profile>
        <id>spark-ganglia-lgpl</id>
        <modules>
            <module>extras/spark-ganglia-lgpl</module>
        </modules>
    </profile>

    <!-- Kinesis integration is not included by default due to ASL-licensed code -->
    <profile>
        <id>kinesis-asl</id>
        <modules>
            <module>extras/kinesis-asl</module>
        </modules>
    </profile>

    <profile>
        <id>java8-tests</id>
        <build>
            <plugins>
                <!-- Needed for publishing test jars as it is needed by java8-tests -->
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-jar-plugin</artifactId>
                    <executions>
                        <execution>
                            <goals>
                                <goal>test-jar</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>

        <modules>
            <module>extras/java8-tests</module>
        </modules>

    </profile>

    <!-- A series of build profiles where customizations for particular Hadoop releases can be made -->

    <profile>
        <id>hadoop-0.23</id>
        <!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
        <dependencies>
            <dependency>
                <groupId>org.apache.avro</groupId>
                <artifactId>avro</artifactId>
            </dependency>
        </dependencies>
        <properties>
            <hadoop.version>0.23.10</hadoop.version>
        </properties>
    </profile>

    <profile>
        <id>hadoop-2.2</id>
        <properties>
            <hadoop.version>2.2.0</hadoop.version>
            <protobuf.version>2.5.0</protobuf.version>
        </properties>
    </profile>

    <profile>
        <id>hadoop-2.3</id>
        <properties>
            <hadoop.version>2.3.0</hadoop.version>
            <protobuf.version>2.5.0</protobuf.version>
            <jets3t.version>0.9.0</jets3t.version>
        </properties>
    </profile>

    <profile>
        <id>hadoop-2.4</id>
        <properties>
            <hadoop.version>2.4.0</hadoop.version>
            <protobuf.version>2.5.0</protobuf.version>
            <jets3t.version>0.9.0</jets3t.version>
        </properties>
    </profile>

    <profile>
        <id>yarn-alpha</id>
        <modules>
            <module>yarn</module>
        </modules>
    </profile>

    <profile>
        <id>yarn</id>
        <modules>
            <module>yarn</module>
        </modules>
    </profile>

    <profile>
        <id>mapr3</id>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
        <properties>
            <hadoop.version>1.0.3-mapr-3.0.3</hadoop.version>
            <yarn.version>2.3.0-mapr-4.0.0-FCS</yarn.version>
            <hbase.version>0.94.17-mapr-1405</hbase.version>
            <zookeeper.version>3.4.5-mapr-1406</zookeeper.version>
        </properties>
    </profile>

    <profile>
        <id>mapr4</id>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
        <properties>
            <hadoop.version>2.3.0-mapr-4.0.0-FCS</hadoop.version>
            <yarn.version>2.3.0-mapr-4.0.0-FCS</yarn.version>
            <hbase.version>0.94.17-mapr-1405-4.0.0-FCS</hbase.version>
            <zookeeper.version>3.4.5-mapr-1406</zookeeper.version>
        </properties>
        <dependencies>
            <dependency>
                <groupId>org.apache.curator</groupId>
                <artifactId>curator-recipes</artifactId>
                <version>2.4.0</version>
                <exclusions>
                    <exclusion>
                        <groupId>org.apache.zookeeper</groupId>
                        <artifactId>zookeeper</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
            <dependency>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
                <version>3.4.5-mapr-1406</version>
            </dependency>
        </dependencies>
    </profile>

    <!-- Build without Hadoop dependencies that are included in some runtime environments. -->
    <profile>
        <id>hadoop-provided</id>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
        <dependencies>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-api</artifactId>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-common</artifactId>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-server-web-proxy</artifactId>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-yarn-client</artifactId>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.avro</groupId>
                <artifactId>avro</artifactId>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.avro</groupId>
                <artifactId>avro-ipc</artifactId>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
                <version>${zookeeper.version}</version>
                <scope>provided</scope>
            </dependency>
        </dependencies>
    </profile>

    <profile>
        <id>hive</id>
        <activation>
            <activeByDefault>false</activeByDefault>
        </activation>
        <modules>
            <module>sql/hive-thriftserver</module>
        </modules>
    </profile>

</profiles>

[jcanny]

The challenge is to include all the jars with native libs that BIDMat uses. These are currently:
JCUDA
LZ4
HDF5-Java
BIDMat's own native libs.

All of the native libs except HDF5 can be bundled into a Jar, and will automatically be unpacked when the calling class is loaded. The JVM will need access to a tmp directory to do the unpacking.

HD5 doesnt support this (or didnt as of the version we use). I dont remember if it can run at all without the native libs. I dont think so. It may not be too relevant if running on Spark where I/O is managed by Spark. Does Spark have LZ4 support already?

It might be better to make a single assembly jar with all of the dependent jars and native libs are included. Otherwise someone would have to commit to maintaining repo copies of the dependencies.

[databig]

@jcanny
I will try, Please make sure Am I right or not?
1, Claim the Version Number
<bidmat.version>1.0.0</bidmat.version>
<jcuda.version>6.5</jcuda.version>
2, Add into Dependencies

bidmat?

${bidmat.version}

【
jcuda?
jcuda?
${/jcuda.version}


HDF5-Java?
HDF5-Java?
???


LZ4?
LZ4?
???

【？】 means I am not sure, could you please kindly confirm it?

Tks.

[jcanny]

We should talk on the phone so I"m clear on what you're trying to do
(build with maven?). Most of the libs are not in repos, so we'd have to
figure something else out.

-John
On 3/18/2015 2:24 PM, databig wrote:

@jcanny https://github.com/jcanny
I will try, Please make sure Am I right or not?
1, Claim the Version Number
1.0.0
6.5
2, Add into Dependencies

bidmat?

${bidmat.version}

jcuda?
jcuda?
${/jcuda.version}

HDF5-Java?
HDF5-Java?
???

LZ4?
LZ4?
???

【？】 means I am not sure, could you please kindly confirm it?

Tks.

—
Reply to this email directly or view it on GitHub
#22 (comment).

[jcanny]

I have a team working on this now, so it should be resolved over the next few months.

[RoiViber]

@jcanny Hi. Any resolution on that? do you have open source code for coupling BidMat with Spark? thanks.