Skip to content

Commit

Permalink
[SPARK-19409][BUILD][TEST-MAVEN] Fix ParquetAvroCompatibilitySuite fa…
Browse files Browse the repository at this point in the history
…ilure due to test dependency on avro

## What changes were proposed in this pull request?

After using Apache Parquet 1.8.2, `ParquetAvroCompatibilitySuite` fails on **Maven** test. It is because `org.apache.parquet.avro.AvroParquetWriter` in the test code used new `avro 1.8.0` specific class, `LogicalType`. This PR aims to fix the test dependency of `sql/core` module to use avro 1.8.0.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/2530/consoleFull

```
ParquetAvroCompatibilitySuite:
*** RUN ABORTED ***
  java.lang.NoClassDefFoundError: org/apache/avro/LogicalType
  at org.apache.parquet.avro.AvroParquetWriter.writeSupport(AvroParquetWriter.java:144)
```

## How was this patch tested?

Pass the existing test with **Maven**.

```
$ build/mvn -Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive -Phive-thriftserver test
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:07 h
[INFO] Finished at: 2017-02-04T05:41:43+00:00
[INFO] Final Memory: 77M/987M
[INFO] ------------------------------------------------------------------------
```

Author: Dongjoon Hyun <[email protected]>

Closes #16795 from dongjoon-hyun/SPARK-19409-2.
  • Loading branch information
dongjoon-hyun authored and srowen committed Feb 8, 2017
1 parent e8d3fca commit 0077bfc
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 0 deletions.
7 changes: 7 additions & 0 deletions resource-managers/mesos/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,13 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-tags_${scala.binary.version}</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.mesos</groupId>
<artifactId>mesos</artifactId>
Expand Down
13 changes: 13 additions & 0 deletions sql/core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,19 @@
<artifactId>parquet-avro</artifactId>
<scope>test</scope>
</dependency>
<!--
This version of avro test-dep is different from the one defined
in the parent pom. The parent pom has avro 1.7.7 test-dep for Hadoop.
Here, ParquetAvroCompatibilitySuite uses parquet-avro's AvroParquetWriter
which uses avro 1.8.0+ specific API. In Maven 3, we need to have
this here to have different versions for the same artifact.
-->
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
Expand Down

0 comments on commit 0077bfc

Please sign in to comment.