Cannot compile master branch.

[haojinIntel]

We try to compile the master branch and meet the error message like:

[haojinIntel]

@zhouyuan @weiting-chen @zhixingheyi-tian Please help to track the issue. Thanks！

[weiting-chen]

@haojinIntel Could you also provide the full compile command you are using. Thanks.

[haojinIntel]

@weiting-chen The command of maven is showed below:

And the command of cmake for arrow is like:

[weiting-chen]

@haojinIntel
This is an issue that the code cannot pass this Spark built-in ut and we can produce in our environment as well.
But we need more time to address and fix it, @rui-mo will help to submit a PR to remove this UT and let the compile can run successfully.
One workaround for the current code is to add "-Dmaven.test.skip" to ignore all the scala ut during mvn command.

[zhouyuan]

@weiting-chen @rui-mo i have a local jenkins running daily - this didnt show up in my jenkins env.
Java.Util.Map[K, V] can not be applied to Java.Util.HashMap(String, Array(String))
i suspect this is a compiler issue, the scala-maven-plugin is set to 4.4.0 to bypass some issue on 4.5.1
#325
not sure using 4.5.2 could fix this

[rui-mo]

@zhouyuan hi, did you test with"-Dmaven.test.skip"? with this config, this error will not show up.

[zhouyuan]

@rui-mo
yes, my tests is running w tests compiled.

@haojinIntel can you also provide your JDK info?

this seems occur on openjdk only, oracle-jdk runs fine

[rui-mo]

Here are some new findings:

       <plugin>
         <groupId>net.alchim31.maven</groupId>
         <artifactId>scala-maven-plugin</artifactId>
         <version>4.4.0</version>
         <configuration>
           <recompileMode>${scala.recompile.mode}</recompileMode>
         </configuration>

If deleting " 4.4.0" or replacing "4.4.0" to "4.5.2", below cmd runs successfully.

mvn clean package -am -DskipTests -Dbuild_arrow=OFF -Dbuild_protobuf=OFF -Phadoop-3.2

Below command still failed with the above error:

mvn test -DmembersOnlySuites=org.apache.spark.sql.travis -am -DfailIfNoTests=false -Dexec.skip=true

But If changing the scala-maven-plugin version to 4.3.0 or 4.3.1 (spark is using 4.3.0), the test compilation works well on my env.

[weiting-chen]

Deleting "4.4.0" not work for me. Report the same error with "4.4.0"
It looks like change to "4.5.2" is the only solution in my environment.

[zhouyuan]

Here are some new findings:

       <plugin>
         <groupId>net.alchim31.maven</groupId>
         <artifactId>scala-maven-plugin</artifactId>
         <version>4.4.0</version>
         <configuration>
           <recompileMode>${scala.recompile.mode}</recompileMode>
         </configuration>

If deleting " 4.4.0" or replacing "4.4.0" to "4.5.2", below cmd runs successfully.

mvn clean package -am -DskipTests -Dbuild_arrow=OFF -Dbuild_protobuf=OFF -Phadoop-3.2

But looks like tests are not compiled if using this command.

Below command still failed with the above error:

mvn test -DmembersOnlySuites=org.apache.spark.sql.travis -am -DfailIfNoTests=false -Dexec.skip=true

But If changing the scala-maven-plugin version to 4.3.0 or 4.3.1 (spark is using 4.3.0), the test compilation works well on my env.

@rui-mo
below picture showed tests are skipped, but not mean they are not compiled

[zhouyuan]

@haojinIntel can you please also share the JDK you are using?

[weiting-chen]

4.3.0 and 4.3.1 also work in my env with openjdk version "1.8.0_282"

[haojinIntel]

@zhouyuan I use oracle-jdk 1.8.0_112.

[rui-mo]

Adding "-P full-scala-compiler" may help solve this issue. Below command compiles successfully on my env with master branch.
mvn test -P full-scala-compiler -DmembersOnlySuites=org.apache.spark.sql.travis -am -DfailIfNoTests=false -Dexec.skip=true

[weiting-chen]

@haojinIntel
We have merged a PR to fix this issue in master by using scala-maven-plugin with 4.3.0.
Please help to retry the master code.
Thanks.

[haojinIntel]

@weiting-chen The issue can be fixed in my env. Thanks.