Cannot compile Intel-mllib of master branch.

[haojinIntel]

We try to compile Intel-mllib of master branch and meet the error like:

The maven command is "mvn clean package -DskipTests -Pspark-3.1.1" and we've successfully installed oneapi and oneccl.

[haojinIntel]

@xwu99 @zhixingheyi-tian Please help to track the issue. Thanks!

[haojinIntel]

@xwu99 Do you have any instructions about this issue?

[xwu99]

Is it related to #77? Are you able to to build before? Are you able to build for 3.0.x profiles?
Pls use build.sh script instead mvn alone and it will setup proper environments.

[zhixingheyi-tian]

@haojinIntel

Would you paste the specific UT

[haojinIntel]

I've paste the whole message below:

[root@bdpe-sky2 mllib-dal]# mvn clean package -Pspark-3.1.1
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building OAP Project OAP MLlib 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ oap-mllib ---
[INFO] Deleting /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/target
[INFO]
[INFO] --- build-helper-maven-plugin:3.2.0:add-source (add-source) @ oap-mllib ---
[INFO] Source directory: /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/src/spark-3.1.1/main/java added.
[INFO] Source directory: /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/src/spark-3.1.1/main/scala added.
[INFO]
[INFO] --- build-helper-maven-plugin:3.2.0:add-test-source (add-test-source) @ oap-mllib ---
[INFO] Test Source directory: /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/src/spark-3.1.1/test/scala added.
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ oap-mllib ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 1 resource
[INFO] Copying 2 resources
[INFO] Copying 0 resource
[INFO]
[INFO] --- scala-maven-plugin:4.4.0:add-source (scala-compile-first) @ oap-mllib ---
[INFO] Add Source directory: /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/src/main/scala
[INFO] Add Test Source directory: /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/src/test/scala
[INFO]
[INFO] --- scala-maven-plugin:4.4.0:compile (scala-compile-first) @ oap-mllib ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /root/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.5-bin_2.12.10__52.0-1.3.5_20200330T000931.jar
[INFO] Compiling 14 Scala sources and 9 Java sources to /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/target/classes ...
[INFO] Done compiling.
[INFO] compile in 12.4 s
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ oap-mllib ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 9 source files to /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/target/classes
[INFO]
[INFO] --- maven-antrun-plugin:1.8:run (default) @ oap-mllib ---
[INFO] Executing tasks

build-natives:
[echo] Building native code
[exec] Cleaning up
[exec] rm -rf ./OneCCL.o ./OneDAL.o ./service.o ./error_handling.o ./KMeansDALImpl.o ./PCADALImpl.o ./ALSDALImpl.o ./ALSShuffle.o ./NaiveBayesDALImpl.o ./LinearRegressionDALImpl.o ../../../target/libMLlibDAL.so
[exec]
[exec] Building file: OneCCL.cpp
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "OneCCL.o" "OneCCL.cpp"
[exec] Building file: OneDAL.cpp
[exec] Building file: service.cpp
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "OneDAL.o" "OneDAL.cpp"
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "service.o" "service.cpp"
[exec] Building file: error_handling.cpp
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "error_handling.o" "error_handling.cpp"
[exec] Building file: KMeansDALImpl.cpp
[exec] Building file: PCADALImpl.cpp
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "KMeansDALImpl.o" "KMeansDALImpl.cpp"
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "PCADALImpl.o" "PCADALImpl.cpp"
[exec] Building file: ALSDALImpl.cpp
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "ALSDALImpl.o" "ALSDALImpl.cpp"
[exec] Building file: ALSShuffle.cpp
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "ALSShuffle.o" "ALSShuffle.cpp"
[exec] Building file: NaiveBayesDALImpl.cpp
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "NaiveBayesDALImpl.o" "NaiveBayesDALImpl.cpp"
[exec] Building file: LinearRegressionDALImpl.cpp
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -I /opt/Beaver/jdk/include -I /opt/Beaver/jdk/include/linux -I /tmp/oneCCL/build/_install/include -I /opt/intel/oneapi/dal/2021.2.0/include -I ./javah -I ./ -c -o "LinearRegressionDALImpl.o" "LinearRegressionDALImpl.cpp"
[exec] Finished building: error_handling.cpp
[exec]
[exec] Finished building: ALSShuffle.cpp
[exec]
[exec] Finished building: OneCCL.cpp
[exec]
[exec] Finished building: OneDAL.cpp
[exec]
[exec] Finished building: service.cpp
[exec]
[exec] Finished building: PCADALImpl.cpp
[exec]
[exec] Finished building: NaiveBayesDALImpl.cpp
[exec]
[exec] Finished building: KMeansDALImpl.cpp
[exec]
[exec] Finished building: LinearRegressionDALImpl.cpp
[exec]
[exec] Finished building: ALSDALImpl.cpp
[exec]
[exec] Building target: ../../../target/libMLlibDAL.so
[exec] Invoking: Linker
[exec] clang++ -g -Wall -Wno-deprecated-declarations -fPIC -std=c++11 -shared -o ../../../target/libMLlibDAL.so ./OneCCL.o ./OneDAL.o ./service.o ./error_handling.o ./KMeansDALImpl.o ./PCADALImpl.o ./ALSDALImpl.o ./ALSShuffle.o ./NaiveBayesDALImpl.o ./LinearRegressionDALImpl.o -L/tmp/oneCCL/build/_install/lib -lccl -L/opt/intel/oneapi/dal/2021.2.0/lib/intel64 -l:libdaal_core.a -l:libdaal_thread.a -L/opt/intel/oneapi/tbb/2021.2.0/env/../lib/intel64/gcc4.8 -ltbb -ltbbmalloc
[exec] Finished building target: ../../../target/libMLlibDAL.so
[exec]
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ oap-mllib ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 1 resource
[INFO] Copying 2 resources
[INFO] Copying 1 resource
[INFO]
[INFO] --- scala-maven-plugin:4.4.0:testCompile (scala-test-compile) @ oap-mllib ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /root/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.5-bin_2.12.10__52.0-1.3.5_20200330T000931.jar
[INFO] Compiling 5 Scala sources to /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/target/test-classes ...
[ERROR] /home/oap/Beaver/package/source_code/oap-tools/oap-mllib/mllib-dal/src/test/scala/org/apache/spark/ml/recommendation/MLlibALSSuite.scala:1005: value cleanShuffleDependencies is not a member of object org.apache.spark.ml.recommendation.ALS
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30.261 s
[INFO] Finished at: 2021-07-02T14:38:25+08:00
[INFO] Final Memory: 165M/3951M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.4.0:testCompile (scala-test-compile) on project oap-mllib: Execution scala-test-compile of goal net.alchim31.maven:scala-maven-plugin:4.4.0:testCompile failed. CompileFailed -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

[xwu99]

It shoud be fixed with #88