Dependency conflict between guava versions of spark and google storage api

[lanastazia]

Hello,

I’m deploying a Spark application, that has a dependency on the google storage java client, to list files in the bucket, and treat them one after another with Spark.

It seems that adding this dependency along with Spark causes a conflict with guava, and all my attempts to exclude certain guava (like here https://stackoverflow.com/questions/44935550/gae-cloud-storage-guava-nosuchmethoderror-on-appspot), try different guava versions and spark versions don’t succeed.

The error is only triggered in “assembly” mode (required to run the spark application on Dataproc), not when the application is run in local.

Do you know if there is a way to make these two libraries compatible?

Here is an example to reproduce this error https://github.com/lanastazia/spark-guava-issue

Here is the whole trace of this error :

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
	at com.google.api.gax.retrying.BasicRetryingFuture.<init>(BasicRetryingFuture.java:82)
	at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:88)
	at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:74)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:75)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.StorageImpl.listBlobs(StorageImpl.java:334)
	at com.google.cloud.storage.StorageImpl.list(StorageImpl.java:290)
	at Main$.main(Main.scala:20)
	at Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Thanks !

[ajaaym]

@lanastazia did you try excluding the older version of Guava?

[lanastazia]

@ajaaym yes, I tried to exclude the older version of Guava evicted by the latest one, but it doesn't work.

[ajaaym]

@lanastazia please add spark.executor.userClassPathFirst=true and spark.driver.userClassPathFirst=true in properties as below.

[lanastazia]

It works, thanks a lot !

[ajaaym]

@elharo @saturnism FYI, another diamond dependency related to guava.

[ejona86]

@ajaaym, I don't think this very related to Guava. I think I root-caused this in grpc/grpc-java#5592 (comment) . I think that Guava is just what failed first. I believe Protobuf also would have failed.

This is not the standard diamond dependency issue, either. It relates to class loaders, not package managers.