py4j.Py4JException: Constructor com.databricks.libraries.JavaJarId([class java.net.URI, class java.lang.String, class java.lang.String]

[AbishekS01]

https://databrickslabs.github.io/mosaic/#project-support

which says that Mosaic should still work on "Photon enabled" clusters.

We are currently running several Mosaic jobs on Photon-enabled clusters, but after an upgrade to runtime 13.3, we got an error:

py4j.Py4JException: Constructor com.databricks.libraries.JavaJarId([class java.net.URI, class java.lang.String, class java.lang.String])

I will be downgrading the cluster back to 13.2 to see if that fixes it in the short term, but I would like guidance on how to use mosaic GEO features (most importantly st_geomfromgeojson and st_intersects, etc).

[landlord-matt]

We got the same error running mosaic==0.3.5 on runtime version 13.3 on a Personal Compute. This used to work on 12.2 LTS. Upgrading to mosaic==0.3.11 does not make any difference. Not that I think it matters, but we're running Databricks in Azure.

enable_mosaic(spark, dbutils)

MosaicLibraryHandler(config.mosaic_spark)

     71 converters = self.sc._jvm.scala.collection.JavaConverters
     73 JarURI = JavaURI.create("file:" + self._jar_path)
---> 74 lib = JavaJarId(
     75     JarURI,
     76     ManagedLibraryId.defaultOrganization(),
     77     NoVersionModule.simpleString(),
     78 )

py4j.Py4JException: Constructor com.databricks.libraries.JavaJarId([class java.net.URI, class java.lang.String, class java.lang.String]) does not exist
	at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:203)
	at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:220)
	at py4j.Gateway.invoke(Gateway.java:255)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:115)
	at java.lang.Thread.run(Thread.java:750)

[landlord-matt]

I realise now that it says that Databricks Runtime 13 isn't supported until mosaic version 0.4 in the README. That was added on June 13, almost 3 months ago. It seems the project stalled as 13.3 is now the current LTS?

[landlord-matt]

They now answered in this thread that they will fix this by the end of October 2023.