Mosaic version that support Databricks DBR13.x

[LynLi-TA]

Hi,

According to https://databrickslabs.github.io/mosaic/usage/installation.html, quote, “Mosaic 0.3 series does not support DBR 13 (coming soon with Mosaic 0.4 series)”, but our team has to upgrade to DBR 13+ to utilize compute libraries on Unity Catalog Compute with Shared Access mode which require 13.1+.

Any ideas on when the next series of Mosaic that supports DBR13 will be released?

Bests,
L.

[mjohns-databricks]

@LynLi-TA we have been prioritizing a raster focused release coming very soon with Mosaic 0.3.12. Then we will focus on Spark 3.4 / DBR 13.3 LTS compatible release with Mosaic 0.3.13 which we want to deploy by end of OCT. The docs were updated from the "Mosaic 0.4 series" information which has changed.

[landlord-matt]

I discovered that this prevents us to use shared clusters and Unity Catalog. In order to use Unity Catalog, you have to have access mode shared. In order to use installed libraries such as Mosaic in the shared mode, you need Databricks Runtime 13.0. At least we have single user mode, but it would be nice if this was not delayed.

[landlord-matt]

@mjohns-databricks : Any update on this? Today it is 30/10 and there doesn't seem to be any release of either new raster functions or 13.0 compatibility.

Update: They did release version 0.3.12 on 31/10 with new raster functions, but no Databricks 13 support.

[camlee]

@mjohns-databricks: I am also interest in this since we use Unity Catalog and want to use mosaic for some geospatial transformations. Now that some time has passed and the raster release is out, how have things progressed for DBR 13? Understanding it can be hard to estimate, is a release with DBR 13 compatibility imminent, close, or further out? Ex. days, weeks, or months? Would help me with my planning. Cheers!

[rendorHaevyn]

@mjohns-databricks: I am also interest in this since we use Unity Catalog and want to use mosaic for some geospatial transformations. Now that some time has passed and the raster release is out, how have things progressed for DBR 13? Understanding it can be hard to estimate, is a release with DBR 13 compatibility imminent, close, or further out? Ex. days, weeks, or months? Would help me with my planning. Cheers!

As of last week

[mjohns-databricks]

We are finalizing the release that support DBR 13 series right now. It is planned as Mosaic 0.4.0. Goal is to have it out in the next 1-2 weeks.

[landlord-matt]

I don't want to complain, BUT it has now been two weeks :(

[mjohns-databricks]

It is getting close, we are finishing 0.3.13 release bundle with some needed raster updates. Then you will see several PRs waiting to be merged that go with 0.4.0 (DBR 13).

[LynLi-TA]

It is getting close, we are finishing 0.3.13 release bundle with some needed raster updates. Then you will see several PRs waiting to be merged that go with 0.4.0 (DBR 13).

Hi, it's been 2weeks since the last update. I am wondering how this's going.

[caiofgf]

Is there any new prediction for the release that supports DBR 13 series?

[jonasmw94]

Any updates? And maybe a bit unrelated, but will you update with support for DBR 14 as well? We are unable to use this library unless we can have Unity Catalog and UDFs (Databricks on GCP).

[mjohns-databricks]

We are "finally" releasing Mosaic 0.4.0 today with initial support for DBR 13. Would note that Mosaic 0.3.x series with DBR 12.2 LTS supports Unity Catalog catalog and schemas, so that should not have been a blocker. For this latest series, refer to this section <https://github.com/databrickslabs/mosaic/tree/main?tab=readme-ov-file#mosaic-04x-series-latest> of the README for Unity Catalog capabilities (and limitations as it relates to a JVM-first framework such as Mosaic). DBR 14 will be a further transition of Mosaic to support Unity Catalog and other targeted product features. It will be a different release series than 0.4.x. Thank you for your patience during this transitional period.

…

On Fri, Jan 12, 2024 at 7:24 AM Jonas Mossin Wagle ***@***.***> wrote: Any updates? And maybe a bit unrelated, but will you update with support for DBR 14 as well? We are unable to use this library unless we can have Unity Catalog and UDFs (Databricks on GCP). — Reply to this email directly, view it on GitHub <#437 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG5ECU3LAJAILZW5PZNDPHTYOETQZAVCNFSM6AAAAAA5TTDBX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBZGA2DMMZXGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Michael Johns* * Geospatial Specialist Leader* ***@***.*** https://www.linkedin.com/in/michaeljohns2/ <https://databricks.com/>

[landlord-matt]

That's great news!

Would note that Mosaic 0.3.x series with DBR 12.2 LTS supports Unity Catalog catalog and schemas, so that should not have been a blocker.

They could have updated how things work, but when I checked a couple of months ago for Databricks Azure, it was possible to run Mosaic 0.3.x with Unity Catalog in Access Mode = Single User, but not in Access Mode = Shared. In our case, they prefer us to use the shared cluster.

[camlee]

On the topic of DBR 14, it looks like Delta Live Tables (DLT) recently updated their preview channel to be based on DBR 14 instead of 13 as it was when this issue was opened. Stable there is still DBR 12. https://docs.databricks.com/en/release-notes/delta-live-tables/2023/50/index.html#databricks-runtime-versions-used-by-this-release

My team was using Mosaic on DLT with hive metastore and now want to continue using it but with Unity Catalog. Am I mistaken or would Mosaic need to support whatever DBR version DLT is using for that to be doable?