Issue with Databricks Labs Data Generator with Databricks Runtime 14.1 on shared cluster

[ronanstokes-db]

Hi all

There is an issue when running the Data Generator on Unity Catalog environment in the 14.1 runtime. The DataGenerator will try to determine the appropriate number of partitions to use when they are not specified.

When running with this release on a Unity Catalog enabled shared cluster, you will receive an error if you dont explicitly specify the number of partitions.

The workaround is to explicitly specify the number of partitions. We are working on a solution to this and will update this over the next couple of days.

This does not affect the other access modes for Unity Catalog enabled clusters, or the 13.3 LTS runtime (which we recommend for now)