[Databricks] Security exception is thrown on a high concurrency cluster with auth passthrough.

[markbaas]

In Databricks, I'm trying to connect with my cosmos db using a high concurrency cluster with aad passthrough.

After settings the spark settings:

%python
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountEndpoint", "https://blabla.documents.azure.com:443/")
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountKey", dbutils.secrets.get("main", "cosmosMasterKey"))
spark.conf.set("spark.sql.catalog.cosmosCatalog", "com.azure.cosmos.spark.CosmosCatalog")

I'm running the following sql code:

select * from cosmosCatalog.bladb.blacollection limit 10

The following exception is through:

Error in SQL statement: SecurityException: Only default session catalog is supported for Credential Passthrough or Table ACL enabled cluster. Try to load: com.azure.cosmos.spark.CosmosCatalog

[markbaas]

Seems it won't work with regular cluster in combination with passthrough either.

[ralphke]

@joshfree I do get the same error message on an Databricks premium cluster version 9.1 LTS when trying to access a cosmosDB database. Neither read from the dataset nor write works as expected.
This is the code artifact I am using:
cosmosEndpoint = "https://.documents.azure.com:443/"
cosmosMasterKey = ""
cosmosDatabaseName = "southridge"
cosmosContainerName = "movies"
cosmosApplicationName = "Databricks"

spark.conf.set("spark.sql.catalog.cosmosCatalog", "com.azure.cosmos.spark.CosmosCatalog")
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountEndpoint", cosmosEndpoint)
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountKey", cosmosMasterKey)
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.container", cosmosContainerName)
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.applicationName", cosmosApplicationName)

spark.sql("CREATE DATABASE IF NOT EXISTS cosmosCatalog.{};".format(cosmosDatabaseName))
spark.sql("CREATE TABLE IF NOT EXISTS cosmosCatalog.{}.{} using cosmos.oltp TBLPROPERTIES(partitionKeyPath = '/id', manualThroughput = '400')".format(cosmosDatabaseName, cosmosContainerName))

[joshfree]

Closing old issue with no activity since Oct 4, 2021