Azure CLI authentication support

[letmaik]

The driver supports authentication via:

• AAD managed identity
• AAD app (incl. variants of Key Vault and certificates)
• AAD access token
• Manual device code (printed to console)
• Custom token provider callback

I would like to authenticate via Azure CLI credentials in the same way you would via AzureCliCredential() through the Azure SDK for Python: https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.azureclicredential?view=azure-python.

[letmaik]

As a temporary solution, I'm relying on "accessToken" and acquire the token manually:

from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()

df.write.format("com.microsoft.kusto.spark.datasource").option(
    "accessToken",
    credential.get_token(f"https://{cluster}.kusto.windows.net").token
)
# other options...

[ag-ramachandran]

Hi @letmaik , will consider this for a subsequent release

There were 2 considerations why AZ CLI was not considered
a) If we have dev servers, the AZ environments are shared. There are challenges with respect to security with AZ logins on shared nodes
b) Each worker needs the auth to happen, in a multinode set up this is challenging

The easiest is to get an auth token (MSAL / Azure identity API) or performing a REST call for the Auth, getting the JWT token and using it in the auth token. This is how it is preferred in notebooks as well.

[letmaik]

@ag-ramachandran You're right that Azure CLI auth doesn't fit into the common scenarios where Spark is used. In my case, I'm running a pyspark script on a single node and merely using it for the ability of not running out of memory while staying within the limits on that node in terms of disk space, etc. Having said that, CLI auth may be useful for local dev testing or CI testing that doesn't run on a full cluster.