Update Databricks Azure qual tool to set env variable for ABFS paths

[cindyyuanjiang]

Fixes #981

spark_rapids qualification --eventlogs=abfss://xxxxxxxxxx

Running the above command fails because spark_rapids qualification tool tries to detect platform from ABFS event logs but it cannot verify that without credentials.

Instead of asking the users to specify either AZURE_STORAGE_ACCOUNT_NAME or AZURE_STORAGE_CONNECTION_STRING env variables like:

export AZURE_STORAGE_ACCOUNT_NAME=xxxxxxxx
export AZURE_STORAGE_CONNECTION_STRING=xxxxxxxx
spark_rapids qualification --eventlogs=abfss://xxxxxxxxxx

This PR sets the env vars in the qualification tools so that users don't have to do that.

[amahussein]

Thanks @cindyyuanjiang !
Some few comments to keep the design flow.

[amahussein]

This is very specific to AdlsPath class which extends CspPath. It is better to be inside adlspath.py.
I did not look deeper in the code, but my intuition is that this code block should be inside adlspath.py by overriding is_protocol_prefix

[cindyyuanjiang]

Moved this to adlspath.py, thanks!

[amahussein]

• It is always useful to use type hints to get some checks on type safety and code incorrectness. Python started adopted this typing/atype-annotations more widely to overcome the problem of type-safety that makes python difficult to work with in large projects.

   def get_abfs_account_name(path: str) -> str :

• I don't see util.py is the best place for this because it is a functionality that's specific to ADLS driver.
• Should this method try to read the env-variable. if it is not set, then it extracts it from the URL?
• Also, with defining this new method, I would expect 2 sites calling this method: 1- in pytools (old way we xtracted information for Azure); 2- During the initialization in the new spark_rapids_tools
  aka, DBAzureLocalRapidsJob.get_account_name strips the account-name from the URL; regardless of the initialization we did. It should read

[cindyyuanjiang]

Thanks @amahussein! Updated this.

[amahussein]

We don't need this because this overrides the purpose of the design that each Class will be able to recognize its own URL.

[cindyyuanjiang]

removed this, thanks!

[parthosa]

Thanks @cindyyuanjiang for this change. We are automatically setting AZURE_STORAGE_ACCOUNT_NAME in this PR.

In addition to this, AZURE_STORAGE_CONNECTION_STRING would also be required. Do we plan to handle this as well?

[cindyyuanjiang]

In addition to this, AZURE_STORAGE_CONNECTION_STRING would also be required. Do we plan to handle this as well?

Thanks @parthosa! Setting AZURE_STORAGE_ACCOUNT_NAME is enough to verify the abfss path.

[parthosa]

Thanks @cindyyuanjiang. LGTM

[amahussein]

for more readability:

@classmethod
def is_protocol_prefix(cls, value: str) -> bool:
    valid_prefix = super().is_protocol_prefix(value)
    if valid_prefix:
        # check if the AZURE_STORAGE_ACCOUNT_NAME env_variable is defined.
        # If not, set it to avoid failures when user is not ..bla..bla
        # https://github.com/NVIDIA/spark-rapids-tools/issues/981
        if "AZURE_STORAGE_ACCOUNT_NAME" not in os.environ:
            account_name = cls.get_abfs_account_name(value)
            os.environ["AZURE_STORAGE_ACCOUNT_NAME"] = account_name
     return valid_prefix

[cindyyuanjiang]

Thanks @amahussein! Updated this.

[amahussein]

Thanks @cindyyuanjiang
Great fix!