Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Airflow Connection Type Azure Blob Storage - Shared Access Key field #31811

Closed
1 of 2 tasks
manmeetkaur opened this issue Jun 9, 2023 · 3 comments · Fixed by #32082
Closed
1 of 2 tasks

Airflow Connection Type Azure Blob Storage - Shared Access Key field #31811

manmeetkaur opened this issue Jun 9, 2023 · 3 comments · Fixed by #32082

Comments

@manmeetkaur
Copy link

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Airflow Version 2.5.2

Issue: Create a connection of type azure blob storage using the method #3 described in OSS docs.

When we use the storage account name and the access key as in the below screenshot, the connection works just fine.
connection-blob-storage-access-key

Then what is the purpose of the extra field called Blob Storage Shared Access Key(Optional)? When I tried to put the access key in this field, then connection fails with the below error on testing:

invalid url http://

Code reference: https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/hooks/wasb.py#L190-L192

What you think should happen instead

No response

How to reproduce

  • Create a Azure Storage account
  • Go to Access Control (IAM), copy the key
  • Spin up new local airflow environment using Astro CLI with runtime version 7.4.1
  • Go to Airflow UI -> Admin -> Connections -> create a new connection of type Azure blob Storage
  • Enter the name of the storage account in Blob Storage Login and the key copied from Azure to Blob Shared Access Key (Optional) and click on Test connection.

Operating System

Astro runtime 7.4.1 image on MacOS

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@manmeetkaur manmeetkaur added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Jun 9, 2023
@potiuk potiuk added good first issue and removed kind:bug This is a clearly a bug labels Jun 9, 2023
@Adaverse
Copy link
Contributor

Adaverse commented Jun 9, 2023

Will take this up!

@potiuk
Copy link
Member

potiuk commented Jun 9, 2023

feel free.

@hussein-awala hussein-awala removed the needs-triage label for new issues that we didn't triage yet label Jun 9, 2023
@Adaverse
Copy link
Contributor

Then what is the purpose of the extra field called Blob Storage Shared Access Key(Optional)?

To answer above question, we neeed to look into the signature of BlobServiceClient -
BlobServiceClient(account_url: str, credential: str | Dict[str, str] | AzureNamedKeyCredential | AzureSasCredential | TokenCredential | None = None, **kwargs: Any)

We can see account_url is required but credential is optional. The reason why credentail is optional because the account_url can itself contain the credential (example SAS). Since you are filling the shared access key field, it expects the account_url field to be filled as well of the form https://{account_name}.blob.core.windows.net and since this field is blank in your case hence the error. The Blob Storage Login is completly ignored in this case.

The first case works just fine because URL is being built here using Blob Storage Login

Will create a PR to handle this case. But I hope it clears the doubt here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants