Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add AzureBlobStorageContainer block #139

Merged
merged 12 commits into from
Mar 14, 2024

Conversation

desertaxle
Copy link
Member

@desertaxle desertaxle commented Mar 13, 2024

Adds an AzureBlobStorageContainer block as a replacement for the Azure block in the core library. Works for both deployment flow code storage and result storage.

Closes #135

Example

from prefect_azure import AzureBlobStorageCredentials
from prefect_azure.blob_storage import AzureBlobStorageContainer

credentials = AzureBlobStorageCredentials(
    connection_string="connection_string",
)
container = AzureBlobStorageContainer(
    container_name="container",
    credentials=credentials,
)
# Downloads all files from the specified folder
container.download_folder_to_path(
    from_folder="folder",
    to_folder="local_folder"
)

Screenshots

Checklist

  • References any related issue by including "Closes #" or "Closes ".
    • If no issue exists and your change is not a small fix, please create an issue first.
  • Includes tests or only affects documentation.
  • Passes pre-commit checks.
    • Run pre-commit install && pre-commit run --all locally for formatting and linting.
  • Includes screenshots of documentation updates.
    • Run mkdocs serve view documentation locally.

@serinamarie
Copy link
Contributor

serinamarie commented Mar 14, 2024

I'd expect to get this error sooner but that's probably for another PR (I've created an issue for it):

ValueError: Connection string is either blank or malformed.
from prefect_azure import AzureBlobStorageCredentials
from prefect_azure.blob_storage import AzureBlobStorageContainer

my_connection_string = "dsfdsf"  # fake

credentials = AzureBlobStorageCredentials(
    connection_string=my_connection_string,
)
container = AzureBlobStorageContainer(
    container_name="testing314",  # container exists
    credentials=credentials,
)
# Downloads all files from the specified folder
container.download_folder_to_path(
    from_folder="requirements.txt",
    to_folder="local_folder"
)
12:48:34.514 | INFO    | prefect.AzureBlobStorageContainer - Downloading folder from container testing314 to path local_folder
Traceback (most recent call last):
  File "/Users/bean/code-oss/prefect-azure/prefect_azure/credentials.py", line 65, in inner
    return func(*args, **kwargs)
  File "/Users/bean/code-oss/prefect-azure/prefect_azure/credentials.py", line 259, in get_container_client
    container_client = ContainerClient.from_connection_string(
  File "/Users/bean/code-oss/prefect-azure/azurevenv/lib/python3.10/site-packages/azure/storage/blob/_container_client.py", line 270, in from_connection_string
    account_url, secondary, credential = parse_connection_str(conn_str, credential, 'blob')
  File "/Users/bean/code-oss/prefect-azure/azurevenv/lib/python3.10/site-packages/azure/storage/blob/_shared/base_client.py", line 371, in parse_connection_str
    raise ValueError("Connection string is either blank or malformed.")
ValueError: Connection string is either blank or malformed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/bean/code-oss/prefect-azure/qa.py", line 14, in <module>
    container.download_folder_to_path(
  File "/Users/bean/code-oss/prefect-azure/azurevenv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 259, in coroutine_wrapper
    return call()
  File "/Users/bean/code-oss/prefect-azure/azurevenv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 431, in __call__
    return self.result()
  File "/Users/bean/code-oss/prefect-azure/azurevenv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 317, in result
    return self.future.result(timeout=timeout)
  File "/Users/bean/code-oss/prefect-azure/azurevenv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 178, in result
    return self.__get_result()
  File "/Users/bean/.pyenv/versions/3.10.4/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/Users/bean/code-oss/prefect-azure/azurevenv/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 388, in _run_async
    result = await coro
  File "/Users/bean/code-oss/prefect-azure/prefect_azure/blob_storage.py", line 283, in download_folder_to_path
    async with self.credentials.get_container_client(
  File "/Users/bean/code-oss/prefect-azure/prefect_azure/credentials.py", line 72, in inner
    raise ValueError(HELP_FMT.format(help_url=HELP_URLS[key])) from exc
ValueError: Please visit https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python#copy-your-credentials-from-the-azure-portal for retrieving the proper connection string.

Copy link
Contributor

@serinamarie serinamarie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🔥

@desertaxle desertaxle merged commit 90f6ae5 into main Mar 14, 2024
4 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create Azure Blob Storage Container block
2 participants