Shareable Viz with Azure

[NeroOkwa]

Description

This ticket is related to parent ticket #1116.
This will enable users to host, publish and share their Kedro-Viz pipeline visualisation with a URL on Azure Blob storage.

Context

We previously enabled users to publish and share versions of their Kedro-Viz pipeline visualisation with a URL on AWS S3 in #1116.

Enabling this feature will increase interoperability with the major cloud providers, providing Kedro-Viz access to non-technical stakeholders, and ultimately increasing adoption of Kedro-Viz.

Acceptance Criteria

A user is able to easily host and share a version of their Kedro-Viz pipeline visualisation with a URL via Azure Blob storage by:

• Configuring their storage account and region.
• Creating an access key.
• In the CLI, adding their environment variable with the access key.
• Running the kedro viz deploy command.

[rashidakanchwala]

We focus on cli first and then UI.

[ravi-kumar-pilla]

Hi Team,

I documented few findings on how we can approach hosting viz on Azure Blob storage.

Connecting fsspec to Azure Blob Storage

Route 1 - Authorizing via Azure ServicePrincipal



What users need to do - Create Storage Account + App Registration + Role permissions

On Azure portal

1. Create an App registration
2. Copy App registration parameters (Application (Client) ID, Directory (Tenant) ID)
3. Create and copy Client secret (Client Secret Value)
4. Create a Storage Account and configure it for static website hosting -
5. Create a storage container
6. Assign app and role to the storage account (assign a role of Storage Blob Data Contributor to the app registered)

On command line before running kedro viz deploy

1. Add environment variables - AZURE_STORAGE_CLIENT_ID (which is the application id in step2), AZURE_STORAGE_TENANT_ID (which is Directory ID in step2), AZURE_STORAGE_CLIENT_SECRET (which is the client secret value in step3)

Route 2 - Using Account Key

What users need to do - Create Storage Account

On Azure portal

1. Create a Storage Account and configure it for static website hosting - https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website-host
2. Copy the Account key from Security + Networking tab under storage account created in step 1

On command line before running kedro viz deploy

1. Add environment variable - AZURE_STORAGE_ACCOUNT_KEY (which is the account key value in step2)

Run deploy command (still needs discussion)
kedro viz deploy --platform=az --region=eastus --bucket-name=shareableviz

Note:

• Bucket name is the storage account name
• Region (Location) can be found in the Essentials section under overview tab of the storage account
• For static website hosting $web is the default container that is created under the storage account. We will upload all the kedro viz files to this container (This container is mandatory for shareableviz)

Outstanding Design questions

1. How to identify cloud provider when executed via command line ? Can we do kedro viz deploy --platform=az --region=eastus --bucket-name=shareableviz ?
2. Platform option can take values like az, aws, gcp, github

Outstanding Development issues

1. Copied files are not retaining MIME types ( All file types are defaulted to application/octet-stream ).

Useful Links


• https://docs.logz.io/docs/user-guide/data-hub/archive-restore/azure-blob-permissions/
• https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website-host
• https://pypi.org/project/adlfs/
• https://github.com/fsspec/adlfs

Thank you

[stephkaiser]

Work-in-progress designs can be found here - https://www.figma.com/file/3kSpvIO1veLKfy9qHxuXwF/Kedro-WIP?type=design&node-id=171%3A124&mode=design&t=qF2O83rywVEOBRHP-1