-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kedro Viz Static Website hosting on Azure #1708
Kedro Viz Static Website hosting on Azure #1708
Conversation
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
the only way we could slice this PR was keep FE and BE seperate. There's some refactoring work which could be removed and put in another PR but it's very minimal. If you like, Ravi or I can quickly give you an overview which will help with the review. |
Hi @astrojuanlu , I tried to make this in 2 PRs but the tests were failing. May be you can give a 2 pass review. There are test files which can be reviewed in the 2nd pass. But, I agree there are lot of changes in this single PR. |
This is certainly not true. There's many ways in which this PR could have been built up, introducing the deployers separately would already have been 1 step. Of course, this involves more work, but the trade off you make with PRs is the following: small easy to follow PRs will be quick to review and will get merged faster, large PRs such as this one require much longer time to review and thus in the end the work will take a lot longer to merge. Ensuring high quality is also harder with big PRs, because it's much easier to miss things. |
Yes, now that I am slicing it the refactor is quite a lot. I will release the refactor PR shortly. Thanks for pushing back on this. |
…ature/shareableviz-az Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
|
||
from kedro_viz import __version__ | ||
|
||
_AZ_PROTOCOL = "abfs" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we reused logic from kedro? I don't think abfs
is the only protocols supported by Azure. See https://github.com/fsspec/adlfs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here we are using Gen2 filesystem to build the remote path (we will mention this in our docs that azure deployment is supported with gen2 filesystem). Could you please point me to what reuse logic from kedro you are referring to ? Thank you
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/kedro-org/kedro/blob/80ad182a220d691c77efc05cc4e2b36dcecb6ac7/kedro/io/core.py#L711
95% identical to fsspec
to handle some extra edge cases. If this only support abfs
it is fine too, but worth mentioning in the docs.
super().__init__() | ||
self._endpoint = endpoint | ||
self._bucket_name = bucket_name | ||
self._path = f"{_AZ_PROTOCOL}://$web" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is $web
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$web is the default container in AzureBlobStorage when someone enables static website hosting. We will mention this in the docs that the users need to have this container to use static hosting.
package/setup.py
Outdated
"aws": ["s3fs>=2021.4,<2024.1"], | ||
"azure": ["adlfs>=2021.4,<2024.1"], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there any documentation explains why this pin differently from kedro
? It would be nice even if it's just an in-line comment. Some users were asking why this is pinned, from my understanding it only affect kedro-viz user who is using the static site deployment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure, the issue was files were being copied differently from fsspec>=2024.1
. The project structure was not maintained. But, you are right I will try to see if we can remove the upperbound, if not will document this with a comment here and also in the deployment docs. Thank you !
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ravi-kumar-pilla I tested with the latest version of fsspec, s3fs and adlfs and its works fine for me.
Signed-off-by: ravi-kumar-pilla <[email protected]>
25a7aff
into
feature/shareableviz-extended-support
* refactor router to accept new deployer inputs (#1739) Signed-off-by: ravi-kumar-pilla <[email protected]> * Refactor Shareableviz CLI (#1740) * refactor router to accept new deployer inputs Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor cli for shareableviz deploy Signed-off-by: ravi-kumar-pilla <[email protected]> * merge router change Signed-off-by: ravi-kumar-pilla <[email protected]> * PR comments fix Signed-off-by: ravi-kumar-pilla <[email protected]> --------- Signed-off-by: ravi-kumar-pilla <[email protected]> * Kedro Viz Static Website hosting on Azure (#1708) * CLI command kedro viz build added * Lint fix * lint fix * Lint fix * add mypy ignore * Missing build file added * Lint error fix * BaseDeployer class added * Unused code removed * Fix lint issue * azure deploy initial draft Signed-off-by: ravi-kumar-pilla <[email protected]> * added base_deployer * add deployer factory * partial working draft Signed-off-by: ravi-kumar-pilla <[email protected]> * Test and comments of deployers updated * test draft Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * remove circular dependency Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * revert back consent Signed-off-by: ravi-kumar-pilla <[email protected]> * minor updates Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytests Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for azure shareableviz Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor and add timeout Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor cli Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytest Signed-off-by: ravi-kumar-pilla <[email protected]> * add release note Signed-off-by: ravi-kumar-pilla <[email protected]> * fix flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * fix PR comments and flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * testing flaky c y test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * resolve conflicts Signed-off-by: ravi-kumar-pilla <[email protected]> * fix PR comments Signed-off-by: ravi-kumar-pilla <[email protected]> * add back cypress flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove cypress flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove duplicate pytest parameter Signed-off-by: ravi-kumar-pilla <[email protected]> * remove fsspec upper bound Signed-off-by: ravi-kumar-pilla <[email protected]> --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: Jitendra Gundaniya <[email protected]> Co-authored-by: rashidakanchwala <[email protected]> * Kedro Viz Static Website Hosting on GCP (#1711) * CLI command kedro viz build added * Lint fix * lint fix * Lint fix * add mypy ignore * Missing build file added * Lint error fix * BaseDeployer class added * Unused code removed * Fix lint issue * azure deploy initial draft Signed-off-by: ravi-kumar-pilla <[email protected]> * added base_deployer * add deployer factory * partial working draft Signed-off-by: ravi-kumar-pilla <[email protected]> * Test and comments of deployers updated * test draft Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * remove circular dependency Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * revert back consent Signed-off-by: ravi-kumar-pilla <[email protected]> * initial draft Signed-off-by: ravi-kumar-pilla <[email protected]> * minor updates Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytests Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for azure shareableviz Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor and add timeout Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor cli Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytest Signed-off-by: ravi-kumar-pilla <[email protected]> * add release note Signed-off-by: ravi-kumar-pilla <[email protected]> * fix flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * fix PR comments and flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * testing flaky c y test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for gcp Signed-off-by: ravi-kumar-pilla <[email protected]> * fix gcp pytest coverage Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytest Signed-off-by: ravi-kumar-pilla <[email protected]> * revert file permission change --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: Jitendra Gundaniya <[email protected]> Co-authored-by: rashidakanchwala <[email protected]> * Sharable Kedro-Viz UI for AWS, GCP and Azure (#1718) * shareable viz with multiple platform UI * test fix * Relative path fix for user entered url * Test updated * Test updated * UI alignment and endpoint help text added. Signed-off-by: <> * Add Kedro-Viz documentation URLs and update modal content Signed-off-by: <> * Code review suggestions added Signed-off-by: <> * Refactor form field names in ShareableUrlModal component Signed-off-by: <> * Add cursor style to input field in shareable-url-modal.js Signed-off-by: <> * Add cursor style to input field in shareable-url-modal.js Signed-off-by: <> * Variable names updated. Signed-off-by: <> * add exception handlers Signed-off-by: ravi-kumar-pilla <[email protected]> * Disclaimer note added and shareable-url-modal component refactored by moving major jsx to other file Signed-off-by: <> * Wrapped the prop methods with useCallback Signed-off-by: <> * info text updated. Signed-off-by: <> * Test fixed Signed-off-by: <> * test fix and rearranging jsx Signed-off-by: <> * Cypress test fix Signed-off-by: <> * button text updated Signed-off-by: <> * Keeping button text as Publish for a link seetings Signed-off-by: <> --------- Signed-off-by: <> Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: ravi-kumar-pilla <[email protected]> * Kedro-viz doc link added in disclaimer note. Signed-off-by: <> --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Signed-off-by: <> Co-authored-by: Jitendra Gundaniya <[email protected]> Co-authored-by: rashidakanchwala <[email protected]> Co-authored-by: Jitendra Gundaniya <[email protected]>
…dular pipelines flag (#1799) * shareable viz with multiple platform UI * test fix * Relative path fix for user entered url * Test updated * Test updated * refactor router to accept new deployer inputs (#1739) Signed-off-by: ravi-kumar-pilla <[email protected]> * Refactor Shareableviz CLI (#1740) * refactor router to accept new deployer inputs Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor cli for shareableviz deploy Signed-off-by: ravi-kumar-pilla <[email protected]> * merge router change Signed-off-by: ravi-kumar-pilla <[email protected]> * PR comments fix Signed-off-by: ravi-kumar-pilla <[email protected]> --------- Signed-off-by: ravi-kumar-pilla <[email protected]> * Kedro Viz Static Website hosting on Azure (#1708) * CLI command kedro viz build added * Lint fix * lint fix * Lint fix * add mypy ignore * Missing build file added * Lint error fix * BaseDeployer class added * Unused code removed * Fix lint issue * azure deploy initial draft Signed-off-by: ravi-kumar-pilla <[email protected]> * added base_deployer * add deployer factory * partial working draft Signed-off-by: ravi-kumar-pilla <[email protected]> * Test and comments of deployers updated * test draft Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * remove circular dependency Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * revert back consent Signed-off-by: ravi-kumar-pilla <[email protected]> * minor updates Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytests Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for azure shareableviz Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor and add timeout Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor cli Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytest Signed-off-by: ravi-kumar-pilla <[email protected]> * add release note Signed-off-by: ravi-kumar-pilla <[email protected]> * fix flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * fix PR comments and flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * testing flaky c y test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * resolve conflicts Signed-off-by: ravi-kumar-pilla <[email protected]> * fix PR comments Signed-off-by: ravi-kumar-pilla <[email protected]> * add back cypress flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove cypress flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove duplicate pytest parameter Signed-off-by: ravi-kumar-pilla <[email protected]> * remove fsspec upper bound Signed-off-by: ravi-kumar-pilla <[email protected]> --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: Jitendra Gundaniya <[email protected]> Co-authored-by: rashidakanchwala <[email protected]> * Kedro Viz Static Website Hosting on GCP (#1711) * CLI command kedro viz build added * Lint fix * lint fix * Lint fix * add mypy ignore * Missing build file added * Lint error fix * BaseDeployer class added * Unused code removed * Fix lint issue * azure deploy initial draft Signed-off-by: ravi-kumar-pilla <[email protected]> * added base_deployer * add deployer factory * partial working draft Signed-off-by: ravi-kumar-pilla <[email protected]> * Test and comments of deployers updated * test draft Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * remove circular dependency Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * revert back consent Signed-off-by: ravi-kumar-pilla <[email protected]> * initial draft Signed-off-by: ravi-kumar-pilla <[email protected]> * minor updates Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytests Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for azure shareableviz Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor and add timeout Signed-off-by: ravi-kumar-pilla <[email protected]> * refactor cli Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytest Signed-off-by: ravi-kumar-pilla <[email protected]> * add release note Signed-off-by: ravi-kumar-pilla <[email protected]> * fix flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * fix PR comments and flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * testing flaky c y test Signed-off-by: ravi-kumar-pilla <[email protected]> * remove flaky test Signed-off-by: ravi-kumar-pilla <[email protected]> * add pytest for gcp Signed-off-by: ravi-kumar-pilla <[email protected]> * fix gcp pytest coverage Signed-off-by: ravi-kumar-pilla <[email protected]> * fix lint Signed-off-by: ravi-kumar-pilla <[email protected]> * update pytest Signed-off-by: ravi-kumar-pilla <[email protected]> * revert file permission change --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: Jitendra Gundaniya <[email protected]> Co-authored-by: rashidakanchwala <[email protected]> * Initial commit * Set query params from local storage on load * Test fix * test fix * Test fixes * Some method docs added. * Shortening existing URL query params focused_id, selected_id, selected_name and pipeline_id * Fix experiment tracking search query test * Clear filter button added * Missing test added * Release note added * Clear to Reset * Release note & retain qparams updated * fixes * almost done * missing nodeTypes added * URL Params handling moved to user-generated-pathname.js * Review suggestion added * getKeysByValue moved * updateStateWithFilters renamed * Reset button status check removed * Reset buton status test removed * Reset button status check revert * Reset button UI update * code review suggestions for reset button status * state compare for nodeTypes * On group filter click fix * Task to node mapping added * All nodeTypes match UI labels with URLparmas * Removed console.log * modified test as per query param name update --------- Signed-off-by: ravi-kumar-pilla <[email protected]> Co-authored-by: Ravi Kumar Pilla <[email protected]> Co-authored-by: rashidakanchwala <[email protected]>
Description
Resolves #1674
Development notes
s3fs
andadlfs
tosetup.py -> extras_require
. This will help users to install platform specific dependencies based on the cloud platform. For examplepip install 'kedro-viz[aws]'
will install aws deployer specific dependenciesNOTE: There will be a new PR to update documentation for shareable-viz
QA notes
Setup Environment for Azure
Route 1 - Authorizing via Azure ServicePrincipal
What users need to do - Create Storage Account + App Registration + Role permissions
On Azure portal
On command line before running kedro viz deploy
Route 2 - Using Account Key
What users need to do - Create Storage Account
On Azure portal
On command line before running kedro viz deploy
Run deploy command
kedro viz deploy --platform=azure --endpoint=https://example-bucket.z13.web.core.windows.net/ --bucket-name=example-bucket
Note:
Storage account -> Capabilities -> Static Website Hosting
$web
is the default container that is created under the storage account. We will upload all the kedro viz files to this container (This container is mandatory for shareableviz)Checklist
RELEASE.md
file