[dagster-wandb] Integration with Weights & Biases #10470

chrishiste · 2022-11-10T20:32:35Z

Co-authored-by: Dave La Chasse [email protected]
Co-authored-by: Ashraf Shaik [email protected]

Note: W&B partners with the Dagster team on this. If you are eager to use this integration for your own projects, please hold while we get this polished. Don't hesitate to provide feedback or ask questions :)

Summary & Motivation

This PR contains the integration code with Weights & Biases.

Making it easy within Dagster to:

use and create W&B Artifacts
use and create Registered Models in W&B Model Registry
run training jobs on dedicated compute using W&B Launch
use the wandb client in ops and assets

You will find code for the integration and examples on how to use the integration.

For more in depth documentation check the current draft.

Useful references:

Notes and questions for the Dagster team:

We're using the local storage to store downloaded Artifacts. Does Dagster always run with a local filesystem that we can use? And what would happen when there is no more disk space?
We linked to https://docs.dagster.io/integrations/wandb in the examples/with_wandb/README.md. Can you confirm it's where we documentation would eventually live?
I'm expecting your review to be mostly on the form over the content but we're open to refactor the code to suit Dagster coding style better. Don't hesitate to be vocal if something could be improved.

How I Tested These Changes

This integration has been extensively tested manually and through unit tests (included in the code).

Known Issues
tox fails currently due to a protobuf bug. There is an open Github issue. I was able to remove the error by pinning another version but I'm not sure of the implications. Probably better to wait for a fix until we release this.

vercel · 2022-11-10T20:32:38Z

@chrishiste is attempting to deploy a commit to the Elementl Team on Vercel.

A member of the Team first needs to authorize it.

vercel · 2022-11-10T20:32:40Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
dagster	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 8, 2023 at 7:32PM (UTC)

1 Ignored Deployment

Name	Status	Preview	Comments	Updated
dagit-storybook	⬜️ Ignored (Inspect)			Feb 8, 2023 at 7:32PM (UTC)

chrishiste · 2022-11-10T20:35:08Z

python_modules/dagster/setup.py

@@ -99,7 +99,7 @@ def get_version() -> str:
            "grpcio-tools>=1.32.0,<1.44.0",  # related to above grpcio pins
            "mock==3.0.5",
            "objgraph",
-            "protobuf==3.13.0",  # without this, pip will install the most up-to-date protobuf
+            "protobuf==3.20.3",  # without this, pip will install the most up-to-date protobuf


Note: this is updating the dagster module itself.

@yuhan is this concerning? That is changing the protobuf version in python_modules/dagster/setup.py

is this blocking this PR? asking because i'd recommend splitting this out to a standalone PR so in case of anything reverting could be easier.

Without this change, tests won't run due to incompatible versions. I can move this out to another PR. Should I do this?
Then we merge that single line change. Before this one.
@yuhan

ah i see. lets include it then.

ok finally removed the pin here in a separate PR: #11974

yuhan

Thanks a ton for putting this together. Overall it looks great!

A few feedback around the example:

could we add a tests dir examples/with_wandb/with_wandb_tests in the example so we can include the unit tests all together in our CI. example: https://github.com/dagster-io/dagster/blob/master/examples/development_to_production/development_to_production_tests
naming wise, i tend to use nouns for asset names (which usually represent some objects like artifacts, tables, models, etc) and verbs for op names (which often are operations therefore verbs)

i saw the tests are failing likely related to our set up for new module or example. i'll take a stab and try fixing those in this branch.

examples/with_wandb/with_wandb/assets/example/README.md

python_modules/dagster-wandb/dagster_wandb/io_manager.py

yuhan · 2022-11-17T08:52:33Z

python_modules/dagster-wandb/dagster_wandb/resources.py

+    description="Resource for interacting with Weights & Biases",
+)
+def wandb_resource(context: InitResourceContext) -> Dict[str, Any]:
+    """Resource for interacting with Weights & Biases."""


could maybe add an example for how to use this resource in the docstring. example: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-snowflake/dagster_snowflake/resources.py#L277-L314

Would it be okay to add a link to the documentation instead of something like the example you shared?

the nice thing about including the code snippets here is it'd show up in the api docs directly, for example: https://docs.dagster.io/_apidocs/libraries/dagster-airbyte so the reader doesn't need to navigate away

I see that makes sense I will create a commit including this, will let you know when it's ready

examples/with_wandb/with_wandb/repository.py

examples/with_wandb/with_wandb/assets/custom_serialization_example.py

python_modules/dagster-wandb/dagster_wandb/io_manager.py

chrishiste

@yuhan Thanks for the review. I've pushed another commit including your feedback. Please have a look 😄

examples/with_wandb/with_wandb/assets/advanced_example.py

chrishiste · 2022-12-02T18:34:10Z

python_modules/dagster/setup.py

@@ -99,7 +99,7 @@ def get_version() -> str:
            "grpcio-tools>=1.32.0,<1.44.0",  # related to above grpcio pins
            "mock==3.0.5",
            "objgraph",
-            "protobuf==3.13.0",  # without this, pip will install the most up-to-date protobuf
+            "protobuf==3.20.3",  # without this, pip will install the most up-to-date protobuf


@yuhan is this concerning? That is changing the protobuf version in python_modules/dagster/setup.py

python_modules/dagster-wandb/dagster_wandb/version.py

examples/with_wandb/with_wandb/repository.py

examples/with_wandb/with_wandb/assets/custom_serialization_example.py

chrishiste · 2022-12-02T19:07:15Z

python_modules/dagster-wandb/dagster_wandb/resources.py

+    description="Resource for interacting with Weights & Biases",
+)
+def wandb_resource(context: InitResourceContext) -> Dict[str, Any]:
+    """Resource for interacting with Weights & Biases."""


Would it be okay to add a link to the documentation instead of something like the example you shared?

python_modules/dagster-wandb/dagster_wandb/io_manager.py

Co-authored-by: Dave La Chasse <[email protected]> Co-authored-by: Ashraf Shaik <[email protected]>

yuhan

We're using the local storage to store downloaded Artifacts. Does Dagster always run with a local filesystem that we can use? And what would happen when there is no more disk space?

dagster is always running with a local filesystem, but when it's deployed in a remote setup, for example, each run could be in its own container which means local filesystem will be ephemera in a run scope, i.e. files downloaded to run 1's local filesystem won't be available to the next run.

i think it's ok to default to local filesystem, but it'd be great to give an option for users to download to other filesystems, e.g. an option to hook to an s3 bucket. -- but this isn't blocking this PR IMO. we can add it in a later iteration.

examples/with_wandb/with_wandb/assets/example/README.md

yuhan · 2023-01-09T05:52:16Z

examples/with_wandb/README.md

+
+This directory contains examples showcasing the integration with Weights & Biases (W&B).
+
+View this example in the Dagster docs at https://docs.dagster.io/integrations/wandb.


We linked to https://docs.dagster.io/integrations/wandb in the examples/with_wandb/README.md. Can you confirm it's where we documentation would eventually live?

is the idea to link to a guide/tutorial or an api doc? the integrations/ pages are usually hands-on guides whereas the PR will generate the api doc at https://docs.dagster.io/_apidocs/libraries/dagster-wandb
i saw you also linked the api doc to https://docs.wandb.ai/guides/integrations/dagster. so do you think this link can go to the guide on w&b's site or we can reword it to point to the api doc on dagster's site?

Good call, I changed all those links to point to the full docs on our side

yuhan · 2023-01-09T05:55:18Z

examples/with_wandb/with_wandb/repository.py

+from .ops.simple_job import simple_job_example
+
+
+@repository


heads up: as we're moving the top-level entry to using Definitions, i'll update this file to use the new apis.

context: #11167

yuhan · 2023-01-09T05:56:52Z

python_modules/dagster/setup.py

@@ -99,7 +99,7 @@ def get_version() -> str:
            "grpcio-tools>=1.32.0,<1.44.0",  # related to above grpcio pins
            "mock==3.0.5",
            "objgraph",
-            "protobuf==3.13.0",  # without this, pip will install the most up-to-date protobuf
+            "protobuf==3.20.3",  # without this, pip will install the most up-to-date protobuf


is this blocking this PR? asking because i'd recommend splitting this out to a standalone PR so in case of anything reverting could be easier.

yuhan

Thank you again for putting this together and bearing with my slow reviews. I can take it from here and aim to include in the next release : )

chrishiste · 2023-01-23T12:53:05Z

@yuhan Glad to hear it, thanks for the help :)

vercel · 2023-02-07T07:36:12Z

Deployment failed with the following error:

Resource is limited - try again in 6 hours (more than 100, code: "api-deployments-free-per-day").

yuhan · 2023-02-07T07:46:20Z

cherry-picked commits from #11830 here to fix the tests + merge latest master into this branch to prep for merging :)

yuhan · 2023-02-07T07:51:47Z

the follow up is #12059 which converts the example to using the latest Definitions apis

vercel · 2023-02-07T08:57:39Z

Deployment failed with the following error:

Resource is limited - try again in 5 hours (more than 100, code: "api-deployments-free-per-day").

vercel · 2023-02-07T09:57:51Z

Deployment failed with the following error:

Resource is limited - try again in 4 hours (more than 100, code: "api-deployments-free-per-day").

vercel · 2023-02-08T19:04:32Z

Deployment failed with the following error:

Resource is limited - try again in 3 minutes (more than 100, code: "api-deployments-free-per-day").

### Summary & Motivation depends on #10470 this PR converts the example from using repository to defs as the entry point, per changes in 1.1.7 (#11167) ### How I Tested These Changes - local dagit can load - unit test

Co-authored-by: Dave La Chasse <[email protected]> Co-authored-by: Ashraf Shaik <[email protected]> Note: W&B partners with the Dagster team on this. If you are eager to use this integration for your own projects, please hold while we get this polished. Don't hesitate to provide feedback or ask questions :) ### Summary & Motivation This PR contains the integration code with [Weights & Biases](https://docs.wandb.ai/). Making it easy within Dagster to: - use and create W&B [Artifacts](https://docs.wandb.ai/guides/artifacts) - use and create Registered Models in W&B [Model Registry](https://docs.wandb.ai/guides/models) - run training jobs on dedicated compute using W&B [Launch](https://docs.wandb.ai/guides/launch) - use the [wandb](https://github.com/wandb/wandb) client in ops and assets You will find code for the integration and examples on how to use the integration. For more in depth documentation check the [current draft](https://docs.google.com/document/d/10qOyhbJKnJR4kazGqMF22db6UhXpeHop3_QV90eaiW4/edit). **Useful references:** - https://docs.wandb.ai/ref/python/artifact - https://docs.wandb.ai/ref/python/run - https://docs.wandb.ai/guides/launch **Notes and questions for the Dagster team:** - We're using the local storage to store downloaded Artifacts. Does Dagster always run with a local filesystem that we can use? And what would happen when there is no more disk space? - We linked to `https://docs.dagster.io/integrations/wandb` in the `examples/with_wandb/README.md`. Can you confirm it's where we documentation would eventually live? - I'm expecting your review to be mostly on the form over the content but we're open to refactor the code to suit Dagster coding style better. Don't hesitate to be vocal if something could be improved. ### How I Tested These Changes This integration has been extensively tested manually and through unit tests (included in the code). **Known Issues** tox fails currently due to a protobuf bug. There is an open Github [issue](protocolbuffers/protobuf#10075). I was able to remove the error by pinning another version but I'm not sure of the implications. Probably better to wait for a fix until we release this. --------- Co-authored-by: Dave La Chasse <[email protected]> Co-authored-by: Ashraf Shaik <[email protected]> Co-authored-by: yuhan <[email protected]>

### Summary & Motivation depends on #10470 this PR converts the example from using repository to defs as the entry point, per changes in 1.1.7 (#11167) ### How I Tested These Changes - local dagit can load - unit test

chrishiste commented Nov 10, 2022

View reviewed changes

yuhan self-requested a review November 17, 2022 22:15

yuhan reviewed Nov 29, 2022

View reviewed changes

chrishiste commented Dec 2, 2022

View reviewed changes

chrishiste requested a review from yuhan December 6, 2022 19:09

chrishiste and others added 7 commits December 9, 2022 15:01

[dagster-wandb] Integration with Weights & Biases

d39e48f

Co-authored-by: Dave La Chasse <[email protected]> Co-authored-by: Ashraf Shaik <[email protected]>

PR amends

0606380

Moved the integration code to the right directory

2081e3e

fix: Bugs with partitioned jobs

f8a7918

feat: Change compute kind to W&B

d16d8ef

feat: Add W&B logo

b2540a7

docs: Improve integration documentation

2f559ac

chrishiste force-pushed the wandb-integration branch from 1e20d2e to 2f559ac Compare December 15, 2022 14:07

yuhan reviewed Jan 9, 2023

View reviewed changes

docs: Update links to full docs

e9b9e62

vercel bot deployed to Preview – dagster January 19, 2023 09:18 View deployment

yuhan approved these changes Jan 21, 2023

View reviewed changes

yuhan mentioned this pull request Feb 2, 2023

W&B integration follow up 1/ fixing tests #11830

Closed

update setup.py

3f82c68

vercel bot deployed to Preview – dagster February 7, 2023 07:15 View deployment

yuhan added 7 commits February 6, 2023 23:33

update python_modules/libraries/dagster-wandb/setup.py

2d8e67f

make ruff

b031604

regen api docs

65892f6

make black

cf9397e

fix examples/with_wandb tests

e592a43

fix mypy errors in example, add can_load test, refer repo in init

cf8bc5e

make black, add with_wandb to example download allowlist

028397a

yuhan added 2 commits February 6, 2023 23:41

Merge remote-tracking branch 'origin/master' into pr/10470

9a83af9

regen api docs

9dadac9

pyright

0c9ce71

yuhan force-pushed the wandb-integration branch from da39465 to 0c9ce71 Compare February 7, 2023 09:57

more pyright

497891f

vercel bot deployed to Preview – dagster February 7, 2023 17:18 View deployment

yuhan mentioned this pull request Feb 7, 2023

W&B integration follow up 2/ update example to defs #12059

Merged

yuhan added 2 commits February 7, 2023 14:21

Merge remote-tracking branch 'origin/master' into pr/10470

06534b1

more pyright

bce7cc9

yuhan requested a review from erinkcochran87 as a code owner February 8, 2023 00:59

Merge remote-tracking branch 'origin/master' into pr/10470

55cb943

Empty-Commit

1cedf73

vercel bot deployed to Preview – dagster February 8, 2023 19:32 View deployment

yuhan merged commit 7aecf4a into dagster-io:master Feb 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dagster-wandb] Integration with Weights & Biases #10470

[dagster-wandb] Integration with Weights & Biases #10470

chrishiste commented Nov 10, 2022 •

edited

Loading

vercel bot commented Nov 10, 2022

vercel bot commented Nov 10, 2022 •

edited

Loading

chrishiste Nov 10, 2022

chrishiste Dec 2, 2022

yuhan Jan 9, 2023

chrishiste Jan 19, 2023

yuhan Jan 19, 2023

yuhan Feb 7, 2023

yuhan left a comment

yuhan Nov 17, 2022

chrishiste Dec 2, 2022

yuhan Dec 7, 2022

chrishiste Dec 9, 2022

chrishiste left a comment

chrishiste Dec 2, 2022

chrishiste Dec 2, 2022

yuhan left a comment

yuhan Jan 9, 2023

chrishiste Jan 19, 2023

yuhan Jan 9, 2023

yuhan Jan 9, 2023

yuhan left a comment

chrishiste commented Jan 23, 2023

vercel bot commented Feb 7, 2023

yuhan commented Feb 7, 2023

yuhan commented Feb 7, 2023

vercel bot commented Feb 7, 2023

vercel bot commented Feb 7, 2023

vercel bot commented Feb 8, 2023


		This directory contains examples showcasing the integration with Weights & Biases (W&B).

		View this example in the Dagster docs at https://docs.dagster.io/integrations/wandb.

[dagster-wandb] Integration with Weights & Biases #10470

[dagster-wandb] Integration with Weights & Biases #10470

Conversation

chrishiste commented Nov 10, 2022 • edited Loading

Summary & Motivation

How I Tested These Changes

vercel bot commented Nov 10, 2022

vercel bot commented Nov 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuhan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrishiste left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuhan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuhan left a comment

Choose a reason for hiding this comment

chrishiste commented Jan 23, 2023

vercel bot commented Feb 7, 2023

yuhan commented Feb 7, 2023

yuhan commented Feb 7, 2023

vercel bot commented Feb 7, 2023

vercel bot commented Feb 7, 2023

vercel bot commented Feb 8, 2023

chrishiste commented Nov 10, 2022 •

edited

Loading

vercel bot commented Nov 10, 2022 •

edited

Loading