Adds comet-ml plugin example #1702

thomasjpfan · 2024-07-03T03:10:41Z

This PR adds a comet-ml example.

To fully run the example, you need to use this ImageSpec, because the plugin is in a PR:

comet_ml_package = (
    "git+https://github.com/thomasjpfan/flytekit.git@"
    "d04d0129f99eb7e7edcffeae04d4f401f49be6d6#subdirectory=plugins/flytekit-comet-ml"
)

image = ImageSpec(
    builder="fast-builder",
    name="unionai",
    apt_packages=["git"],
    packages=[
        "torch==2.3.1",
        "comet-ml==3.43.2",
        "lightning==2.3.0",
        comet_ml_package,
        "unionai==0.1.43",
        "torchvision",
    ],
    registry="ghcr.io/thomasjpfan",
)

flyteorg/flytekit#2550

Future-Outlier · 2024-07-17T17:33:37Z

examples/comet_ml_plugin/comet_ml_plugin/comet_ml_example.py

+        "comet-ml==3.43.2",
+        "lightning==2.3.0",
+        "flytekitplugins-comet-ml",
+        "unionai==0.1.43",


We don't need the unionai dependency, right?
Should we add this in Union's doc instead of Open Source's doc?

This was a mistake. unionai is not needed.

Future-Outlier · 2024-07-17T17:35:18Z

examples/comet_ml_plugin/comet_ml_plugin/comet_ml_example.py

+
+REGISTRY = "localhost:30000"
+image = ImageSpec(
+    name="unionai",


same as below, we should use ghcr.io/flyteorg

For ImageSpec, most users will not have push access to ghcr.io/flyteorg, so they will need to change the registry to run it.

Using localhost:30000 means I am optimizing for the sandbox use case. An alternative is to use os.getenv("REGISTRY", "localhost:30000"), and let it be an environment variable.

No problem, I like your idea

Future-Outlier

LGTM

Future-Outlier · 2024-07-17T18:21:29Z

examples/comet_ml_plugin/comet_ml_plugin/comet_ml_example.py

+
+REGISTRY = "localhost:30000"
+image = ImageSpec(
+    name="unionai",


No problem, I like your idea

Future-Outlier · 2024-07-17T18:22:09Z

examples/comet_ml_plugin/comet_ml_plugin/comet_ml_example.py

+#     dynamic-log-links:
+#       - comet-ml-execution-id:
+#           displayName: Comet
+#           templateUris: {{ .taskConfig.host }}/{{ .taskConfig.workspace }}/{{ .taskConfig.project_name }}/{{ .executionName }}{{ .nodeId }}{{ .taskRetryAttempt }}{{ .taskConfig.link_suffix }}


We should add double quote here.

Future-Outlier · 2024-07-17T18:22:19Z

examples/comet_ml_plugin/comet_ml_plugin/comet_ml_example.py

+#           templateUris: {{ .taskConfig.host }}/{{ .taskConfig.workspace }}/{{ .taskConfig.project_name }}/{{ .executionName }}{{ .nodeId }}{{ .taskRetryAttempt }}{{ .taskConfig.link_suffix }}
+#       - comet-ml-custom-id:
+#           displayName: Comet
+#           templateUris: {{ .taskConfig.host }}/{{ .taskConfig.workspace }}/{{ .taskConfig.project_name }}/{{ .taskConfig.experiment_key }}


neverett

Left a few suggestions, otherwise LGTM

neverett · 2024-07-23T20:49:59Z

examples/comet_ml_plugin/README.md

+.. tags:: Integration, Data, Metrics, Intermediate
+```
+
+Comet’s machine learning platform integrates with your existing infrastructure and tools so you can manage, visualize, and optimize models—from training runs to production monitoring. This plugin integrates Flyte with Comet by configuring links between the two platforms.


Suggested change

Comet’s machine learning platform integrates with your existing infrastructure and tools so you can manage, visualize, and optimize models—from training runs to production monitoring. This plugin integrates Flyte with Comet by configuring links between the two platforms.

Comet’s machine learning platform integrates with your existing infrastructure and tools so you can manage, visualize, and optimize models from training runs to production monitoring. This plugin integrates Flyte with Comet by configuring links between the two platforms.

neverett · 2024-07-23T20:50:47Z

examples/comet_ml_plugin/comet_ml_plugin/comet_ml_example.py

+#
+# # Comet Example
+# Comet’s machine learning platform integrates with your existing infrastructure and
+# tools so you can manage, visualize, and optimize models—from training runs to


Suggested change

# tools so you can manage, visualize, and optimize models—from training runs to

# tools so you can manage, visualize, and optimize models from training runs to

neverett · 2024-07-23T20:51:04Z

examples/comet_ml_plugin/comet_ml_plugin/comet_ml_example.py

+
+# %% [markdown]
+# First, we specify the project and workspace that we will use with Comet's platform
+# Please update `PROJECT_NAME` and `WORKSPACE` to the value associated with your account.


Suggested change

# Please update `PROJECT_NAME` and `WORKSPACE` to the value associated with your account.

# Please update `PROJECT_NAME` and `WORKSPACE` to the values associated with your account.

thomasjpfan · 2024-08-01T16:31:38Z

@neverett The comet-ml check is passing now: https://github.com/flyteorg/flytesnacks/actions/runs/10182511543/job/28165251450?pr=1702

There are a bunch other tests that are failing tho.

neverett · 2024-08-01T18:06:42Z

@thomasjpfan can you try merging master in and rerunning?

fiedlerNr9 · 2024-08-13T15:37:31Z

Running the example given in this PR was not working for me with ImageSpec.
MisconfigurationException: CometLogger requires either api_key or save_dir during initialization.
Execution on Dogfood

I had to add the following command to create a home dir & assign permissions for the flytekit user ->

image = ImageSpec(
    name="comet-ml",
    packages=[
        "torch==2.3.1",
        "comet-ml==3.43.2",
        "lightning==2.3.0",
        "flytekitplugins-comet-ml",
        "torchvision",
    ],
    commands=["mkdir -p /home/flytekit", "chown -R flytekit /home/flytekit"],
    registry=REGISTRY,
)

Anyone experiencing the same?

thomasjpfan · 2024-08-13T20:43:04Z

For this example to work, it needs ImageSpec(builder="default", ...). The envd builder does not create a home directory for it's user.

Signed-off-by: Thomas J. Fan <[email protected]>

thomasjpfan requested review from neverett and ppiegaze as code owners July 3, 2024 03:10

thomasjpfan mentioned this pull request Jul 3, 2024

Adds comet-ml plugin flyteorg/flytekit#2550

Merged

Future-Outlier reviewed Jul 17, 2024

View reviewed changes

Future-Outlier previously approved these changes Jul 17, 2024

View reviewed changes

thomasjpfan dismissed Future-Outlier’s stale review via e54796e July 17, 2024 19:52

Future-Outlier previously approved these changes Jul 17, 2024

View reviewed changes

thomasjpfan dismissed Future-Outlier’s stale review via 066bf9e July 18, 2024 19:50

neverett previously approved these changes Jul 23, 2024

View reviewed changes

thomasjpfan dismissed neverett’s stale review via acbbe8a July 24, 2024 13:45

ppiegaze previously approved these changes Nov 11, 2024

View reviewed changes

thomasjpfan dismissed ppiegaze’s stale review via b5d0f5b November 11, 2024 20:03

Adds comet-ml plugin example

8256865

Signed-off-by: Thomas J. Fan <[email protected]>

thomasjpfan force-pushed the thomasjpfan/comet-ml-init branch from 0c6983b to 8256865 Compare November 12, 2024 19:54

thomasjpfan enabled auto-merge (squash) November 13, 2024 15:30

ppiegaze self-requested a review November 13, 2024 22:21

ppiegaze approved these changes Nov 13, 2024

View reviewed changes

thomasjpfan merged commit 72aa5b2 into flyteorg:master Nov 13, 2024
61 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds comet-ml plugin example #1702

Adds comet-ml plugin example #1702

thomasjpfan commented Jul 3, 2024 •

edited

Loading

Future-Outlier Jul 17, 2024

thomasjpfan Jul 17, 2024

Future-Outlier Jul 17, 2024

thomasjpfan Jul 17, 2024

Future-Outlier Jul 17, 2024

Future-Outlier left a comment

Future-Outlier Jul 17, 2024

Future-Outlier Jul 17, 2024

Future-Outlier Jul 17, 2024

neverett left a comment

neverett Jul 23, 2024

neverett Jul 23, 2024

neverett Jul 23, 2024

thomasjpfan commented Aug 1, 2024

neverett commented Aug 1, 2024

fiedlerNr9 commented Aug 13, 2024 •

edited

Loading

thomasjpfan commented Aug 13, 2024

	Comet’s machine learning platform integrates with your existing infrastructure and tools so you can manage, visualize, and optimize models—from training runs to production monitoring. This plugin integrates Flyte with Comet by configuring links between the two platforms.
	Comet’s machine learning platform integrates with your existing infrastructure and tools so you can manage, visualize, and optimize models from training runs to production monitoring. This plugin integrates Flyte with Comet by configuring links between the two platforms.

	# tools so you can manage, visualize, and optimize models—from training runs to
	# tools so you can manage, visualize, and optimize models from training runs to

	# Please update `PROJECT_NAME` and `WORKSPACE` to the value associated with your account.
	# Please update `PROJECT_NAME` and `WORKSPACE` to the values associated with your account.

Adds comet-ml plugin example #1702

Adds comet-ml plugin example #1702

Conversation

thomasjpfan commented Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Future-Outlier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neverett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomasjpfan commented Aug 1, 2024

neverett commented Aug 1, 2024

fiedlerNr9 commented Aug 13, 2024 • edited Loading

thomasjpfan commented Aug 13, 2024

thomasjpfan commented Jul 3, 2024 •

edited

Loading

fiedlerNr9 commented Aug 13, 2024 •

edited

Loading