Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sentinel for tags without values #20388

Merged
merged 1 commit into from
Mar 19, 2024
Merged

sentinel for tags without values #20388

merged 1 commit into from
Mar 19, 2024

Conversation

sryza
Copy link
Contributor

@sryza sryza commented Mar 10, 2024

Summary & Motivation

Adds a special sentinel value for tags that indicates "this tag is just a string, not a key-value pair". How this sentinel will be used:

  • In the CLI, when a selection like --select tag:foo is provided, we'll interpret it as AssetSelection.tag("foo", "__dagster_novalue") (implemented in this PR)
  • The dbt integration will provide this value for all tags (because dbt tags are strings).
  • In the UI, we'll hide this sentinel value. I.e. a tags dict like {"foo": "fooval", "bar": "__dagster_novalue"} will be rendered as "foo=fooval, bar" in the UI.

How I Tested These Changes

Copy link
Member

@schrockn schrockn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider doing a non-string sentinel? I could be convinced either way. But if we are going to do this partition keys too I want to be consistent.

@sryza
Copy link
Contributor Author

sryza commented Mar 11, 2024

Did you consider doing a non-string sentinel? I could be convinced either way. But if we are going to do this partition keys too I want to be consistent.

Yeah I've gone back and forth on this and didn't reach a clear answer. I started with the string sentinel because it was easier to implement - doesn't require changing types everywhere. Going with a non-string sentinel would send us in the direction of accepting that on run tags as well, which seemed like it could have downstream implications. Could definitely be convinced either way.

@sryza sryza force-pushed the asset-selection-tags branch from cbc1643 to c7ef26e Compare March 12, 2024 00:44
sryza added a commit that referenced this pull request Mar 12, 2024
## Summary & Motivation

This PR passes tags up through graphql to the UI and displays them on
the asset overview page. It doesn't yet support the no-value tags
introduced in this PR: #20388.

Stacks on top of #20351.

<img width="1167" alt="image"
src="https://github.com/dagster-io/dagster/assets/654855/97bcb447-06e6-4cd1-a0f2-4fdfffdd1d0d">

## How I Tested These Changes
@sryza sryza changed the base branch from asset-selection-tags to tags-asset-selection March 18, 2024 20:54
@sryza sryza force-pushed the tag-no-value branch 2 times, most recently from 3b5f6af to a1230f2 Compare March 18, 2024 21:22
@sryza sryza force-pushed the tags-asset-selection branch from 7673295 to 35cdf77 Compare March 18, 2024 21:24
@sryza sryza force-pushed the tags-asset-selection branch from 35cdf77 to 903a174 Compare March 18, 2024 21:34
@clairelin135
Copy link
Contributor

I'm ok with the string version of this, given that runless event tags are also strings rather than objects. I can't see any major benefits to making it an object sentinel instead.

That being said, looks like our APIs only accept dict mappings right now, so where will this be used? Is the intent to automatically ingest dbt tags and punt on python APIs?

@sryza sryza force-pushed the tags-asset-selection branch from 903a174 to cdde23f Compare March 18, 2024 22:02
@sryza
Copy link
Contributor Author

sryza commented Mar 18, 2024

That being said, looks like our APIs only accept dict mappings right now, so where will this be used? Is the intent to automatically ingest dbt tags and punt on python APIs?

Good question. I think there three stages here:

  1. Automatically ingest dbt tags. That can be done as soon as this PR is merged.
  2. Expose the TAG_NO_VALUE constant as a public API. I think we should do this before launch, but want to have a little more time to bikeshed on the name.
  3. Have tags arguments accept AbstractSet[str] values. I'm still not sure we should do this. Would like to run it by Nick before rushing into it.

Base automatically changed from tags-asset-selection to master March 18, 2024 23:27
branch-name: tag-no-value
@clairelin135
Copy link
Contributor

@sryza that makes sense to me, I'm comfortable with (1) and (2).

@sryza sryza merged commit 84e712f into master Mar 19, 2024
1 check passed
@sryza sryza deleted the tag-no-value branch March 19, 2024 18:06
benpankow pushed a commit that referenced this pull request Mar 21, 2024
## Summary & Motivation

Adds a special sentinel value for tags that indicates "this tag is just
a string, not a key-value pair". How this sentinel will be used:
- In the CLI, when a selection like `--select tag:foo` is provided,
we'll interpret it as `AssetSelection.tag("foo", "__dagster_novalue")`
(implemented in this PR)
- The dbt integration will provide this value for all tags (because dbt
tags are strings).
- In the UI, we'll hide this sentinel value. I.e. a tags dict like
`{"foo": "fooval", "bar": "__dagster_novalue"}` will be rendered as
"foo=fooval, bar" in the UI.

## How I Tested These Changes
PedramNavid pushed a commit that referenced this pull request Mar 28, 2024
## Summary & Motivation

This PR passes tags up through graphql to the UI and displays them on
the asset overview page. It doesn't yet support the no-value tags
introduced in this PR: #20388.

Stacks on top of #20351.

<img width="1167" alt="image"
src="https://github.com/dagster-io/dagster/assets/654855/97bcb447-06e6-4cd1-a0f2-4fdfffdd1d0d">

## How I Tested These Changes
PedramNavid pushed a commit that referenced this pull request Mar 28, 2024
## Summary & Motivation

Adds a special sentinel value for tags that indicates "this tag is just
a string, not a key-value pair". How this sentinel will be used:
- In the CLI, when a selection like `--select tag:foo` is provided,
we'll interpret it as `AssetSelection.tag("foo", "__dagster_novalue")`
(implemented in this PR)
- The dbt integration will provide this value for all tags (because dbt
tags are strings).
- In the UI, we'll hide this sentinel value. I.e. a tags dict like
`{"foo": "fooval", "bar": "__dagster_novalue"}` will be rendered as
"foo=fooval, bar" in the UI.

## How I Tested These Changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants