You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
U03Q27RQY76: We've been using dagster for a few months, and have accumulated enough jobs that we need to do some reorg-ing of one massive repository to multiple smaller repositories. If I change which repository a given dagster job is a part of, does that retain the run/partition history? I did a rename locally and looked like it carried history over, but just wanted to get confirmation before pursuing this...especially since I don't have partition history locally
UM49TQ8EB: Hi Adam. What version are you running? As of 1.0 this is no longer true. We used to show runs of identically named jobs, independent of repository, but as of 1.0 we only show runs that were created by jobs for that specific repository. Ideally we’d have some run migration CLI tools, but I can walk you through how to do those run migrations by running a data migration against your DB if you’re interested.
U03Q27RQY76: We're on 0.15.8, but I plan to check out 1.0 later this week (not sure the private module changes will work with our custom executor and partition selector). I figured a migration might be necessary - totally willing to write/run one!
UM49TQ8EB: All runs (that are up to date on run migrations) should contain a hidden run tag in the run_tags table with key .dagster/repository and value of the form <repository_name>@<location_name> … We use this value to filter by the job’s current repository. If you move the job to a differently named repository, it won’t be able to find the historic runs unless you run an update to the new repository name. If you’re splitting the mega-repo into smaller repos, you may have to join the run_tags against the runs table to filter by job name to get the appropriate new repository name.
UM49TQ8EB: <@U018K0G2Y85> issue create utility CLI to migrate job runs from one repository name to another
Message from the maintainers:
Do you care about this too? Give it a 👍. We factor engagement into prioritization.
The text was updated successfully, but these errors were encountered:
Issue from the Dagster Slack
This issue was generated from the slack conversation at: https://dagster.slack.com/archives/C01U954MEER/p1660003947447499?thread_ts=1660003947.447499&cid=C01U954MEER
Conversation excerpt
U03Q27RQY76: We've been using dagster for a few months, and have accumulated enough jobs that we need to do some reorg-ing of one massive repository to multiple smaller repositories. If I change which repository a given dagster job is a part of, does that retain the run/partition history? I did a rename locally and looked like it carried history over, but just wanted to get confirmation before pursuing this...especially since I don't have partition history locally
UM49TQ8EB: Hi Adam. What version are you running? As of
1.0
this is no longer true. We used to show runs of identically named jobs, independent of repository, but as of1.0
we only show runs that were created by jobs for that specific repository. Ideally we’d have some run migration CLI tools, but I can walk you through how to do those run migrations by running a data migration against your DB if you’re interested.U03Q27RQY76: We're on 0.15.8, but I plan to check out 1.0 later this week (not sure the private module changes will work with our custom executor and partition selector). I figured a migration might be necessary - totally willing to write/run one!
UM49TQ8EB: All runs (that are up to date on run migrations) should contain a hidden run tag in the
run_tags
table with key.dagster/repository
and value of the form<repository_name>@<location_name>
… We use this value to filter by the job’s current repository. If you move the job to a differently named repository, it won’t be able to find the historic runs unless you run an update to the new repository name. If you’re splitting the mega-repo into smaller repos, you may have to join therun_tags
against theruns
table to filter by job name to get the appropriate new repository name.UM49TQ8EB: <@U018K0G2Y85> issue create utility CLI to migrate job runs from one repository name to another
Message from the maintainers:
Do you care about this too? Give it a 👍. We factor engagement into prioritization.
The text was updated successfully, but these errors were encountered: