Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create utility CLI to migrate job runs from one repository name to another #9317

Closed
dagsir bot opened this issue Aug 9, 2022 · 1 comment
Closed
Assignees

Comments

@dagsir
Copy link

dagsir bot commented Aug 9, 2022

Issue from the Dagster Slack

This issue was generated from the slack conversation at: https://dagster.slack.com/archives/C01U954MEER/p1660003947447499?thread_ts=1660003947.447499&cid=C01U954MEER


Conversation excerpt

U03Q27RQY76: We've been using dagster for a few months, and have accumulated enough jobs that we need to do some reorg-ing of one massive repository to multiple smaller repositories. If I change which repository a given dagster job is a part of, does that retain the run/partition history? I did a rename locally and looked like it carried history over, but just wanted to get confirmation before pursuing this...especially since I don't have partition history locally
UM49TQ8EB: Hi Adam. What version are you running? As of 1.0 this is no longer true. We used to show runs of identically named jobs, independent of repository, but as of 1.0 we only show runs that were created by jobs for that specific repository. Ideally we’d have some run migration CLI tools, but I can walk you through how to do those run migrations by running a data migration against your DB if you’re interested.
U03Q27RQY76: We're on 0.15.8, but I plan to check out 1.0 later this week (not sure the private module changes will work with our custom executor and partition selector). I figured a migration might be necessary - totally willing to write/run one!
UM49TQ8EB: All runs (that are up to date on run migrations) should contain a hidden run tag in the run_tags table with key .dagster/repository and value of the form <repository_name>@<location_name> … We use this value to filter by the job’s current repository. If you move the job to a differently named repository, it won’t be able to find the historic runs unless you run an update to the new repository name. If you’re splitting the mega-repo into smaller repos, you may have to join the run_tags against the runs table to filter by job name to get the appropriate new repository name.
UM49TQ8EB: <@U018K0G2Y85> issue create utility CLI to migrate job runs from one repository name to another


Message from the maintainers:

Do you care about this too? Give it a 👍. We factor engagement into prioritization.

@prha
Copy link
Member

prha commented Oct 25, 2022

This was resolved with 1.0.13

@prha prha closed this as completed Oct 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant