Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Remove model snapshot legacy doc ids #62434

Merged
merged 4 commits into from
Sep 17, 2020
Merged

[ML] Remove model snapshot legacy doc ids #62434

merged 4 commits into from
Sep 17, 2020

Conversation

dimitris-athanasiou
Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou commented Sep 16, 2020 •
edited
Loading

Removes methods that were no longer used regarding version 5.4 doc ids of ModelState.

Also adds clean up of 5.4 model state and quantile docs in the daily maintenance.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

@droberts195
Copy link
Contributor

We have never done anything to force users to delete jobs they created in 5.4, so I think documents with these IDs could still exist in the index if users still have old jobs left over from 5.4. Having said that, we don't let people open 5.4 jobs, so there isn't any need to keep the state. We could just unconditionally delete state documents for 5.4 jobs in our nightly maintenance.

@droberts195
Copy link
Contributor

Just looking at this again, I see the method literally wasn't used. That means we've had a bug for a long time that if someone deletes a 5.4 job then we don't delete its state documents. So there will be users who have these orphaned 5.4 state documents in their state indices. We could clean up current and future cases by deleting all 5.4 state documents during the nightly maintenance.

@dimitris-athanasiou
Copy link
Contributor Author

@elasticmachine update branch

@dimitris-athanasiou
Copy link
Contributor Author

@elasticmachine update branch

droberts195
droberts195 approved these changes Sep 17, 2020
View reviewed changes
Copy link
Contributor

@droberts195 droberts195 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dimitris-athanasiou dimitris-athanasiou merged commit 5ee2d70 into elastic:master Sep 17, 2020
@dimitris-athanasiou dimitris-athanasiou deleted the remove-model-snapshot-legacy-doc-ids branch September 17, 2020 16:24
dimitris-athanasiou added a commit to dimitris-athanasiou/elasticsearch that referenced this pull request Sep 17, 2020
@dimitris-athanasiou
[7.x][ML] Remove model snapshot legacy doc ids (elastic#62434)
f248e82 
Removes methods that were no longer used regarding version 5.4 doc ids of ModelState.

Also adds clean up of 5.4 model state and quantile docs in the daily maintenance.

Backport of elastic#62434
@dimitris-athanasiou dimitris-athanasiou mentioned this pull request Sep 17, 2020
Merged
dimitris-athanasiou added a commit that referenced this pull request Sep 17, 2020
@dimitris-athanasiou
[7.x][ML] Remove model snapshot legacy doc ids (#62434) (#62569)
7118ff7 
Removes methods that were no longer used regarding version 5.4 doc ids of ModelState.

Also adds clean up of 5.4 model state and quantile docs in the daily maintenance.

Backport of #62434
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@droberts195 droberts195 droberts195 approved these changes

Assignees
No one assigned
Labels
:ml Machine learning >refactoring v7.10.0 v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dimitris-athanasiou @elasticmachine @droberts195 @jakelandis