[ML] Throttle the delete-by-query of expired results #47177
Conversation
Due to elastic#47003 many clusters will have built up a large backlog of expired results. On upgrading to a version where that bug is fixed users could find that the first ML daily maintenance task deletes a very large amount of documents. This change introduces throttling to the delete-by-query that the ML daily maintenance uses to delete expired results: - Average 200 documents per second - Maximum of 10 million documents per day (There is no throttling for state/forecast documents as these are expected to be lower volume.) Relates elastic#47103
|
Pinging @elastic/ml-core |
|
run elasticsearch-ci/default-distro |
|
run elasticsearch-ci/packaging-sample |
| // Delete the documents gradually. | ||
| // With DEFAULT_SCROLL_SIZE = 1000 this implies we spread deletion of 1 million documents over 5000 seconds ~= 83 minutes. | ||
| // And we delete a maximum of 10000 batches per day (= 10 million documents per day if DEFAULT_SCROLL_SIZE = 1000). | ||
| // If more documents than this have expired then some will be deleted on a subsequent day. |
There was a problem hiding this comment.
this is executed per job, so the throttling applies only to 1 job. What if you have 1000's of jobs? Do we need more protection 1 level up (maybe based on time)?
There was a problem hiding this comment.
another observation (not sure if it makes a difference): The query does not disable scoring, does it make sense to wrap it as constant_score? This might be more delete friendly.
There was a problem hiding this comment.
We do not start the next job's deletion before the previous is complete.
There was a problem hiding this comment.
We do not start the next job's deletion before the previous is complete
I think the requests per second limit is still valuable, but with a separate scroll per job a 10 million docs limit is pretty useless. I will instead impose a time limit, say 8 hours, at the outermost level of the expired data removal.
There was a problem hiding this comment.
The query does not disable scoring, does it make sense to wrap it as constant_score?
I think sorting by _doc disables scoring. This is also the simplest order to retrieve the documents from Lucene. So I will add a sort by _doc to all our delete-by-query searches related to expired data and it should give both benefits.
Some more tests are still required, but these changes are a start
Due to #47003 many clusters will have built up a large backlog of expired results. On upgrading to a version where that bug is fixed users could find that the first ML daily maintenance task deletes a very large amount of documents. This change introduces throttling to the delete-by-query that the ML daily maintenance uses to delete expired results to limit it to deleting an average 200 documents per second. (There is no throttling for state/forecast documents as these are expected to be lower volume.) Additionally a rough time limit of 8 hours is applied to the whole delete expired data action. (This is only rough as it won't stop part way through a single operation - it only checks the timeout between operations.) Relates #47103
Due to #47003 many clusters will have built up a large backlog of expired results. On upgrading to a version where that bug is fixed users could find that the first ML daily maintenance task deletes a very large amount of documents. This change introduces throttling to the delete-by-query that the ML daily maintenance uses to delete expired results to limit it to deleting an average 200 documents per second. (There is no throttling for state/forecast documents as these are expected to be lower volume.) Additionally a rough time limit of 8 hours is applied to the whole delete expired data action. (This is only rough as it won't stop part way through a single operation - it only checks the timeout between operations.) Relates #47103
Due to #47003 many clusters will have built up a large backlog of expired results. On upgrading to a version where that bug is fixed users could find that the first ML daily maintenance task deletes a very large amount of documents. This change introduces throttling to the delete-by-query that the ML daily maintenance uses to delete expired results to limit it to deleting an average 200 documents per second. (There is no throttling for state/forecast documents as these are expected to be lower volume.) Additionally a rough time limit of 8 hours is applied to the whole delete expired data action. (This is only rough as it won't stop part way through a single operation - it only checks the timeout between operations.) Relates #47103
Due to #47003 many clusters will have built up a
large backlog of expired results. On upgrading to
a version where that bug is fixed users could find
that the first ML daily maintenance task deletes
a very large amount of documents.
This change introduces throttling to the
delete-by-query that the ML daily maintenance uses
to delete expired results to limit it to deleting an
average 200 documents per second. (There is no
throttling for state/forecast documents as these
are expected to be lower volume.)
Additionally a rough time limit of 8 hours is applied
to the whole delete expired data action. (This is only
rough as it won't stop part way through a single
operation - it only checks the timeout between
operations.)
Relates #47103