Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ResponseOps][Task Manager][mget Claimer] fix problem with limited concurrency starvation of unlimited concurrency tasks #184937

Closed
pmuellr opened this issue Jun 6, 2024 · 1 comment · Fixed by #187809
Assignees
Labels
Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.16.0

Comments

@pmuellr
Copy link
Member

pmuellr commented Jun 6, 2024

In PR implement task claiming strategy mget #180485 we implemented an alternative task claiming strategy, but it has the following problem:

If the number of limited concurrency tasks to run next exceeds the count of the tasks we do the initial search for, no tasks with unlimited concurrency will be returned for this polling cycle, as they would be "starved" by the limited concurrency tasks.

This is because we are searching all task types in one search, then applying limited concurrency on the results. We need to change this to do multiple searches - the unlimited concurrency search should not search on any limited concurrency tasks, and vice versa. Presumably we'd be doing one search for unlimited concurrency, and an additional search for each task type with limited concurrency.

I think we can do this with an _msearch. I'm thinking we put all the common matching clauses we need into the top-level filter of the query, in hopes that they make use of the potential filter caching. Then the query can provide the specific matching clauses for that sub-search.

@pmuellr pmuellr added Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Jun 6, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.16.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants