Make GetTrainedModelsStatsAction cancellable #87931
Comments
|
Pinging @elastic/ml-core (Team:ML) |
|
Based on the Slack thread that went with this I think it's actually |
|
Hm. Sorry, I think I made a mistake translating what I saw in the thread dump to the REST action I mentioned in Slack. The threads in question were here: I'm not sure that this is due to Metricbeat now. Still, +1 on adding support for cancellation to all the things. |
|
OK, I saw But it looks like it really is @benwtrent as well as making the action cancellable it looks like we need a more efficient way to list which ingest pipelines a particular trained model is used in. It looks like it's instantiating all the processors at the moment, then looking for inference processors, taking the ID and throwing everything else away. But instantiating a Grok processor seems to be very expensive. So maybe we need a specialist method at a lower level that can search for inference processors in an ingest pipeline without instantiating every other processor. |
|
Arg, when I first wrote this code, I was extracting from opaque maps. Then I figured "Ya know, we could build the ingest pipelines and just extract from there and not iterate maps...". Welp, Time to go back and find that old code... 🤦 |
This was referenced Jun 23, 2022
benwtrent
added a commit
that referenced
this issue
Jun 24, 2022
This change makes all the trained model APIs cancellable, and addresses the handful of APIs that rely on our abstract resource structure. closes: #87931
benwtrent
added a commit
that referenced
this issue
Jun 28, 2022
Previous, get trained model stats API would build every pipeline defined in cluster state. This is problematic when MANY pipelines are defined. Especially if those pipelines take some time to parse (consider GROK). This improvement is part of fixing: #87931
benwtrent
added a commit
to benwtrent/elasticsearch
that referenced
this issue
Jun 28, 2022
Previous, get trained model stats API would build every pipeline defined in cluster state. This is problematic when MANY pipelines are defined. Especially if those pipelines take some time to parse (consider GROK). This improvement is part of fixing: elastic#87931
elasticsearchmachine
pushed a commit
that referenced
this issue
Jun 28, 2022
Previous, get trained model stats API would build every pipeline defined in cluster state. This is problematic when MANY pipelines are defined. Especially if those pipelines take some time to parse (consider GROK). This improvement is part of fixing: #87931
benwtrent
added a commit
to benwtrent/elasticsearch
that referenced
this issue
Jun 28, 2022
Previous, get trained model stats API would build every pipeline defined in cluster state. This is problematic when MANY pipelines are defined. Especially if those pipelines take some time to parse (consider GROK). This improvement is part of fixing: elastic#87931
elasticsearchmachine
pushed a commit
that referenced
this issue
Jun 28, 2022
Previous, get trained model stats API would build every pipeline defined in cluster state. This is problematic when MANY pipelines are defined. Especially if those pipelines take some time to parse (consider GROK). This improvement is part of fixing: #87931
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I encountered a Cloud cluster with an overworked master due (partly) to processing multiple calls to
GET /_ml/anomaly_detectors/_all/_statsoriginating from an external Metricbeat monitoring process. Metricbeat imposes a 10s timeout after which it closes the HTTP connection and tries again. However,GetTrainedModelsStatsActiondoes not notice if the client connection closes (i.e. the REST handler does not useRestCancellableNodeClientand the resulting transport task is not aCancellableTask) so it carries on wastefully processing the request even after the client timeout.Relates #55550
The text was updated successfully, but these errors were encountered: