[ML] allow for larger models in the inference step for data frame analytics #76116
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/ml-core (Team:ML) |
davidkyle
reviewed
Aug 9, 2021
| TrainedModelConfig config = loadModelFromResource(modelId, false).build().ensureParsedDefinition(xContentRegistry); | ||
| TrainedModelConfig config = loadModelFromResource(modelId, false) | ||
| .build() | ||
| .ensureParsedDefinitionUnsafe(xContentRegistry); |
There was a problem hiding this comment.
This should respect the value of the unsafe parameter the same as line 432
There was a problem hiding this comment.
@davidkyle I don't think it should. Models in the resourceFiles are provided in the jar distribution. So, I don't think we should ever check their stream length on parsing.
There was a problem hiding this comment.
So, I don't think we should ever check their stream length on parsing.
Because we know how big they are? Why not enforce that with a simple check.
There was a problem hiding this comment.
It is superfluous to me as the only way to adjust this resource is to modify the resource files directly on disk and since we control these resource models, we already know and trust their sizes.
benwtrent
added
the
auto-backport
💔 Backport failed
To backport manually run: |
benwtrent
added a commit
to benwtrent/elasticsearch
that referenced
this pull request
Aug 9, 2021
…lytics (elastic#76116) When a user creates a Data frame analytics model, it is possible that the inference step fails due to he model being too large to fit in the JVM. Example error messages: ``` [foo] failed running inference on model [foo-1628085713000]; cause was [Data too large, data for [foo-1628085713000] would be [...], which is larger than the limit of [...]] ``` ``` [foo] failed running inference on model [foo-1628085713000]; cause was [Cannot parse model definition as the content is larger than the maximum stream size of [...] bytes. Max stream size is 10% of the JVM heap or 1GB whichever is smallest] ``` This commit partially addresses these error by allowing the circuit breaker to handle the OOM prevention. Since the model was recently created by an internal process, this is acceptable. relates to elastic#76093
elasticsearchmachine
pushed a commit
that referenced
this pull request
Aug 9, 2021
…lytics (#76116) (#76256) When a user creates a Data frame analytics model, it is possible that the inference step fails due to he model being too large to fit in the JVM. Example error messages: ``` [foo] failed running inference on model [foo-1628085713000]; cause was [Data too large, data for [foo-1628085713000] would be [...], which is larger than the limit of [...]] ``` ``` [foo] failed running inference on model [foo-1628085713000]; cause was [Cannot parse model definition as the content is larger than the maximum stream size of [...] bytes. Max stream size is 10% of the JVM heap or 1GB whichever is smallest] ``` This commit partially addresses these error by allowing the circuit breaker to handle the OOM prevention. Since the model was recently created by an internal process, this is acceptable. relates to #76093
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a user creates a Data frame analytics model, it is possible that the inference step fails due to he model being too large to fit in the JVM.
Example error messages:
This commit partially addresses these error by allowing the circuit breaker to handle the OOM prevention. Since the model was recently created by an internal process, this is acceptable.
relates to #76093