ingest: only compile templates if the value is templated #37120
Labels
:Data Management/Ingest Node
Execution or management of Ingest Pipelines including GeoIP
Comments
jakelandis
added
the
:Data Management/Ingest Node
|
Pinging @elastic/es-core-features |
This was referenced Jan 7, 2019
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
Jan 8, 2019
Prior to this change, any field in an ingest node processor that supports
script templates would be compiled as mustache template regardless if they
contain a template or not. Compiling normal text as mustache templates is
harmless. However, each compilation counts against the script compilation
circuit breaker. A large number of processors without any templates or scripts
could un-intuitively trip the too many script compilations circuit breaker.
This change simple checks for '{{' in the text before it attempts to compile.
fixes elastic#37120
jakelandis
added a commit
that referenced
this issue
Jan 9, 2019
* ingest: compile mustache template only if field includes '{{''
Prior to this change, any field in an ingest node processor that supports
script templates would be compiled as mustache template regardless if they
contain a template or not. Compiling normal text as mustache templates is
harmless. However, each compilation counts against the script compilation
circuit breaker. A large number of processors without any templates or scripts
could un-intuitively trip the too many script compilations circuit breaker.
This change simple checks for '{{' in the text before it attempts to compile.
fixes #37120
jakelandis
added a commit
that referenced
this issue
Jan 10, 2019
* ingest: compile mustache template only if field includes '{{''
Prior to this change, any field in an ingest node processor that supports
script templates would be compiled as mustache template regardless if they
contain a template or not. Compiling normal text as mustache templates is
harmless. However, each compilation counts against the script compilation
circuit breaker. A large number of processors without any templates or scripts
could un-intuitively trip the too many script compilations circuit breaker.
This change simple checks for '{{' in the text before it attempts to compile.
fixes #37120
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Many of the ingest node processors support template snippets (mustache). Each field that supports template snippets requires compilation the first time the pipeline is used.
Elasticsearch has circuit breaker such that no more than 75 scripts per 5 minutes may be compiled. Every script (not only ingest node scripts) is subject to this (configurable) limit. A moderate number of processors can trip this limit, even if no templated values are defined.
For example, a single set processor always counts as 2 compilations since the
fieldandvalueboth support template snippets. The compilation happens even if thefieldandvaluedo not define a{{ }}template.This issue is to address the unnecessary compilation for fields that support template snippets. The goal to reduce the number of compilations to only the minimum number actually required.
This issue is more likely on newer versions of Elasticsearch since in 6.4 additional fields were added to support template snippets and 6.5 the
ifcondition was added, which also requires a script complication (if defined).The text was updated successfully, but these errors were encountered: