Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide better compilation and cache stats to remove compiation rate limits #62899

Closed
3 tasks done
stu-elastic opened this issue Sep 24, 2020 · 2 comments
Closed
3 tasks done

Provide better compilation and cache stats to remove compiation rate limits #62899

stu-elastic opened this issue Sep 24, 2020 · 2 comments
Labels
>breaking :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache >enhancement Team:Core/Infra Meta label for core/infra team

Comments

@stu-elastic
Copy link
Contributor

stu-elastic commented Sep 24, 2020 •
edited
Loading

Ingest pipelines submit their scripts in ways that may be asynchronous to the user such as during startup or when adding new beats configs. Similarly, watcher scripts may need compilation at times that aren't clear to the user.

Breaking: We should remove all compilation rate limits and instead provide cache stats for users to correctly size their cache.

Progress:

PR for 8.0 for deprecation and system context rate limiting removal: #79453
@stu-elastic stu-elastic added >enhancement :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache labels Sep 24, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (:Core/Infra/Scripting)

@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Sep 24, 2020
@stu-elastic stu-elastic mentioned this issue Nov 4, 2020
Closed
@rjernst rjernst added the needs:triage Requires assignment of a team area label label Dec 3, 2020
@stu-elastic stu-elastic removed the needs:triage Requires assignment of a team area label label Dec 9, 2020
@rjernst rjernst mentioned this issue Jul 30, 2021
Merged
@masseyke masseyke mentioned this issue Aug 13, 2021
Closed
80 tasks
@stu-elastic stu-elastic changed the title Exempt compilations initiated by the system from compilation rate limits Provide better compilation and cache stats to remove compiation rate limits Sep 22, 2021
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Sep 23, 2021
@stu-elastic
Script: compile/cache eviction history metric placeholders
35c9264 
Adds 5m/15m/24h metrics to _nodes/stats when those metrics
are non-zero.  Those metrics are not yet populated.

BWC: history metrics are only sent between v8.0.0+ nodes,
v7.16.0 nodes will not send those metrics until they are populated.

Refs: elastic#62899
@stu-elastic stu-elastic mentioned this issue Sep 23, 2021
Merged
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Sep 23, 2021
@stu-elastic
Script: compile/cache eviction history metric placeholders
401ee41 
Adds 5m/15m/24h metrics to _nodes/stats when those metrics
are non-zero.  Those metrics are not yet populated.

BWC: history metrics are only sent between v8.0.0+ nodes,
v7.16.0 nodes will not send those metrics until they are populated.

Refs: elastic#62899
stu-elastic added a commit that referenced this issue Sep 27, 2021
@stu-elastic
Script: compile/cache eviction history metric placeholders (#78257)
db75c4b 
Adds 5m/15m/24h metrics to _nodes/stats when those metrics
are non-zero.  Those metrics are not yet populated.

BWC: history metrics are only sent between v8.0.0+ nodes,
v7.16.0 nodes will not send those metrics until they are populated.

Refs: #62899
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Sep 27, 2021
@stu-elastic
Script: compile/cache eviction history metric placeholders (elastic#7…
d3aa517 
…8257)

Adds 5m/15m/24h metrics to _nodes/stats when those metrics
are non-zero.  Those metrics are not yet populated.

BWC: history metrics are only sent between v8.0.0+ nodes,
v7.16.0 nodes will not send those metrics until they are populated.

Refs: elastic#62899
Backport: db75c4b
@stu-elastic stu-elastic mentioned this issue Sep 27, 2021
Merged
stu-elastic added a commit that referenced this issue Sep 27, 2021
@stu-elastic
Script: compile/cache eviction history metric placeholders (#78257) (#…
650b493 
…78333)

Adds 5m/15m/24h metrics to _nodes/stats when those metrics
are non-zero.  Those metrics are not yet populated.

BWC: history metrics are only sent between v8.0.0+ nodes,
v7.16.0 nodes will not send those metrics until they are populated.

Refs: #62899
Backport: db75c4b
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Oct 13, 2021
@stu-elastic
Script: Time series compile and cache evict metrics
a6c7a93 
Collects compilation and cache eviction metrics for
each script context.

Metrics are available in _nodes/stats in 5m/15m/1d
buckets.

Refs: elastic#62899
@stu-elastic stu-elastic mentioned this issue Oct 13, 2021
Merged
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Oct 18, 2021
@stu-elastic
Script: Undeprecate general cache
24e5e7f 
General cache settings are no longer deprecated for 8.0
* `script.cache.max_size`
* `script.cache.expire`
* `script.max_compilations_rate`

Refs: elastic#62899
@stu-elastic stu-elastic mentioned this issue Oct 18, 2021
Merged
stu-elastic added a commit that referenced this issue Oct 19, 2021
@stu-elastic
Script: Undeprecate general cache (#79402)
1df96c5 
* Script: Undeprecate general cache

General cache settings are no longer deprecated for 8.0
* `script.cache.max_size`
* `script.cache.expire`
* `script.max_compilations_rate`

Refs: #62899
@stu-elastic stu-elastic mentioned this issue Oct 19, 2021
Closed
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Oct 19, 2021
@stu-elastic
Script: Restore the scripting general cache
e709412 
Restore the scripting general cache in preparation for deprecation of
the context cache, then removal of the context cache in master.

Brings back the general cache settings:
* `script.max_compilations_rate`
* `script.cache.expire`
* `script.cache.max_size`

`script.cache.max_size` and `script.cache.expire` are now dynamic settings.

The context cache is still default, the switch to general cache as default will happen on context cache deprecation.

System script contexts can now opt-out of compilation rate limiting using a flag rather than a sentinel rate limit value.  Duplicated defaults for system contexts have been coalesced.

Other than that, this code is the same as what was in 7.x to make a `master`-first commit and backport strategy easy.

Refs: elastic#62899
@stu-elastic stu-elastic mentioned this issue Oct 19, 2021
Merged
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Oct 19, 2021
@stu-elastic
Script: Restore the scripting general cache
255af60 
Restore the scripting general cache in preparation for deprecation of
the context cache, then removal of the context cache in master.

Brings back the general cache settings:
* `script.max_compilations_rate`
* `script.cache.expire`
* `script.cache.max_size`

`script.cache.max_size` and `script.cache.expire` are now dynamic settings.

The context cache is still default, the switch to general cache as default will happen on context cache deprecation.

System script contexts can now opt-out of compilation rate limiting using a flag rather than a sentinel rate limit value.  Duplicated defaults for system contexts have been coalesced.

Other than that, this code is the same as what was in 7.x to make a `master`-first commit and backport strategy easy.

Refs: elastic#62899
Backport: elastic#79414
@stu-elastic stu-elastic mentioned this issue Oct 19, 2021
Merged
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Oct 19, 2021
@stu-elastic
Script: Restore the scripting general cache
02119d7 
Restore the scripting general cache in preparation for deprecation of
the context cache, then removal of the context cache in master.

Brings back the general cache settings:
* `script.max_compilations_rate`
* `script.cache.expire`
* `script.cache.max_size`

`script.cache.max_size` and `script.cache.expire` are now dynamic settings.

The context cache is still default, the switch to general cache as default will happen on context cache deprecation.

System script contexts can now opt-out of compilation rate limiting using a flag rather than a sentinel rate limit value.  Duplicated defaults for system contexts have been coalesced.

Other than that, this code is the same as what was in 7.x to make a `master`-first commit and backport strategy easy.

Refs: elastic#62899
Backport: elastic#79414
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Oct 19, 2021
@stu-elastic
Script: Deprecate script context cache
f82ce90 
Deprecate the script context cache in favor of the general cache.

Users should use the following settings:
`script.max_compilations_rate` to set the max compilation rate
  for user scripts such as filter scripts.  Certain script contexts
  that submit scripts outside of the control of the user are
  exempted from this rate limit.  Examples include runtime fields,
  ingest and watcher.

`script.cache.max_size` to set the max size of the cache.

`script.cache.expire` to set the expiration time for entries in
the cache.

Whats deprecated?
`script.max_compilations_rate: use-context`.  This special
setting value was used to turn on the script context-specific caches.

`script.context.$CONTEXT.cache_max_size`, use `script.cache.max_size`
instead.

`script.context.$CONTEXT.cache_expire`, use `script.cache.expire`
instead.

`script.context.$CONTEXT.max_compilations_rate`, use
`script.max_compilations_rate` instead.

The default cache size was increased from `100` to `3000`, which
was approximately the max cache size when using context-specific caches.

The default compilation rate limit was increased from `75/5m` to
`150/5m` to account for increasing uses of scripts.

Refs: elastic#62899
@stu-elastic stu-elastic mentioned this issue Oct 19, 2021
Closed
stu-elastic added a commit that referenced this issue Oct 19, 2021
@stu-elastic
Script: Opt-out system contexts from script compilation rate limit (#…
8601c9b 
…79459)

System script contexts can now opt-out of compilation rate limiting using a flag rather than a sentinel rate limit value.

Duplicated defaults for system contexts have been coalesced.

Refs: #62899
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Oct 19, 2021
@stu-elastic
Script: Deprecate script context cache
08bd07a 
Deprecate the script context cache in favor of the general cache.

Users should use the following settings:
`script.max_compilations_rate` to set the max compilation rate
  for user scripts such as filter scripts.  Certain script contexts
  that submit scripts outside of the control of the user are
  exempted from this rate limit.  Examples include runtime fields,
  ingest and watcher.

`script.cache.max_size` to set the max size of the cache.

`script.cache.expire` to set the expiration time for entries in
the cache.

Whats deprecated?
`script.max_compilations_rate: use-context`.  This special
setting value was used to turn on the script context-specific caches.

`script.context.$CONTEXT.cache_max_size`, use `script.cache.max_size`
instead.

`script.context.$CONTEXT.cache_expire`, use `script.cache.expire`
instead.

`script.context.$CONTEXT.max_compilations_rate`, use
`script.max_compilations_rate` instead.

The default cache size was increased from `100` to `3000`, which
was approximately the max cache size when using context-specific caches.

The default compilation rate limit was increased from `75/5m` to
`150/5m` to account for increasing uses of scripts.

Refs: elastic#62899
@stu-elastic stu-elastic mentioned this issue Oct 19, 2021
Merged
stu-elastic added a commit that referenced this issue Oct 20, 2021
@stu-elastic
Script: Deprecate script context cache (#79508)
005d8f0 
* Script: Deprecate script context cache

Deprecate the script context cache in favor of the general cache.

Users should use the following settings:
`script.max_compilations_rate` to set the max compilation rate
  for user scripts such as filter scripts.  Certain script contexts
  that submit scripts outside of the control of the user are
  exempted from this rate limit.  Examples include runtime fields,
  ingest and watcher.

`script.cache.max_size` to set the max size of the cache.

`script.cache.expire` to set the expiration time for entries in
the cache.

Whats deprecated?
`script.max_compilations_rate: use-context`.  This special
setting value was used to turn on the script context-specific caches.

`script.context.$CONTEXT.cache_max_size`, use `script.cache.max_size`
instead.

`script.context.$CONTEXT.cache_expire`, use `script.cache.expire`
instead.

`script.context.$CONTEXT.max_compilations_rate`, use
`script.max_compilations_rate` instead.

The default cache size was increased from `100` to `3000`, which
was approximately the max cache size when using context-specific caches.

The default compilation rate limit was increased from `75/5m` to
`150/5m` to account for increasing uses of scripts.

Refs: #62899
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Oct 20, 2021
@stu-elastic
Script: Restore the scripting general cache
a426428 
Deprecate the script context cache in favor of the general cache.

Users should use the following settings:
`script.max_compilations_rate` to set the max compilation rate
  for user scripts such as filter scripts.  Certain script contexts
  that submit scripts outside of the control of the user are
  exempted from this rate limit.  Examples include runtime fields,
  ingest and watcher.

`script.cache.max_size` to set the max size of the cache.

`script.cache.expire` to set the expiration time for entries in
the cache.

Whats deprecated?
`script.max_compilations_rate: use-context`.  This special
setting value was used to turn on the script context-specific caches.

`script.context.$CONTEXT.cache_max_size`, use `script.cache.max_size`
instead.

`script.context.$CONTEXT.cache_expire`, use `script.cache.expire`
instead.

`script.context.$CONTEXT.max_compilations_rate`, use
`script.max_compilations_rate` instead.

The default cache size was increased from `100` to `3000`, which
was approximately the max cache size when using context-specific caches.

The default compilation rate limit was increased from `75/5m` to
`150/5m` to account for increasing uses of scripts.

System script contexts can now opt-out of compilation rate limiting
using a flag rather than a sentinel rate limit value.

7.16: Script: Deprecate script context cache elastic#79508
Refs: elastic#62899

7.16: Script: Opt-out system contexts from script compilation rate limit elastic#79459
Refs: elastic#62899
stu-elastic added a commit to stu-elastic/elasticsearch that referenced this issue Oct 20, 2021
@stu-elastic
Script: Restore the scripting general cache
a9aa5d8 
Deprecate the script context cache in favor of the general cache.

Users should use the following settings:
`script.max_compilations_rate` to set the max compilation rate
  for user scripts such as filter scripts.  Certain script contexts
  that submit scripts outside of the control of the user are
  exempted from this rate limit.  Examples include runtime fields,
  ingest and watcher.

`script.cache.max_size` to set the max size of the cache.

`script.cache.expire` to set the expiration time for entries in
the cache.

Whats deprecated?
`script.max_compilations_rate: use-context`.  This special
setting value was used to turn on the script context-specific caches.

`script.context.$CONTEXT.cache_max_size`, use `script.cache.max_size`
instead.

`script.context.$CONTEXT.cache_expire`, use `script.cache.expire`
instead.

`script.context.$CONTEXT.max_compilations_rate`, use
`script.max_compilations_rate` instead.

The default cache size was increased from `100` to `3000`, which
was approximately the max cache size when using context-specific caches.

The default compilation rate limit was increased from `75/5m` to
`150/5m` to account for increasing uses of scripts.

System script contexts can now opt-out of compilation rate limiting
using a flag rather than a sentinel rate limit value.

7.16: Script: Deprecate script context cache elastic#79508
Refs: elastic#62899

7.16: Script: Opt-out system contexts from script compilation rate limit elastic#79459
Refs: elastic#62899
stu-elastic added a commit that referenced this issue Oct 21, 2021
@stu-elastic
Script: Restore the scripting general cache (#79453)
808b70d 
Deprecate the script context cache in favor of the general cache.

Users should use the following settings:
`script.max_compilations_rate` to set the max compilation rate
  for user scripts such as filter scripts.  Certain script contexts
  that submit scripts outside of the control of the user are
  exempted from this rate limit.  Examples include runtime fields,
  ingest and watcher.

`script.cache.max_size` to set the max size of the cache.

`script.cache.expire` to set the expiration time for entries in
the cache.

Whats deprecated?
`script.max_compilations_rate: use-context`.  This special
setting value was used to turn on the script context-specific caches.

`script.context.$CONTEXT.cache_max_size`, use `script.cache.max_size`
instead.

`script.context.$CONTEXT.cache_expire`, use `script.cache.expire`
instead.

`script.context.$CONTEXT.max_compilations_rate`, use
`script.max_compilations_rate` instead.

The default cache size was increased from `100` to `3000`, which
was approximately the max cache size when using context-specific caches.

The default compilation rate limit was increased from `75/5m` to
`150/5m` to account for increasing uses of scripts.

System script contexts can now opt-out of compilation rate limiting
using a flag rather than a sentinel rate limit value.

7.16: Script: Deprecate script context cache #79508
Refs: #62899

7.16: Script: Opt-out system contexts from script compilation rate limit #79459
Refs: #62899
stu-elastic added a commit that referenced this issue Nov 3, 2021
@stu-elastic
Script: Time series compile and cache evict metrics (#79078)
30e15ba 
Collects compilation and cache eviction metrics for
each script context.

Metrics are available in _nodes/stats in 5m/15m/1d
buckets.

Refs: #62899
@stu-elastic
Copy link
Contributor Author

We'll wait for feedback on the metrics to see if compilation rate limits can be removed in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>breaking :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache >enhancement Team:Core/Infra Meta label for core/infra team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rjernst @elasticmachine @stu-elastic