Script: Deprecate script context cache

Deprecate the script context cache in favor of the general cache.

Users should use the following settings:
`script.max_compilations_rate` to set the max compilation rate
  for user scripts such as filter scripts.  Certain script contexts
  that submit scripts outside of the control of the user are
  exempted from this rate limit.  Examples include runtime fields,
  ingest and watcher.

`script.cache.max_size` to set the max size of the cache.

`script.cache.expire` to set the expiration time for entries in
the cache.

Whats deprecated?
`script.max_compilations_rate: use-context`.  This special
setting value was used to turn on the script context-specific caches.

`script.context.$CONTEXT.cache_max_size`, use `script.cache.max_size`
instead.

`script.context.$CONTEXT.cache_expire`, use `script.cache.expire`
instead.

`script.context.$CONTEXT.max_compilations_rate`, use
`script.max_compilations_rate` instead.

The default cache size was increased from `100` to `3000`, which
was approximately the max cache size when using context-specific caches.

The default compilation rate limit was increased from `75/5m` to
`150/5m` to account for increasing uses of scripts.

Refs: elastic#62899

docs/reference/migration/migrate_7_16.asciidoc

@@ -321,4 +321,77 @@ avoid deprecation warnings and opt into the future behaviour, include the query
 parameter `?return_200_for_cluster_health_timeout` in your request.
 ====
 
+[[script-context-cache-removed]]
+.The `script.max_compilations_rate` setting cannot be `use-context`.
+[%collapsible]
+====
+*Details* +
+The `script.max_compilations_rate` node setting cannot be `use-context`.
+Previously, you could set this setting to `use-context` to have a
+script cache, compilation rate limit settings, cache expiration and
+cache size per cluster.
+
+However, these settings were used to handle system scripts triggering
+compilation rate limits and to compensate for an undersized general script
+cache.
+Now, system scripts are exempt from compilation rate limiting and
+the general script cache size is dynamically with a higher default
+size.
+
+Do not use `use-context` as a value for this setting.  Instead, remove
+the setting to use the default or set it to a custom rate limit, such as
+`200/10m`, which allows 200 compilations for every 10 minute period.
+
+*Impact* +
+Discontinue use of the removed setting value. Specifying the setting value
+in `elasticsearch.yml` will result in an error on startup.
+====
+
+[[script-context-cache-max-compile-removed]]
+.The `script.context.$CONTEXT.max_compilations_rate` setting has been removed.
+[%collapsible]
+====
+*Details* +
+Script context-specific compilation rate limit settings have been removed.  Use
+`script.max_compilations_rate` to set the rate limit for all user scripts.
+
+The context-specific settings were used to handle system scripts triggering
+compilation rate limits and to compensate for an undersized general script
+cache.
+
+Now, system scripts are exempt from compilation rate limiting and
+the general script cache size is dynamically updatable with a higher default
+size.
+
+*Impact* +
+Discontinue use of the removed setting. Specifying the setting in
+`elasticsearch.yml` will result in an error on startup.
+====
+
+[[script-context-cache-size-removed]]
+.The `script.context.$CONTEXT.cache_max_size` setting has been removed.
+[%collapsible]
+====
+*Details* +
+Script context-specific cache settings have been removed.  Use
+`script.cache.max_size` to set the size of the script cache for all scripts.
+
+*Impact* +
+Discontinue use of the removed setting. Specifying the setting in
+`elasticsearch.yml` will result in an error on startup.
+====
+
+[[script-context-cache-expire-removed]]
+.The `script.context.$CONTEXT.cache_expire` setting has been removed.
+[%collapsible]
+====
+*Details* +
+Script context-specific cache settings have been removed.  Use
+`script.cache.expire` to set the expiration of scripts in the script cache
+for all scripts.
+
+*Impact* +
+Discontinue use of the removed setting. Specifying the setting in
+`elasticsearch.yml` will result in an error on startup.
+====
 // end::notable-breaking-changes[]

docs/reference/modules/indices/circuit_breaker.asciidoc

@@ -126,11 +126,11 @@ within a period of time.
 See the "prefer-parameters" section of the <<modules-scripting-using,scripting>>
 documentation for more information.
 
-`script.context.$CONTEXT.max_compilations_rate`::
+`script.max_compilations_rate`::
     (<<dynamic-cluster-setting,Dynamic>>)
     Limit for the number of unique dynamic scripts within a certain interval
-    that are allowed to be compiled for a given context. Defaults to `75/5m`,
-    meaning 75 every 5 minutes.
+    that are allowed to be compiled. Defaults to `150/5m`,
+    meaning 150 every 5 minutes.
 
 [[regex-circuit-breaker]]
 [discrete]

docs/reference/scripting/using.asciidoc

@@ -120,12 +120,8 @@ the `multiplier` parameter without {es} recompiling the script.
 }
 ----
 
-For most contexts, you can compile up to 75 scripts per 5 minutes by default.
-For ingest contexts, the default script compilation rate is unlimited. You
-can change these settings dynamically by setting
-`script.context.$CONTEXT.max_compilations_rate`. For example, the following
-setting limits script compilation to 100 scripts every 10 minutes for the
-{painless}/painless-field-context.html[field context]:
+You can compile up to 150 scripts per 5 minutes by default.
+For ingest contexts, the default script compilation rate is unlimited.
 
 [source,js]
 ----
@@ -406,8 +402,8 @@ small.
 
 All scripts are cached by default so that they only need to be recompiled
 when updates occur. By default, scripts do not have a time-based expiration.
-You can change this behavior by using the `script.context.$CONTEXT.cache_expire` setting.
-Use the `script.context.$CONTEXT.cache_max_size` setting to configure the size of the cache.
+You can change this behavior by using the `script.cache.expire` setting.
+Use the `script.max_size_in_bytes` setting to configure the size of the cache.
 
 NOTE: The size of scripts is limited to 65,535 bytes. Set the value of `script.max_size_in_bytes` to increase that soft limit. If your scripts are
 really large, then consider using a

server/src/main/java/org/elasticsearch/script/ScriptService.java

@@ -23,6 +23,8 @@
 import org.elasticsearch.cluster.metadata.Metadata;
 import org.elasticsearch.cluster.service.ClusterService;
 import org.elasticsearch.common.Strings;
+import org.elasticsearch.common.logging.DeprecationCategory;
+import org.elasticsearch.common.logging.DeprecationLogger;
 import org.elasticsearch.common.settings.ClusterSettings;
 import org.elasticsearch.common.settings.Setting;
 import org.elasticsearch.common.settings.Setting.Property;
@@ -50,6 +52,7 @@
 public class ScriptService implements Closeable, ClusterStateApplier, ScriptCompiler {
 
     private static final Logger logger = LogManager.getLogger(ScriptService.class);
+    private static final DeprecationLogger deprecationLogger = DeprecationLogger.getLogger(ScriptService.class);
 
     static final String DISABLE_DYNAMIC_SCRIPTING_SETTING = "script.disable_dynamic";
 
@@ -59,16 +62,22 @@ public class ScriptService implements Closeable, ClusterStateApplier, ScriptComp
     static final String USE_CONTEXT_RATE_KEY = "use-context";
 
     public static final Setting<Integer> SCRIPT_GENERAL_CACHE_SIZE_SETTING =
-        Setting.intSetting("script.cache.max_size", 100, 0, Property.NodeScope);
+        Setting.intSetting("script.cache.max_size", 3000, 0, Property.Dynamic, Property.NodeScope);
     public static final Setting<TimeValue> SCRIPT_GENERAL_CACHE_EXPIRE_SETTING =
-        Setting.positiveTimeSetting("script.cache.expire", TimeValue.timeValueMillis(0), Property.NodeScope);
+        Setting.positiveTimeSetting("script.cache.expire", TimeValue.timeValueMillis(0), Property.Dynamic, Property.NodeScope);
     public static final Setting<Integer> SCRIPT_MAX_SIZE_IN_BYTES =
         Setting.intSetting("script.max_size_in_bytes", 65535, 0, Property.Dynamic, Property.NodeScope);
     public static final Setting<ScriptCache.CompilationRate> SCRIPT_GENERAL_MAX_COMPILATIONS_RATE_SETTING =
-        new Setting<>("script.max_compilations_rate", USE_CONTEXT_RATE_KEY,
+        new Setting<>("script.max_compilations_rate", "150/5m",
             (String value) -> value.equals(USE_CONTEXT_RATE_KEY) ? USE_CONTEXT_RATE_VALUE: new ScriptCache.CompilationRate(value),
             Property.Dynamic, Property.NodeScope);
 
+    public static final String USE_CONTEXT_RATE_KEY_DEPRECATION_MESSAGE = "[" + USE_CONTEXT_RATE_KEY + "] is deprecated for the setting [" +
+        SCRIPT_GENERAL_MAX_COMPILATIONS_RATE_SETTING.getKey() + "] as system scripts are now exempt from the rate limit." +
+        "Set to a value such as [150/5m] (a rate of 150 compilations per five minutes) to rate limit user scripts in case the" +
+        "script cache [" + SCRIPT_GENERAL_CACHE_SIZE_SETTING.getKey() + "] is undersized causing script compilation thrashing.";
+
+
     // Per-context settings
     static final String CONTEXT_PREFIX = "script.context.";
 
@@ -77,13 +86,14 @@ public class ScriptService implements Closeable, ClusterStateApplier, ScriptComp
     public static final Setting.AffixSetting<Integer> SCRIPT_CACHE_SIZE_SETTING =
         Setting.affixKeySetting(CONTEXT_PREFIX,
             "cache_max_size",
-            key -> Setting.intSetting(key, SCRIPT_GENERAL_CACHE_SIZE_SETTING, 0, Property.NodeScope, Property.Dynamic));
+            key -> Setting.intSetting(key, SCRIPT_GENERAL_CACHE_SIZE_SETTING, 0,
+                                      Property.NodeScope, Property.Dynamic, Property.Deprecated));
 
     public static final Setting.AffixSetting<TimeValue> SCRIPT_CACHE_EXPIRE_SETTING =
         Setting.affixKeySetting(CONTEXT_PREFIX,
             "cache_expire",
             key -> Setting.positiveTimeSetting(key, SCRIPT_GENERAL_CACHE_EXPIRE_SETTING, TimeValue.timeValueMillis(0),
-                                               Property.NodeScope, Property.Dynamic));
+                                               Property.NodeScope, Property.Dynamic, Property.Deprecated));
 
     // Unlimited compilation rate for context-specific script caches
     static final String UNLIMITED_COMPILATION_RATE_KEY = "unlimited";
@@ -94,7 +104,7 @@ public class ScriptService implements Closeable, ClusterStateApplier, ScriptComp
             key -> new Setting<ScriptCache.CompilationRate>(key, "75/5m",
                 (String value) -> value.equals(UNLIMITED_COMPILATION_RATE_KEY) ? ScriptCache.UNLIMITED_COMPILATION_RATE:
                                                                                  new ScriptCache.CompilationRate(value),
-                Property.NodeScope, Property.Dynamic));
+                Property.NodeScope, Property.Dynamic, Property.Deprecated));
 
     private static final ScriptCache.CompilationRate SCRIPT_COMPILATION_RATE_ZERO = new ScriptCache.CompilationRate(0, TimeValue.ZERO);
 
@@ -205,6 +215,10 @@ public ScriptService(Settings settings, Map<String, ScriptEngine> engines, Map<S
         this.setCacheHolder(settings);
     }
 
+    public static boolean isUseContextCacheSet(Settings settings) {
+        return SCRIPT_GENERAL_MAX_COMPILATIONS_RATE_SETTING.get(settings).equals(USE_CONTEXT_RATE_VALUE);
+    }
+
     /**
      * This is overridden in tests to disable compilation rate limiting.
      */
@@ -249,6 +263,10 @@ void registerClusterSettingsListeners(ClusterSettings clusterSettings) {
      */
     void validateCacheSettings(Settings settings) {
         boolean useContext = SCRIPT_GENERAL_MAX_COMPILATIONS_RATE_SETTING.get(settings).equals(USE_CONTEXT_RATE_VALUE);
+        if (useContext) {
+            deprecationLogger.critical(DeprecationCategory.SCRIPTING, "scripting-context-cache",
+                    USE_CONTEXT_RATE_KEY_DEPRECATION_MESSAGE);
+        }
         List<Setting.AffixSetting<?>> affixes = Arrays.asList(SCRIPT_MAX_COMPILATIONS_RATE_SETTING, SCRIPT_CACHE_EXPIRE_SETTING,
                                                               SCRIPT_CACHE_SIZE_SETTING);
         List<String> customRates = new ArrayList<>();
@@ -277,7 +295,7 @@ void validateCacheSettings(Settings settings) {
                     String.join(", ", customRates) + "] if compile rates disabled via [" +
                     SCRIPT_DISABLE_MAX_COMPILATIONS_RATE_SETTING.getKey() + "]");
             }
-            if (useContext == false) {
+            if (useContext == false && SCRIPT_GENERAL_MAX_COMPILATIONS_RATE_SETTING.exists(settings)) {
                 throw new IllegalArgumentException("Cannot set custom general compilation rates [" +
                     SCRIPT_GENERAL_MAX_COMPILATIONS_RATE_SETTING.getKey() + "] to [" +
                     SCRIPT_GENERAL_MAX_COMPILATIONS_RATE_SETTING.get(settings) + "] if compile rates disabled via [" +
@@ -568,6 +586,7 @@ void setCacheHolder(Settings settings) {
         } else if (current.general.rate.equals(SCRIPT_GENERAL_MAX_COMPILATIONS_RATE_SETTING.get(settings)) == false ||
                    current.general.cacheExpire.equals(SCRIPT_GENERAL_CACHE_EXPIRE_SETTING.get(settings)) == false ||
                    current.general.cacheSize != SCRIPT_GENERAL_CACHE_SIZE_SETTING.get(settings)) {
+            // General compilation rate, cache expiration or cache size changed
             cacheHolder.set(generalCacheHolder(settings));
         }
     }