Deprecate aggregation ValueScripts in favor of Runtime Fields

[not-napoleon]

Value scripts provide a tool for modifying field values at query time. They can be used for things like applying a correction or normalization to values, or changing the types of a field (e.g. parsing a number out of a string field). They also add a lot of complexity to the type system aggregations use (ValuesSource), and their behavior with missing values is not obvious (e.g. if you specify a value script and a missing value, is the script applied to the missing value?). There are also some tricky issues around value scripts (e.g. #53135)

Enter Runtime Fields. By and large, runtime fields solve the same problem - dynamically modifying a field value at query time. Runtime fields are well integrated into the mapping framework and have much better test coverage than value scripts. They're more centrally documented and present a clear, global abstraction as opposed to an implementation detail of an aggregation.

This issue is to discuss if, how, and when we should drop support for value scripts in favor of runtime fields.

Note: this is only referring to value scripts (e.g. https://www.elastic.co/guide/en/elasticsearch/reference/7.11/search-aggregations-metrics-min-aggregation.html#_value_script_6) not "regular" scripts. Value scripts are intended to operate on a single field, while the more general case operates on a whole document.

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[javanna]

Thanks @not-napoleon I can think of a couple more places where scripts could be replaced in the search API in favour of runtime fields. Maybe we should discuss these altogether and come up with a deprecation plan for them all.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[nik9000]

Here are a few places where we document script parameters that aren't runtime fields. grep ain't great for this, but its what I have at hand.

$ find docs/reference/ -type f | grep -v ingest | grep -v reindex | grep -v /update | grep -v /bulk | grep -v predicate-tokenfilter | grep -v /runtime | xargs grep -n \"script\" | sort | sed 's/^/* [ ] /'

• docs/reference/aggregations.asciidoc:343: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/composite-aggregation.asciidoc:159: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/composite-aggregation.asciidoc:215: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/datehistogram-aggregation.asciidoc:686: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/diversified-sampler-aggregation.asciidoc:111: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/multi-terms-aggregation.asciidoc:181: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/range-aggregation.asciidoc:187: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/range-aggregation.asciidoc:208: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/range-aggregation.asciidoc:225: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/range-aggregation.asciidoc:285: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/significantterms-aggregation.asciidoc:422: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/bucket/terms-aggregation.asciidoc:422: Convert bucket aggs docs to runtime fields #71202
• docs/reference/aggregations/metrics/avg-aggregation.asciidoc:51: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/avg-aggregation.asciidoc:70: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/avg-aggregation.asciidoc:95: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/boxplot-aggregation.asciidoc:120: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/boxplot-aggregation.asciidoc:91: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/cardinality-aggregation.asciidoc:194: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/cardinality-aggregation.asciidoc:214: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/extendedstats-aggregation.asciidoc:113: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/extendedstats-aggregation.asciidoc:134: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/extendedstats-aggregation.asciidoc:160: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/max-aggregation.asciidoc:108: "script" : { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/max-aggregation.asciidoc:60: "script" : { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/max-aggregation.asciidoc:80: "script" : { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/median-absolute-deviation-aggregation.asciidoc:126: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/median-absolute-deviation-aggregation.asciidoc:150: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/min-aggregation.asciidoc:109: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/min-aggregation.asciidoc:61: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/min-aggregation.asciidoc:81: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/percentile-aggregation.asciidoc:179: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/percentile-aggregation.asciidoc:207: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/percentile-rank-aggregation.asciidoc:136: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/percentile-rank-aggregation.asciidoc:165: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/rate-aggregation.asciidoc:288: "script": { <1> Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/stats-aggregation.asciidoc:103: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/stats-aggregation.asciidoc:58: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/stats-aggregation.asciidoc:78: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/string-stats-aggregation.asciidoc:142: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/string-stats-aggregation.asciidoc:163: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/string-stats-aggregation.asciidoc:188: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/sum-aggregation.asciidoc:126: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/sum-aggregation.asciidoc:67: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/sum-aggregation.asciidoc:93: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/tophits-aggregation.asciidoc:223: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/t-test-aggregation.asciidoc:157: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/valuecount-aggregation.asciidoc:55: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/valuecount-aggregation.asciidoc:74: "script": { Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/weighted-avg-aggregation.asciidoc:162: "script": "doc.grade.value + 1" Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/metrics/weighted-avg-aggregation.asciidoc:165: "script": "doc.weight.value + 1" Convert metric aggs docs runtime fields #71260
• docs/reference/aggregations/pipeline.asciidoc:147: "script": "params.hats / params.bags"
• docs/reference/aggregations/pipeline.asciidoc:215: "script": {
• docs/reference/aggregations/pipeline/bucket-script-aggregation.asciidoc:23: "script": "params.my_var1 / params.my_var2"
• docs/reference/aggregations/pipeline/bucket-script-aggregation.asciidoc:84: "script": "params.tShirtSales / params.totalSales * 100"
• docs/reference/aggregations/pipeline/bucket-selector-aggregation.asciidoc:27: "script": "params.my_var1 > params.my_var2"
• docs/reference/aggregations/pipeline/bucket-selector-aggregation.asciidoc:72: "script": "params.totalSales > 200"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:162: "script": "return values.length > 0 ? values[0] : Double.NaN"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:21: "script": "MovingFunctions.min(values)"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:232: "script": "MovingFunctions.max(values)"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:274: "script": "MovingFunctions.min(values)"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:316: "script": "MovingFunctions.sum(values)"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:360: "script": "MovingFunctions.stdDev(values, MovingFunctions.unweightedAvg(values))"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:411: "script": "MovingFunctions.unweightedAvg(values)"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:456: "script": "MovingFunctions.linearWeightedAvg(values)"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:507: "script": "MovingFunctions.ewma(values, 0.3)"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:564: "script": "MovingFunctions.holt(values, 0.3, 0.1)"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:61: "script": "MovingFunctions.unweightedAvg(values)"
• docs/reference/aggregations/pipeline/movfn-aggregation.asciidoc:629: "script": "if (values.length > 5*2) {MovingFunctions.holtWinters(values, 0.3, 0.1, 0.1, 5, false)}"
• docs/reference/analysis/tokenfilters/condition-tokenfilter.asciidoc:140: "script": {
• docs/reference/analysis/tokenfilters/condition-tokenfilter.asciidoc:30: "script": {
• docs/reference/data-streams/use-a-data-stream.asciidoc:217: "script": { (_update_by_query)
• docs/reference/how-to/recipes/scoring.asciidoc:131: "script": { (script_score query)
• docs/reference/index-modules/similarity.asciidoc:200: "script": { (scripted similarity)
• docs/reference/index-modules/similarity.asciidoc:368: "script": { (scripted similarity)
• docs/reference/indices/index-templates.asciidoc:104: "script": { (is a runtime field)
• docs/reference/mapping/fields/index-field.asciidoc:46: "script": { Convert _index field example to runtime fields #71304
• docs/reference/mapping/types/boolean.asciidoc:80: "script": { Convert boolean field example to runtime fields #71341
• docs/reference/mapping/types/date_nanos.asciidoc:56: "script" : { Convert date_nanos example script to runtime field #71351
• docs/reference/mapping/types/parent-join.asciidoc:291: "script": { Convert parent-join example script to runtime field #71423
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:135: "script": { @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:243: "script": { @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:280: "script": { @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:316: "script": { @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:353: "script": { @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:390: "script": { @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:426: "script": { @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:491: "script":"return domainSplit(doc['query'].value).get(0);" @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:494: "script":"return domainSplit(doc['query'].value).get(1);" @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/anomaly-detection/ml-configuring-transform.asciidoc:558: "script": { @szabosteve [DOCS] Alters examples in anomaly detection page to use runtime mappings #71745
• docs/reference/ml/ml-shared.asciidoc:898: "script": { (already is a runtime field)
• docs/reference/query-dsl/function-score-query.asciidoc:146: "script": { (can't be a runtime field because it needs score)
• docs/reference/query-dsl/function-score-query.asciidoc:184: "script": { (can't be a runtime field because it needs score)
• docs/reference/query-dsl/has-child-query.asciidoc:150: "script": "_score * doc['click_count'].value" (script_score)
• docs/reference/query-dsl/has-parent-query.asciidoc:132: "script": "_score * doc['view_count'].value" (script_score)
• docs/reference/query-dsl/intervals-query.asciidoc:352: "script" : { (filters intervals)
• docs/reference/query-dsl/script-query.asciidoc:24: "script": { Point script query docs at runtime fields #71291 (documents runtime fields as an alternative)
• docs/reference/query-dsl/script-query.asciidoc:25: "script": { Point script query docs at runtime fields #71291 (documents runtime fields as an alternative)
• docs/reference/query-dsl/script-query.asciidoc:61: "script": { Point script query docs at runtime fields #71291 (documents runtime fields as an alternative)
• docs/reference/query-dsl/script-query.asciidoc:62: "script": { Point script query docs at runtime fields #71291 (documents runtime fields as an alternative)
• docs/reference/query-dsl/script-score-query.asciidoc:104:"script" : {
• docs/reference/query-dsl/script-score-query.asciidoc:122:"script" : {
• docs/reference/query-dsl/script-score-query.asciidoc:135:"script" : {
• docs/reference/query-dsl/script-score-query.asciidoc:160:"script" : {
• docs/reference/query-dsl/script-score-query.asciidoc:184:"script" : {
• docs/reference/query-dsl/script-score-query.asciidoc:207:"script" : {
• docs/reference/query-dsl/script-score-query.asciidoc:268:"script" : {
• docs/reference/query-dsl/script-score-query.asciidoc:26: "script": {
• docs/reference/query-dsl/script-score-query.asciidoc:289:"script" : {
• docs/reference/query-dsl/script-score-query.asciidoc:305:"script" : {
• docs/reference/query-dsl/script-score-query.asciidoc:353: "script": {
• docs/reference/query-dsl/script-score-query.asciidoc:92:"script" : {
• docs/reference/scripting/engine.asciidoc:41: "script": { (script_score)
• docs/reference/scripting/fields.asciidoc:102: "script": {
• docs/reference/scripting/fields.asciidoc:194: "script": {
• docs/reference/scripting/fields.asciidoc:239: "script": {
• docs/reference/scripting/fields.asciidoc:71: "script": {
• docs/reference/scripting/using.asciidoc:10: "script": {
• docs/reference/scripting/using.asciidoc:150: "script": {
• docs/reference/scripting/using.asciidoc:172: "script": {
• docs/reference/scripting/using.asciidoc:219: "script": {
• docs/reference/scripting/using.asciidoc:249: "script": {
• docs/reference/scripting/using.asciidoc:297: "script" : {
• docs/reference/scripting/using.asciidoc:315: "script": {
• docs/reference/scripting/using.asciidoc:336: "script": {
• docs/reference/scripting/using.asciidoc:354: "script" : "ctx._source.new_field = 'value_of_new_field'"
• docs/reference/scripting/using.asciidoc:365: "script" : "ctx._source.remove('new_field')"
• docs/reference/scripting/using.asciidoc:378: "script": {
• docs/reference/scripting/using.asciidoc:448: "script": {
• docs/reference/scripting/using.asciidoc:490: "script": {
• docs/reference/scripting/using.asciidoc:70: "script": { <1>
• docs/reference/search/multi-search.asciidoc:362: "script": { (search template)
• docs/reference/search/multi-search.asciidoc:381: "script": { (search template)
• docs/reference/search/search.asciidoc:457:"script": { Fit runtime field on line #71470
• docs/reference/search/search-template.asciidoc:124: "script": { (search template)
• docs/reference/search/search-template.asciidoc:166: "script" : { (search template)
• docs/reference/search/search-your-data/filter-search-results.asciidoc:276: "script": { (script_score)
• docs/reference/search/search-your-data/retrieve-selected-fields.asciidoc:644: "script": { [DOCS] Reorganize "Retrieve selected fields" how-to for fields API #70112
• docs/reference/search/search-your-data/retrieve-selected-fields.asciidoc:650: "script": { [DOCS] Reorganize "Retrieve selected fields" how-to for fields API #70112
• docs/reference/search/search-your-data/retrieve-selected-fields.asciidoc:680: "script" : "params['_source']['message']" [DOCS] Reorganize "Retrieve selected fields" how-to for fields API #70112
• docs/reference/search/search-your-data/search-your-data.asciidoc:123: "script": {
• docs/reference/search/search-your-data/sort-search-results.asciidoc:608: "script": {
• docs/reference/transform/examples.asciidoc:147: "script": "(params.delay_time / params.flight_time) * 100" (agg bucket selector script)
• docs/reference/transform/examples.asciidoc:249: "script": "(params.max_time - params.min_time)" (agg bucket selector script)
• docs/reference/transform/painless-examples.asciidoc:124: "script": { <2>
• docs/reference/transform/painless-examples.asciidoc:134: "script": { <6>
• docs/reference/transform/painless-examples.asciidoc:178: "script": { <2> (bucket script)
• docs/reference/transform/painless-examples.asciidoc:335: "script": "params.max - params.min" <3> (bucket_script)
• docs/reference/vectors/vector-functions.asciidoc:102: "script": { (script_score)
• docs/reference/vectors/vector-functions.asciidoc:137: "script": { (script_score)
• docs/reference/vectors/vector-functions.asciidoc:176: "script": { (script_score)
• docs/reference/vectors/vector-functions.asciidoc:66: "script": { (script_score)

Anyway, I'm going to rewrite a few of these using runtime fields.

[javanna]

Thanks Nik! I removed the team-discuss label, as we discussed this with the search team, and decided that we should make this issue broader and list all of the different script contexts that we support in the search API and decide which ones can be replaced by runtime fields. I will work on this.

[nik9000]

Sounds good! Mind if I do the aggs ones? I'm sort of in search of something like this.

[javanna]

please go ahead ;)

[javanna]

The deprecation of the different flavor of scripts in the search API in favor of runtime fields is under discussion, but it's important to note that Kibana scripted fields rely on them in different places, and are not going to be removed in 8.0, hence we are not ready to deprecate in Elasticsearch. We are working on mentioning runtime fields in as many places as possible in the docs (thanks @nik9000 ) to make sure that users start using runtime fields which are more powerful. On the long run, we do expect runtime fields to replace a lot of script usages in search, but we are not quite ready to do that just now.

[a03nikki]

I don't know if this page needs to be updated as well or not, but figured I would mention it as the other doc updates relating to using runtime fields over the (old) scripted fields were done under this ticket.

https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-painless-examples.html

(1) Painless examples for transforms
...
(1.2) Getting time features by using aggregations
...
(1.4) Counting HTTP responses by using scripted metric aggregation
...

(I added the numbering to the section headers to make it a little easier to follow compared to the 7.13 deployed version of the page.)

The example from (1.2) "Getting time features by using aggregations" uses a script in a average aggregation. This lead me to https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html#_script_2 which uses runtime fields instead of the (older) "script" option on the aggregation. It doesn't mention at all the older (likely deprecated) aggs.{name}.{agg}.script field. The one for the terms agg is also gone as well.

There is also no mention on the runtime field limitations page that they cannot be used with data transforms. Only there is a performance hit for runtime fields for the user-defined time field.

I did not test it, but I would think the two sections (1.2 and 1.4) could use runtime fields instead of an aggregation script.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)