[Discover] Compile error when filtering on scripted fields #8404

stacey-gammon · 2016-09-21T15:11:04Z

To repro:

Go into the logstash-* index pattern
Click the scripted fields tab
Click add scripted field
Create field with name: "Inverse bytes"
Select language 'Painless'
Type in Script input: doc["bytes"].value * -1
Click Save
Click over to Discover
Ensure scripted fields are shown in the field list and in the documents
Ensure filters are enabled for all scripted fields in list and in documents
Filter by a script value

After following the above steps and attempt to filter by a script value, I run into a Discover: compile error with the following error message:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","script_stack":["... [\"bytes\"].value * -1) == value","                             ^---- HERE"],"script":"(doc[\"bytes\"].value * -1) == value","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetch","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-0","node":"yb6x_Ou6SdK969VVMixU1w","reason":{"type":"query_shard_exception","reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"match_all\" : {\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -3531\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : 1474211056294,\n            \"to\" : 1474470256294,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"format\" : \"epoch_millis\",\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"must_not\" : [\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -5173\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : 0\n            }\n          },\n          \"boost\" : 1.0\n        }\n      }\n    ],\n    \"disable_coord\" : false,\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}","index_uuid":"qjdZ3IWlRO-hqD_GhTIxdA","index":"logstash-0","caused_by":{"type":"script_exception","reason":"compile error","caused_by":{"type":"illegal_argument_exception","reason":"Variable [value] is not defined."},"script_stack":["... [\"bytes\"].value * -1) == value","                             ^---- HERE"],"script":"(doc[\"bytes\"].value * -1) == value","lang":"painless"}}}],"caused_by":{"type":"query_shard_exception","reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"match_all\" : {\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -3531\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : 1474211056294,\n            \"to\" : 1474470256294,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"format\" : \"epoch_millis\",\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"must_not\" : [\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -5173\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(doc[\\\"bytes\\\"].value * -1) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : 0\n            }\n          },\n          \"boost\" : 1.0\n        }\n      }\n    ],\n    \"disable_coord\" : false,\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}","index_uuid":"qjdZ3IWlRO-hqD_GhTIxdA","index":"logstash-0","caused_by":{"type":"script_exception","reason":"compile error","caused_by":{"type":"illegal_argument_exception","reason":"Variable [value] is not defined."},"script_stack":["... [\"bytes\"].value * -1) == value","                             ^---- HERE"],"script":"(doc[\"bytes\"].value * -1) == value","lang":"painless"}}},"status":400}
KbnError@http://localhost:5601/bundles/commons.bundle.js?v=14300:63:25990
RequestFailure@http://localhost:5601/bundles/commons.bundle.js?v=14300:63:27183
http://localhost:5601/bundles/kibana.bundle.js?v=14300:26:28446
http://localhost:5601/bundles/commons.bundle.js?v=14300:75:22358
map@[native code]
map@http://localhost:5601/bundles/commons.bundle.js?v=14300:75:21681
processQueue@http://localhost:5601/bundles/commons.bundle.js?v=14300:38:23623
http://localhost:5601/bundles/commons.bundle.js?v=14300:38:23900
$digest@http://localhost:5601/bundles/commons.bundle.js?v=14300:39:2364
$apply@http://localhost:5601/bundles/commons.bundle.js?v=14300:39:5044
done@http://localhost:5601/bundles/commons.bundle.js?v=14300:37:25033
completeRequest@http://localhost:5601/bundles/commons.bundle.js?v=14300:37:28710
onload@http://localhost:5601/bundles/commons.bundle.js?v=14300:37:29649

The text was updated successfully, but these errors were encountered:

thomasneirynck · 2016-09-21T17:29:20Z

can you check if you are using the "expression" language when creating this field;

?

it works as an expression, but not as a painless script. Not sure if there's some invalid syntax in that example of inverse bytes

thomasneirynck · 2016-09-21T17:49:00Z

below is an example of an ES search request that triggers this error. It appears we get an error when using the script as part of a filter.

POST logstash-0/_search
{  
   "size":500,
   "sort":[  
      {  
         "@timestamp":{  
            "order":"desc",
            "unmapped_type":"boolean"
         }
      }
   ],
   "highlight":{  
      "pre_tags":[  
         "@kibana-highlighted-field@"
      ],
      "post_tags":[  
         "@/kibana-highlighted-field@"
      ],
      "fields":{  
         "*":{  

         }
      },
      "require_field_match":false,
      "fragment_size":2147483647
   },
   "query":{  
      "bool":{  
         "must":[  
            {  
               "query_string":{  
                  "query":"*",
                  "analyze_wildcard":true
               }
            },
            {  
               "script":{  
                  "script":{  
                     "inline":"(-1 * doc['bytes'].value) == value",
                     "lang":"painless",
                     "params":{  
                        "value":0
                     }
                  }
               }
            },
            {  
               "script":{  
                  "script":{  
                     "inline":"(-1 * doc['bytes'].value) == value",
                     "lang":"painless",
                     "params":{  
                        "value":-2724
                     }
                  }
               }
            },
            {  
               "range":{  
                  "@timestamp":{  
                     "gte":1474479018497,
                     "lte":1474479918497,
                     "format":"epoch_millis"
                  }
               }
            }
         ],
         "must_not":[  

         ]
      }
   },
   "aggs":{  
      "2":{  
         "date_histogram":{  
            "field":"@timestamp",
            "interval":"30s",
            "time_zone":"America/New_York",
            "min_doc_count":1
         }
      }
   },
   "stored_fields":[  
      "*"
   ],
   "_source":true,
   "script_fields":{  
      "inverse bytes":{  
         "script":{  
            "inline":"-1 * doc['bytes'].value",
            "lang":"painless"
         }
      }
   },
   "docvalue_fields":[  
      "utc_time",
      "relatedContent.article:modified_time",
      "@timestamp",
      "relatedContent.article:published_time"
   ]
}

Response:

{  
   "error":{  
      "root_cause":[  
         {  
            "type":"script_exception",
            "reason":"compile error",
            "script_stack":[  
               "... * doc['bytes'].value) == value",
               "                             ^---- HERE"
            ],
            "script":"(-1 * doc['bytes'].value) == value",
            "lang":"painless"
         }
      ],
      "type":"search_phase_execution_exception",
      "reason":"all shards failed",
      "phase":"query_fetch",
      "grouped":true,
      "failed_shards":[  
         {  
            "shard":0,
            "index":"logstash-0",
            "node":"apPgvtcuTbSA2V4HWRUWww",
            "reason":{  
               "type":"query_shard_exception",
               "reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"query_string\" : {\n          \"query\" : \"*\",\n          \"fields\" : [ ],\n          \"use_dis_max\" : true,\n          \"tie_breaker\" : 0.0,\n          \"default_operator\" : \"or\",\n          \"auto_generate_phrase_queries\" : false,\n          \"max_determined_states\" : 10000,\n          \"lowercase_expanded_terms\" : true,\n          \"enable_position_increment\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"locale\" : \"und\",\n          \"escape\" : false,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(-1 * doc['bytes'].value) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : 0\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(-1 * doc['bytes'].value) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -2724\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : 1474479018497,\n            \"to\" : 1474479918497,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"format\" : \"epoch_millis\",\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"disable_coord\" : false,\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}",
               "index_uuid":"2Gr8UT6HTTCyaJWywRhrCQ",
               "index":"logstash-0",
               "caused_by":{  
                  "type":"script_exception",
                  "reason":"compile error",
                  "caused_by":{  
                     "type":"illegal_argument_exception",
                     "reason":"Variable [value] is not defined."
                  },
                  "script_stack":[  
                     "... * doc['bytes'].value) == value",
                     "                             ^---- HERE"
                  ],
                  "script":"(-1 * doc['bytes'].value) == value",
                  "lang":"painless"
               }
            }
         }
      ],
      "caused_by":{  
         "type":"query_shard_exception",
         "reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"query_string\" : {\n          \"query\" : \"*\",\n          \"fields\" : [ ],\n          \"use_dis_max\" : true,\n          \"tie_breaker\" : 0.0,\n          \"default_operator\" : \"or\",\n          \"auto_generate_phrase_queries\" : false,\n          \"max_determined_states\" : 10000,\n          \"lowercase_expanded_terms\" : true,\n          \"enable_position_increment\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"locale\" : \"und\",\n          \"escape\" : false,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(-1 * doc['bytes'].value) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : 0\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"script\" : {\n          \"script\" : {\n            \"inline\" : \"(-1 * doc['bytes'].value) == value\",\n            \"lang\" : \"painless\",\n            \"params\" : {\n              \"value\" : -2724\n            }\n          },\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : 1474479018497,\n            \"to\" : 1474479918497,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"format\" : \"epoch_millis\",\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"disable_coord\" : false,\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}",
         "index_uuid":"2Gr8UT6HTTCyaJWywRhrCQ",
         "index":"logstash-0",
         "caused_by":{  
            "type":"script_exception",
            "reason":"compile error",
            "caused_by":{  
               "type":"illegal_argument_exception",
               "reason":"Variable [value] is not defined."
            },
            "script_stack":[  
               "... * doc['bytes'].value) == value",
               "                             ^---- HERE"
            ],
            "script":"(-1 * doc['bytes'].value) == value",
            "lang":"painless"
         }
      }
   },
   "status":400
}

stacey-gammon · 2016-10-12T13:46:50Z

Confirmed with ES that the issue is on our side (See elastic/elasticsearch#20885). It appears like the request should send
"inline":"(-1 * doc['bytes'].value) == params.value"

instead of

"inline":"(-1 * doc['bytes'].value) == value"

Bargs · 2016-10-12T14:34:11Z

Users can theoretically write groovy and other language scripts if they've manually enabled them in ES, so we should probably check those as well.

stacey-gammon · 2016-10-12T14:51:50Z

I think I found the culprit and have a fix, at least for painless & expression. I'm not sure about the other script types. Kibana will show additional options in the language dropdown depending on ES settings? How can I adjust those settings to verify if it works?

Bargs · 2016-10-12T15:38:47Z

You can enable groovy in your grunt esvm config by adding the following code to your grunt esvm (esvm.js) dev config block:

script: {
  engine: {
    groovy: {
      inline: true
    }
  }
}

stacey-gammon · 2016-10-12T17:20:47Z

Good call on checking Groovy - indeed it is not happy with params.value. Checking with @clintongormley to see if this is expected.

Need to use params.value instead of value. Fixes elastic#8404 Add params prefix in another spot for painless scripted fields Fix date histogram with scripted fields Remove format: epoch_millis so the script compiles. I am not 100% confident of the side affect from this (it’s used for non-scripted fields, but I’m not sure where I would put it for scripted fields, or if it’s needed). At any rate, it appears that formatting settings for scripted fields is still being honored, even after removing it from params.

--------- **Commit 1:** Fix our request to ES for filtering on scripted fields Need to use params.value instead of value. Fixes #8404 Add params prefix in another spot for painless scripted fields Fix date histogram with scripted fields Remove format: epoch_millis so the script compiles. I am not 100% confident of the side affect from this (it’s used for non-scripted fields, but I’m not sure where I would put it for scripted fields, or if it’s needed). At any rate, it appears that formatting settings for scripted fields is still being honored, even after removing it from params. * Original sha: 7cdb74d * Authored by Stacey Gammon <[email protected]> on 2016-10-12T14:54:23Z

--------- **Commit 1:** Fix our request to ES for filtering on scripted fields Need to use params.value instead of value. Fixes elastic#8404 Add params prefix in another spot for painless scripted fields Fix date histogram with scripted fields Remove format: epoch_millis so the script compiles. I am not 100% confident of the side affect from this (it’s used for non-scripted fields, but I’m not sure where I would put it for scripted fields, or if it’s needed). At any rate, it appears that formatting settings for scripted fields is still being honored, even after removing it from params. * Original sha: 24622fad83d0388a38d9557e4a1ff7b80ec27083 [formerly 7cdb74d] * Authored by Stacey Gammon <[email protected]> on 2016-10-12T14:54:23Z Former-commit-id: 4b54dc6

stacey-gammon added blocker bug Fixes for quality problems that affect the customer experience v5.0.0 labels Sep 21, 2016

stacey-gammon mentioned this issue Sep 22, 2016

[Discover] - Exception sorting on scripted painless field #8431

Closed

epixa removed the blocker label Sep 30, 2016

stacey-gammon mentioned this issue Oct 12, 2016

illegal_argument_exception when using scripted fields. elastic/elasticsearch#20885

Closed

stacey-gammon mentioned this issue Oct 12, 2016

Scripted fields must reference values via params when filtering #8642

Merged

stacey-gammon closed this as completed in #8642 Oct 13, 2016

This was referenced Oct 13, 2016

[backport] PR #8642 to 5.x - Scripted fields must reference values via params when filtering #8675

Merged

[backport] PR #8642 to 5.0 - Scripted fields must reference values via params when filtering #8676

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Discover] Compile error when filtering on scripted fields #8404

[Discover] Compile error when filtering on scripted fields #8404

stacey-gammon commented Sep 21, 2016 •

edited

Loading

thomasneirynck commented Sep 21, 2016 •

edited

Loading

thomasneirynck commented Sep 21, 2016 •

edited

Loading

stacey-gammon commented Oct 12, 2016

Bargs commented Oct 12, 2016

stacey-gammon commented Oct 12, 2016

Bargs commented Oct 12, 2016

stacey-gammon commented Oct 12, 2016

[Discover] Compile error when filtering on scripted fields #8404

[Discover] Compile error when filtering on scripted fields #8404

Comments

stacey-gammon commented Sep 21, 2016 • edited Loading

thomasneirynck commented Sep 21, 2016 • edited Loading

thomasneirynck commented Sep 21, 2016 • edited Loading

stacey-gammon commented Oct 12, 2016

Bargs commented Oct 12, 2016

stacey-gammon commented Oct 12, 2016

Bargs commented Oct 12, 2016

stacey-gammon commented Oct 12, 2016

stacey-gammon commented Sep 21, 2016 •

edited

Loading

thomasneirynck commented Sep 21, 2016 •

edited

Loading

thomasneirynck commented Sep 21, 2016 •

edited

Loading