Query string filter with fields automatically adding default fields parameters

[stacey-gammon]

In 5.6 this filter worked fine:

{
  "query": {
    "query_string": {
      "analyze_wildcard": true,
      "fields": [
        "user.keyword"
      ],
      "query": "bob"
    }
  }
}

in 6.0, default_fields is added automatically to the filter which breaks the query:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"match_all\" : {\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"query_string\" : {\n          \"query\" : \"bob\",\n          \"default_field\" : \"*\",\n          \"fields\" : [\n            \"referer^1.0\"\n          ],\n          \"type\" : \"best_fields\",\n          \"default_operator\" : \"or\",\n          \"max_determinized_states\" : 10000,\n          \"enable_position_increments\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"escape\" : false,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : null,\n            \"to\" : null,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}","index_uuid":"vRkJQI87Qx-6ZNea2nupgg","index":"logstash-0"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-0","node":"Eba1LxIMRo2CgXn3QLwvYA","reason":{"type":"query_shard_exception","reason":"failed to create query: {\n  \"bool\" : {\n    \"must\" : [\n      {\n        \"match_all\" : {\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"query_string\" : {\n          \"query\" : \"bob\",\n          \"default_field\" : \"*\",\n          \"fields\" : [\n            \"referer^1.0\"\n          ],\n          \"type\" : \"best_fields\",\n          \"default_operator\" : \"or\",\n          \"max_determinized_states\" : 10000,\n          \"enable_position_increments\" : true,\n          \"fuzziness\" : \"AUTO\",\n          \"fuzzy_prefix_length\" : 0,\n          \"fuzzy_max_expansions\" : 50,\n          \"phrase_slop\" : 0,\n          \"analyze_wildcard\" : true,\n          \"escape\" : false,\n          \"boost\" : 1.0\n        }\n      },\n      {\n        \"range\" : {\n          \"@timestamp\" : {\n            \"from\" : null,\n            \"to\" : null,\n            \"include_lower\" : true,\n            \"include_upper\" : true,\n            \"boost\" : 1.0\n          }\n        }\n      }\n    ],\n    \"adjust_pure_negative\" : true,\n    \"boost\" : 1.0\n  }\n}","index_uuid":"vRkJQI87Qx-6ZNea2nupgg","index":"logstash-0","caused_by":{"type":"query_validation_exception","reason":"Validation Failed: 1: [query_string] cannot use [fields] parameter in conjunction with [default_field];"}}}]},"status":400}
    at http://localhost:5701/bundles/kibana.bundle.js?v=15976:233:1369
    at Function.Promise.try (http://localhost:5701/bundles/commons.bundle.js?v=15976:95:30830)
    at http://localhost:5701/bundles/commons.bundle.js?v=15976:95:30200
    at Array.map (<anonymous>)
    at Function.Promise.map (http://localhost:5701/bundles/commons.bundle.js?v=15976:95:30155)
    at callResponseHandlers (http://localhost:5701/bundles/kibana.bundle.js?v=15976:233:985)
    at http://localhost:5701/bundles/kibana.bundle.js?v=15976:232:21395
    at processQueue (http://localhost:5701/bundles/commons.bundle.js?v=15976:39:9912)
    at http://localhost:5701/bundles/commons.bundle.js?v=15976:39:10805

[Bargs]

Guessing it's a result of #13231

Two possible workarounds until we fix it:

1. Don't use the fields property, specify it in the query string instead: user.keyword:bob
2. Remove default_field from query:queryString:options in Advanced Settings. This could make highlighting worse for indices that still rely on an _all field.

[ppf2]

+1. Ran into this when testing an upgrade of an internal app to 6.1. This breaks most of our objects because in 5.x, we used "fields" in Advanced Settings (query:queryString:options) and as a result, every saved object ends up picking up the "fields" parameter in the object definition (kibanaSavedObjectMeta.searchSourceJSON). And now with the default being "default_field": "*" , it is causing ES to throw cannot use [fields] parameter in conjunction with [default_field] on all the objects.

  "query": {
    "query_string": {
      "query": "*",
      "analyze_wildcard": true,
      "fields": [  <------
....
....

[Bargs]

@ppf2 can you elaborate? Your issue sounds slightly different to me. If a user has changed query:queryString:options this update should not affect them. #13231 only updated the default value for query:queryString:options so if the user has overridden the default it should not be a problem.

[ppf2]

@Bargs I think what happened is that we upgraded from 5.x to 6.x. Knowing about the _all change, I updated the Advanced Settings to { "analyze_wildcard": true, "default_field": "*" } (to match the new default that comes out of the box in Kibana in an attempt to use more default settings that goes with the _all change in ES - we previously used fields in the queryString:options). As a result, we hit this issue because it now tries to use both default_field and the fields option (fields settings were saved as part of the objects that were upgraded).

[Bargs]

Thanks for the additional info @ppf2, I think I understand now. I wrote up a separate issue to track this problem: #15828

I don't see an easy fix to this at the moment but I'll discuss it with the team.

[zcola]

Remove default_field from query:queryString:options in Advanced Settings is ok for me. What is the effect of deletion?