Kibana creates bogus filters on fields with null values.

[sprysprocket]

Some background:

• Kibana 4.4.1
• Logstash 2.2.1
• ElasticSearch 2.2.1

This example uses a custom mapping, but I do not believe it is involved. Possibly more relevant is our use of the logstash csv filter, which generates null values for missing columns by default.

Consider a _source document that contains a null value for a not-analyzed field.

{
...many fields removed for brevity
"some_field": null,
}

Kibana will present the value for this field as " - ", and also provides the typical +/- filter buttons. When selecting one of these buttons, the search immediately becomes invalid, and the following error banner is displayed:

"Discover: No text specified for text query ..."

The underlying problem appears to be with the filter syntax. Clicking the edit button on the filter UI element shows the following:

{
  "query": {
    "match": {
      "some_field": {
        "query": null,
        "type": "phrase"
      }
    }
  }
}

It would seem simple enough for Kibana to 1) recognize this as a null value, and 2) create the query in the the appropriate filter context as follows:

{
  "query": {
    "exists": {
      "field": "some_field"
    }
  }
}

Note that manually editing the filter to use this syntax works as expected.

[jbudz]

Happening in 5.x too, see #8164.

[Bargs]

Should filtering on a null value create a filter that matches docs where the field exists, or does not exist? To me it seems like it should be the latter.

[Bargs]

We discussed this on Zoom today. Since null might not always mean the same as "does not exist" for all users we don't think it makes sense to automatically create an exists filter for null values. Instead we'll disable the +/- filter buttons since null values cannot be searched on in Elasticsearch. Users will still be able to create exists filters explicitly by clicking on the exists filter button.

There are some additional scenarios where the +/- buttons should be disabled:

• Value is null
• Value is an empty array
• Value is an array containing only null values
• If the value is an array with both null and non-null values, ignore the null and create filters for the non-nulls.

Elasticsearch also has a feature for making null values searchable. null_value allows users to specify a single value to index in place of null when ES encounters a null valued field. As a separate, future enhancement we could make note of this property at index pattern creation time and use it to filter on the correct replacement value when Kibana encounters a null.

[matthid]

I just hit this today as well. I think the UX now is even worse than the one reported above. Above was reported that there is an error with "No text specified for text query", now I only see Discover: Internal Server Error

Kibana version:
v 7.1.1

Elasticsearch version:
v 7.1.1

Describe the bug:
If you are on the "discovery" page and you klick on a filter button for an null (or not existing) field then you receive an internal server error.

Steps to reproduce:

1. Open Kibana and go to discovery
2. Klick the + button

Expected behavior:
Works and uses the filter !(_exists_:CorrelationId).

Screenshots (if relevant):

Errors in browser console (if relevant):

Error: Internal Server Error
      SearchError@http://elk-logging.host.tld/bundles/commons.bundle.js:3:298201
_callee2$/<.searching</<@http://elk-logging.host.tld/bundles/commons.bundle.js:4:324908
processQueue@http://elk-logging.host.tld/built_assets/dlls/vendors.bundle.dll.js:427:199687
scheduleProcessQueue/<@http://elk-logging.host.tld/built_assets/dlls/vendors.bundle.dll.js:427:200650
$digest@http://elk-logging.host.tld/built_assets/dlls/vendors.bundle.dll.js:427:210412
$apply@http://elk-logging.host.tld/built_assets/dlls/vendors.bundle.dll.js:427:213219
done@http://elk-logging.host.tld/built_assets/dlls/vendors.bundle.dll.js:427:132717
completeRequest@http://elk-logging.host.tld/built_assets/dlls/vendors.bundle.dll.js:427:136329
requestLoaded@http://elk-logging.host.tld/built_assets/dlls/vendors.bundle.dll.js:427:135225

Any additional context:

The filter (in Query DSL) looks like this:

{
  "query": {
    "match": {
      "CorrelationId": {
        "query": null,
        "type": "phrase"
      }
    }
  }
}

[Bargs]

Going back to at least 7.3, an error is now displayed if you try to create a filter on a null value. This is better than creating bogus filters, but the solution outlined in #7189 (comment) would still be the proper fix.

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[Sagesh]

Any plan to fix it in the coming releases?