[Lens] "Top Values" allows setting missing values switch for numeric fields

[maihde]

Kibana version: master @ b5f0bc9

Elasticsearch version: 8.0.0 snapshot

Original install method (e.g. download page, yum, from source, etc.): Source

Describe the bug:

Addition:
Part of this is fixed already - #102640 (comment) is the missing part

Legacy aggregation based visualizations didn't allow "Show missing values" to be selected with "Terms" aggregations against numeric fields. The bug is that Lens attempts to implement similar behavior by preventing "Top values" from being used at all by showing an error that says "To use this function, select a different field.". There are two bugs:

1. A numeric field should be allowed with "Top values" without any error prompt
2. By simply toggling the drop down or not using the "Drop field" you can get "Top values" to work with a numeric field without error.

Describe the feature:

Support missing values in Lens and Terms-based aggregation visualizations on numeric fields.

Steps to reproduce:

1. Insert Data

POST test_data/_doc/
{
  "timestamp": "2021-06-17T02:03:04Z",
  "a_string_field": "A",
  "a_number_field": 0

}
POST test_data/_doc/
{
  "timestamp": "2021-06-17T02:03:04Z",
  "a_string_field": "B"
}

2. Create index pattern
3. Create Lens visualization on test_data with horizontal axis timestamp and vertical axis Count verify 3 records are shown.
4. Click on "Drop a field or click to add" on Break down by
5. Select "Top values" then select a_number_field. Observe that two records are shown with values of 0 and 1 in legend.
6. Expand "Advanced" and enable "Include documents without this field". Observe that only 2 records are shown.
7. Return to top level Lens configuration and clear "Break down by" fields.
8. Drag a_number_field onto Break down by and then click it
9. Click on "Top values" observe that error says "To use this function, select a different field."
10. Click the drop-down box and select a_string_field.keyword and repeat and select a_number_field. Observe that "Top values" configuration is now available.

Expected behavior:

Step 7 - three records are shown
Step 10 - no error message is shown
Step 5 & 11 - behavior is inconsistent about showing error message, but once the error message is removed for numeric fields this will become a non-issue.

Any additional context:

My suspicion is because doing this type of aggregation:

POST test_data/_search?size=0
{
    "aggs": {
      "2": {
        "terms": {
          "field":"a_number_field",
          "order":{"_count":"desc"},
          "size":5,
          "missing": "Missing"
        }
      }
    }
}

Fails with

        "reason" : {
          "type" : "number_format_exception",
          "reason" : "For input string: \"Missing\""
        }

But an alternative is that when the "Show missing values is enabled" to simply add a 'filter' aggregation:

POST test_data/_search?size=0
{
  "aggs": {
    "2": {
      "terms": {
        "field":"a_number_field",
        "order":{"_count":"desc"},
        "size":5
      }
    },
    "missing": {
      "filter": {
        "bool": {
          "must_not": { "exists": { "field": "a_number_field" } }
        }
      }
    }
  }
}

With this implementation, missing values can be used for both string and numeric Terms/Top values aggregations.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[flash1293]

The bug is that Lens attempts to implement similar behavior by preventing "Top values" from being used at all by showing an error that says "To use this function, select a different field."

Top values should work with number fields - it's not related to the missing bucket issues.

@dej611 Does #102384 fix this? Sounds very similar.,

[dej611]

Sounds similar to me as well.

[flash1293]

The fact missing can be set for number fields in Lens is a real bug though - I will keep the issue open to track fixing this part.

[maihde]

@flash1293 my hope is that the fix for allowing setting "missing" values is accomplished by altering the behavior such that missing values can be shown for numeric fields (as shown above) instead of blocking the selection of "missing". I understand this becomes more of a feature request, but our users frequently complain about the inability to use "missing" in conjunction with numeric fields.

[flash1293]

@maihde There are some technical challenges in doing this - I think it's worth a separate issue. However, there should be an easy workaround - extend your mapping and index the number as a keyword as well:

{
  "mappings": {
    "properties": {
      "field1": {
        "type": "double",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

On field1.keyword you can use the missing option and it will work just fine.

Note: You won't be able to format the numbers this way.

The requirement for missing option on numeric fields (without mapping tricks) is absolutely justified.