[Bug] Discover document query timeout - 8.12.0

[aarju]

Kibana version:
8.12.0

Elasticsearch version:
8.12.0

Describe the bug:
When running a wildcard query on our AWS Integration data view the Kibana data query will return quickly and update the timeline view, but the document view will take a long time to populate and sometimes will timeout.

While testing this in the dev tools console I found that by removing the highlight section of the query that is added in discover the query will return quickly as expected.

Steps to reproduce:

1. Run an async query on the aws cloudtrail integration:

POST /logs-aws.cloudtrail*/_async_search?batched_reduce_size=64&ccs_minimize_roundtrips=true&wait_for_completion_timeout=200ms&keep_on_completion=true&keep_alive=60000ms&ignore_unavailable=true&preference=1705924391493
{
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "query_string": {
            "query": "*192.168.1.1*"
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2024-01-21T14:00:00.000Z",
              "lte": "2024-01-22T14:29:45.693Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}

In our cluster this query will timeout and not return results.

2. Remove the "highlight" element from the json and re-run the query

POST /logs-aws.cloudtrail*/_async_search?batched_reduce_size=64&ccs_minimize_roundtrips=true&wait_for_completion_timeout=200ms&keep_on_completion=true&keep_alive=60000ms&ignore_unavailable=true&preference=1705924391493
{
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "query_string": {
            "query": "*192.168.1.1*"
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2024-01-21T14:00:00.000Z",
              "lte": "2024-01-22T14:29:45.693Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

In our cluster this query returns results in 14s

Expected behavior:

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[davismcphee]

@aarju Thanks for reporting! Can you confirm if any of the fields in the index mapping are of type match_only_text?

Linking a couple of likely related issues for our future reference:

• Highlighting can take 50% to 99% of search time elasticsearch#103298
• Filtering via +/- buttons against match_only_text fields really slow #171441

[aarju]

This data view is from the AWS Cloudtrail integration. In the docs they mention 10 fields that are mapped as match_only_text. I don't have permissions on this cluster to view the data view mapping, but we generally stick to the defaults for these integrations so it is likely that they are mapped that way.

[aarju]

@christophercutajar just confirmed that the fields such as error.message are mapped as match_only_text.

{
  ".ds-logs-aws.cloudtrail-ecmaster-2024.01.22-000149": {
    "mappings": {
      "error.message": {
        "full_name": "error.message",
        "mapping": {
          "message": {
            "type": "match_only_text"
          }
        }
      }
    }
  }
} 

[kertal]

like @davismcphee mentioned, this is high likely this issue in ES, elastic/elasticsearch#103298 so currently in Kibana we can't fix this, setting the issue to blocked

[kertal]

@aarju it's been a while and elastic/elasticsearch#103298 was merged 8.14.0, 8.13.1 could you evaluate if the issue is still present? many thx!

[aarju]

@kertal it looks like this bug is no longer present and this issue can be closed. Thanks!

[kertal]

@aarju thx, great to hear! FYI @lukasolson @davismcphee