Metrics App and Metrics Inventory taking very long time to load

[bvader]

@simianhacker
@sorantis

Kibana version: 7.5.1

Elasticsearch version: 7.5.1

Server OS version: ESS GCP

Browser version: Chrome Version 79.0.3945.130 (Official Build) (64-bit)

Browser OS version: MacOS Sierra

Original install method (e.g. download page, yum, from source, etc.): ESS

Describe the bug:

Metrics Inventory is ~1500 hosts, data is shipped by Metricbeat

Metrics Inventory and Metrics Explorer are taking upwards of 70 seconds to load. When using the developer tools in chrome the GraphQL query appears to be be the long running request.

Currently running on 6 x 64GB gcp.data.highio.1

Once the Metrics Inventory is loaded it works pretty well, but then when you navigate to Metrics Explorer the slow loading happens again.

When you switch back to Metrics Inventory it happens again, in other words every time you navigate to either.

Steps to reproduce:

1. Load 1500 Host via Metricbeat
2. Navigate to Metrics Explorer

Expected behavior:

Expect the Metrics Inventory and Metrics Explorer to load quickly, less than 5 seconds.

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

The query/code that appears to be taking the most time is https://github.com/elastic/kibana/blob/master/x-pack/plugins/infra/server/lib/adapters/fields/framework_fields_adapter.ts#L46-L60

This code compiles a list of all the event.dataset values and then uses that to filter down the list of available fields. This is only necessary because Metricbeat is shipping every possible field in the index template. The correct fix is to stop shipping all the fields but only the fields for the active modules.

[simianhacker]

This was addressed with #58553 and should be released in a patch update. We also tested a patched build against this cluster to confirm the fix will indeed resolve this issue.

[cpmoore]

@simianhacker
With #58553 being merged, the datasource request does not take long, however the /api/metrics/snapshot endpoint still does not load on our cluster. I've waited at least a minute with no response.

[simianhacker]

@cpmoore Did it fail or just timeout? Are there any errors in the logs?

[cpmoore]

@simianhacker No errors in the log, the request just eventually times out after 30000ms

[simianhacker]

@cpmoore My guess is you're gonna need to look at your custer's capacity. The query that backs this request uses composite aggregations to gather up all the data and send it to the browser. It might be that your cluster is too small for the amount of data you're working with. Depending on where the timeout is happening, Kibana or Elasticsearch, you might also need to increase the values for those settings.

[cpmoore]

@simianhacker
I'm assuming it uses a request similar to the one below

POST /metricbeat-*/_search
{
  "size": 0,
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-1m",
        "lte": "now"
      }
    }
  },
  "aggs": {
    "hosts_composite": {
      "composite": {
        "size": 1000, 
        "after": {"hosts":"wsrpprd2.corp.fin"}, 
        "sources": [
          {
            "hosts": {
              "terms": {
                "field": "host.name"
              }
            }
          }
        ]
      },
      "aggs": {
        "cpu": {
          "avg": {
            "field": "system.cpu.total.norm.pct"
          }
        }
      }
    }
  }
}

This request returns almost instantly. (The response says "took" : 30)
See attached video, (sorry about the zip, I can't upload mp4s without zipping them)
2020-03-25_18-06-54.zip

Shouldn't the metric inventory take a similar amount of time as the above request?

[simianhacker]

@cpmore Sorry... I totally forgot about this bug: #58503 It's scheduled to go out in 7.6.2

[cpmoore]

@simianhacker
Alrighty cool! Thanks, sorry to hijack this issue.

[simianhacker]

@cpmoore No apologies necessary... Sorry for the bug, hopefully that will be release soon and you can move past this issue.