Metricbeat-indexed kibana_stats docs differ from internally-indexed ones

[igoristic]

Related: #34940.

Tested on 7.4.

Internally-indexed kibana_stats docs contain the following keys that are absent from the Metricbeat-indexed kibana_stats docs:

• kibana_stats.usage.visualization_types (possibly also Relates to [Telemetry] Count of Visualization by Type #22010)
• kibana_stats.response_times.average
• kibana_stats.response_times.max

Sourced from: FAILURE elastic/estf-monitoring-snapshots#7.4 90237f3 multijob-kibana - 20191012053923-E5D2DAC4

JSON: kibana.zip

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[chrisronline]

@igoristic Were you able to reproduce this locally?

[igoristic]

@chrisronline What do you mean by "able to reproduce this locally"?

Do you mean by running the parity test via link? If so then yes. I also included the json results in the description.

Or, do you mean by downloading 7.4 ES, Kibana, and Metricbeat; installing and testing kibana_stats docs with and w/o Metricbeat manually? I think this is a lengthy process that should be reserved for the "investigation/fixing process" and not the "reporting process".

Let me know what you think, I'm happy to test it manually if need be

[chrisronline]

I guess both to some degree.

Since parity tests involve multiple stack products, part of the process of responding to them is identifying where the problem lies - is there something up with Metricbeat? Kibana? or maybe ES? Typically, once we identify this, we outline our steps that lead us to that conclusion and then file an issue in the appropriate repo.

Were you able to identify what exactly is going on? Are those fields always missing? Is it a timing issue? Were these missing ones recently added to 7.4?

[igoristic]

Okay, I'll try to investigate them as well, just thought those were two different phases based on this template: #35799

[chrisronline]

Yea that ticket is similar for sure, but before @ycombinator filed the ticket, he responded to the failure email with some investigation notes after diving in a bit:

This is failing because, once again, we have differences in what's being indexed natively by Kibana vs. when Kibana is monitored with Metricbeat: #35799 :(

I don't know for sure but there were changes made recently in the native collection code, to do with changing the frequency of telemetry (kibana_stats.usage) collection. These changes shouldn't have caused the above differences but they might be worth looking into as that was when the native collection code was most-recently touched. The PR that made those changes is here: #34609.

[igoristic]

This makes sense now. Thank you Chris!

[igoristic]

This same test is now passing, going to see if it comes up again during next couple of days then close it if not

[chrisronline]

Definitely some timing issues happening. We don't necessarily need to investigate now, but we should dive deeper at some point and figure out how we can make these more stable

[cachedout]

@chrisronline I will try to investigate further this week during my on-call rotation.

[cachedout]

I have opened this PR and expect a discussion on these fields may continue there: https://github.com/elastic/elastic-stack-testing/pull/402

[chrisronline]

Tests seem stable now. I'm closing this