[Monitoring Kibana] System test for ensuring parity in indexed docs

[ycombinator]

We are currently working on a solution that will allow users to use Metricbeat to monitor the Elastic stack. The first piece of this is to try and use Metricbeat to monitor Kibana.

The idea is that metricbeat will create exactly the same documents in .monitoring-kibana-* indices as Kibana's bulk uploader does today. In order to verify that this parity is indeed maintained at all times, we should write an automated system test that will do something like the following:

1. Start up Elasticsearch (with the default Basic license is okay)
2. Enable Monitoring in Elasticsearch (via the cluster setting xpack.monitoring.collection.enabled: true )
3. Start up Kibana (with the default Basic license is okay)
4. Let Kibana run for ~20 seconds so a few documents are indexed into .monitoring-kibana-*.
5. Stop Kibana
6. From .monitoring-kibana-*, retrieve a document each for type = kibana_stats and type = kibana_settings.
7. Delete the .monitoring-kibana-* indices.
8. In kibana.yml, set xpack.monitoring.kibana.collection.enabled: false.
9. Start up Kibana again.
10. Enable the kibana module with the stats and settings metricsets in metricbeat.
11. Set xpack.enabled: true in the modules.kibana section in metricbeat.yml.
12. Start metricbeat.
13. Let metricbeat run for ~20 seconds so a few documents are indexed into .monitoring-kibana-*.
14. Stop metricbeat
15. From .monitoring-kibana-*, retrieve a document each for type = kibana_stats and type = kibana_settings.
16. Compare the documents retrieved indexed by Kibana's with those indexed by metricbeat. Verify that their structures are identical (same fields, not necessarily same values).

[ycombinator]

/cc @ruflin @chrisronline @tsullivan @pickypg

[ruflin]

I wonder where we should put these tests, meaning if they fit into the beats repo or are rather outside.

[tsullivan]

@liza-mae does this sort of testing fall under the automated stack integration tests?

[chrisronline]

Maybe here: https://github.com/elastic/integration-test?

[liza-mae]

@tsullivan yes sounds like it and it is here actually: https://github.com/elastic/elastic-stack-testing aka ESTF. I would be happy to answer any questions or help anyone getting started, just reach out to me. I am in the process of getting some standards in place for wider adoption and usage :)

[ycombinator]

Chatted with @liza-mae and @chrisronline off-issue today. Liza gave us a quick tour of ESTF and we've agreed to write these tests over there. So I'm closing this issue in favor of https://github.com/elastic/elastic-stack-testing/issues/25.