[Meta] Add more benchmark schema templates to integrations

[ruflin]

In elastic/elastic-package#1476 the initial schema templates from the corpus generator repo were added to the integrations. Ideally all integrations would have templates but as this is not feasible, lets start small. This is a meta issue to track the top 5 integrations for which schemas should be added. The proposed list is:

• system
• nginx
• k8s
• aws / aws_logs
• mysql

Each of these integrations have quite a few datasets, for detailed tracking the datasets are listed below (this list might change over time). Adding these dataset will potentially also add new requirements to the corpus which is expected. These additions in combination with the stream command will allow anyone to test integrations against a running Elastic Stack instance. The expected outcome is that the integration dashboards show mostly "reasonable" data. It is not expected that necessarily cross linking is possible or the data shown makes a 100% sense. For example CPU usage might fluctuate faster then in reality.

Tracking: Dataset list

Below is the list of integrations and dataset for tracking. Lets add reference to PR or code if completed. For each integration, lets add the most important datasets.

• K8s
  • kubernetes.container
  • kubernetes.pods
  • kubernetes.node
  • kubernetes.state_container
  • kubernetes.state_pod
  • kubernetes.events
• AWS
  • billing
  • ec2_logs
  • ec2_metrics
  • sqs
  • vpcflow
  • waf
  • cloudtrail
• Nginx
  • access
  • error
  • stub_status
• System
  • cpu
  • diskio
  • memory

• MySQL
  • error
  • performance
  • slowlog
  • status
  • galera_status

[lalit-satapathy]

We have a low level issue: #8719 which will be closed (not immediately but after all the WIP PRs are done) and we can use this top level issue for tracking of all package level template additions.

Right now adding Aliabbas as owner, please add anyone else as applicable.

Also adding the issue into relevant project boards.

[lalit-satapathy]

Hi @ruflin,

Since all the data streams benchmark schema templates are completed, can this issue be closed or any additional package(s) we should expand the scope it?

[ruflin]

This is great @lalit-satapathy Are the templates used to run benchmarks that we also look at from time to time?

We can close this issue but lets make sure we follow up on actively using these templates for developing features and running benchmarks.