[O11y][Kubernetes] Rally benchmark kubernetes.node

[ali786XI]

Proposed commit message

• This PR adds benchmarking templates to the node data stream of Kubernetes

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.

How to test this PR locally

Run this command from package root

• elastic-package benchmark rally --benchmark node-benchmark -v
• elastic-package benchmark stream --benchmark node-benchmark -v

Related issues

• Relates [Meta] Add more benchmark schema templates to integrations #8636

Screenshots

--- Benchmark results for package: kubernetes - START ---
╭──────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                             │
├────────────────────────┬─────────────────────────────────────────────────────────┤
│ benchmark              │                                          node-benchmark │
│ description            │         Benchmark 20000 kubernetes.node events ingested │
│ run ID                 │                    65325741-6e4f-4493-8008-2f6bd9b0e546 │
│ package                │                                              kubernetes │
│ start ts (s)           │                                                       0 │
│ end ts (s)             │                                              1707473229 │
│ duration               │                                             474298h7m9s │
│ generated corpora file │ /root/.elastic-package/tmp/rally_corpus/corpus-33646997 │
╰────────────────────────┴─────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────╮
│ parameters                                                         │
├─────────────────────────────────┬──────────────────────────────────┤
│ package version                 │                           1.56.0 │
│ data_stream.name                │                             node │
│ corpora.generator.total_events  │                            20000 │
│ corpora.generator.template.path │ ./node-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                  │
│ corpora.generator.template.type │                           gotext │
│ corpora.generator.config.path   │      ./node-benchmark/config.yml │
│ corpora.generator.config.raw    │                            map[] │
│ corpora.generator.fields.path   │      ./node-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                            map[] │
╰─────────────────────────────────┴──────────────────────────────────╯
╭───────────╮
│ cluster i │
│ nfo       │
├───────┬───┤
│ name  │   │
│ nodes │ 1 │
╰───────┴───╯
╭───────────────────────────────────────╮
│ disk usage for index metrics-kubernet │
│ es.node-ep (for all fields)           │
├──────────────────────────────┬────────┤
│ total                        │ 9.8 MB │
│ inverted_index.total         │ 182 kB │
│ inverted_index.stored_fields │ 7.1 MB │
│ inverted_index.doc_values    │ 2.0 MB │
│ inverted_index.points        │ 474 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                │
├────────────────────────────────────────────────────────────────┬───────────────────────────┤
│ Cumulative indexing time of primary shards                     │   0.20473333333333335 min │
│ Min cumulative indexing time across primary shards             │                     0 min │
│ Median cumulative indexing time across primary shards          │ 0.0005333333333333334 min │
│ Max cumulative indexing time across primary shards             │   0.12678333333333333 min │
│ Cumulative indexing throttle time of primary shards            │                     0 min │
│ Min cumulative indexing throttle time across primary shards    │                     0 min │
│ Median cumulative indexing throttle time across primary shards │                   0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                     0 min │
│ Cumulative merge time of primary shards                        │                     0 min │
│ Cumulative merge count of primary shards                       │                         0 │
│ Min cumulative merge time across primary shards                │                     0 min │
│ Median cumulative merge time across primary shards             │                   0.0 min │
│ Max cumulative merge time across primary shards                │                     0 min │
│ Cumulative merge throttle time of primary shards               │                     0 min │
│ Min cumulative merge throttle time across primary shards       │                     0 min │
│ Median cumulative merge throttle time across primary shards    │                   0.0 min │
│ Max cumulative merge throttle time across primary shards       │                     0 min │
│ Cumulative refresh time of primary shards                      │   0.05068333333333333 min │
│ Cumulative refresh count of primary shards                     │                       415 │
│ Min cumulative refresh time across primary shards              │                     0 min │
│ Median cumulative refresh time across primary shards           │ 0.0004333333333333333 min │
│ Max cumulative refresh time across primary shards              │  0.014566666666666667 min │
│ Cumulative flush time of primary shards                        │   0.10021666666666666 min │
│ Cumulative flush count of primary shards                       │                       254 │
│ Min cumulative flush time across primary shards                │                     0 min │
│ Median cumulative flush time across primary shards             │                0.0031 min │
│ Max cumulative flush time across primary shards                │               0.00785 min │
│ Total Young Gen GC time                                        │                   0.088 s │
│ Total Young Gen GC count                                       │                         6 │
│ Total Old Gen GC time                                          │                       0 s │
│ Total Old Gen GC count                                         │                         0 │
│ Store size                                                     │   0.029821311123669147 GB │
│ Translog size                                                  │    0.04626855626702309 GB │
│ Heap used for segments                                         │                      0 MB │
│ Heap used for doc values                                       │                      0 MB │
│ Heap used for terms                                            │                      0 MB │
│ Heap used for norms                                            │                      0 MB │
│ Heap used for points                                           │                      0 MB │
│ Heap used for stored fields                                    │                      0 MB │
│ Segment count                                                  │                       301 │
│ Total Ingest Pipeline count                                    │                     20037 │
│ Total Ingest Pipeline time                                     │                   2.987 s │
│ Total Ingest Pipeline failed                                   │                         0 │
│ Min Throughput                                                 │            1996.24 docs/s │
│ Mean Throughput                                                │            1996.24 docs/s │
│ Median Throughput                                              │            1996.24 docs/s │
│ Max Throughput                                                 │            1996.24 docs/s │
│ 50th percentile latency                                        │     1151.2397719779983 ms │
│ 100th percentile latency                                       │     1215.1859279256314 ms │
│ 50th percentile service time                                   │     1151.2397719779983 ms │
│ 100th percentile service time                                  │     1215.1859279256314 ms │
│ error rate                                                     │                    0.00 % │
╰────────────────────────────────────────────────────────────────┴───────────────────────────╯

--- Benchmark results for package: kubernetes - END   ---
Done

[elasticmachine]

🚀 Benchmarks report

Package kubernetes 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
container_logs	125000	90909.09	-34090.91 (-27.27%)	💔

To see the full report comment with /test benchmark fullreport

[tommyers-elastic]

i don't see any issues here, but it would be useful to see diffs of the generated events for these PRs compared with say the sample event in the package, to quickly verify any missing or significantly different fields.

[gizas]

I cc a node document from the cluster:
node.json

Related to comment here for removal : #9100 (comment)

[ali786XI]

I cc a node document from the cluster:
node.json

Thanks for sharing the sample response @gizas. I have updated it accordingly. Let me know if the set of fields removed are correct.

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: ad2d1bb

History

• 💚 Build #8939 succeeded 5e3993b
• 💚 Build #8774 succeeded 0bcd90e

cc @aliabbas-elastic