Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[O11y][Kubernetes] Rally benchmark kubernetes.state_pod #9136

Merged
merged 3 commits into from
Feb 15, 2024
Merged

[O11y][Kubernetes] Rally benchmark kubernetes.state_pod #9136

merged 3 commits into from
Feb 15, 2024

Conversation

ali786XI
Copy link
Contributor

Proposed commit message

  • This PR adds benchmarking templates to the state_pod data stream of Kubernetes

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.

How to test this PR locally

Run this command from package root

  • elastic-package benchmark rally --benchmark state_pod-benchmark -v
  • elastic-package benchmark stream --benchmark state_pod-benchmark -v

Related issues

Screenshots

--- Benchmark results for package: kubernetes - START ---
╭────────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                               │
├────────────────────────┬───────────────────────────────────────────────────────────┤
│ benchmark              │                                       state_pod-benchmark │
│ description            │      Benchmark 20000 kubernetes.state_pod events ingested │
│ run ID                 │                      4c270392-77a3-4c11-875a-4ee173e29829 │
│ package                │                                                kubernetes │
│ start ts (s)           │                                                1707812774 │
│ end ts (s)             │                                                1707812820 │
│ duration               │                                                       46s │
│ generated corpora file │ /root/.elastic-package/tmp/rally_corpus/corpus-1190356990 │
╰────────────────────────┴───────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────╮
│ parameters                                                             │
├─────────────────────────────────┬──────────────────────────────────────┤
│ package version                 │                               1.56.0 │
│ data_stream.name                │                            state_pod │
│ corpora.generator.total_events  │                                20000 │
│ corpora.generator.template.path │ ./statepod-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                      │
│ corpora.generator.template.type │                               gotext │
│ corpora.generator.config.path   │      ./statepod-benchmark/config.yml │
│ corpora.generator.config.raw    │                                map[] │
│ corpora.generator.fields.path   │      ./statepod-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                                map[] │
╰─────────────────────────────────┴──────────────────────────────────────╯
╭───────────╮
│ cluster i │
│ nfo       │
├───────┬───┤
│ name  │   │
│ nodes │ 0 │
╰───────┴───╯
╭───────────────────────────────────────╮
│ disk usage for index metrics-kubernet │
│ es.state_pod-ep (for all fields)      │
├──────────────────────────────┬────────┤
│ total                        │ 5.9 MB │
│ inverted_index.total         │ 928 kB │
│ inverted_index.stored_fields │ 3.5 MB │
│ inverted_index.doc_values    │ 1.1 MB │
│ inverted_index.points        │ 438 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭────────────────────────────────────────────────────────────────────────────────────────╮
│ pipeline metrics-kubernetes.state_pod-1.56.0 stats in node OQo_iVeVRSeYfWHrhZJ5_g      │
├────────────────────────────────────────────────┬───────────────────────────────────────┤
│ Totals                                         │ Count: 20000 | Failed: 0 | Time: 29ms │
│ pipeline (global@custom)                       │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics@custom)                      │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics-kubernetes@custom)           │  Count: 20000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics-kubernetes.state_pod@custom) │  Count: 20000 | Failed: 0 | Time: 4ms │
╰────────────────────────────────────────────────┴───────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                 │
├────────────────────────────────────────────────────────────────┬────────────────────────────┤
│ Cumulative indexing time of primary shards                     │                8.46765 min │
│ Min cumulative indexing time across primary shards             │                      0 min │
│ Median cumulative indexing time across primary shards          │    0.08625833333333334 min │
│ Max cumulative indexing time across primary shards             │     2.1194333333333333 min │
│ Cumulative indexing throttle time of primary shards            │                      0 min │
│ Min cumulative indexing throttle time across primary shards    │                      0 min │
│ Median cumulative indexing throttle time across primary shards │                    0.0 min │
│ Max cumulative indexing throttle time across primary shards    │                      0 min │
│ Cumulative merge time of primary shards                        │      4.946816666666667 min │
│ Cumulative merge count of primary shards                       │                       2783 │
│ Min cumulative merge time across primary shards                │                      0 min │
│ Median cumulative merge time across primary shards             │                    0.0 min │
│ Max cumulative merge time across primary shards                │     0.6093833333333334 min │
│ Cumulative merge throttle time of primary shards               │                 0.0598 min │
│ Min cumulative merge throttle time across primary shards       │                      0 min │
│ Median cumulative merge throttle time across primary shards    │                    0.0 min │
│ Max cumulative merge throttle time across primary shards       │                 0.0598 min │
│ Cumulative refresh time of primary shards                      │               117.6942 min │
│ Cumulative refresh count of primary shards                     │                      98008 │
│ Min cumulative refresh time across primary shards              │                      0 min │
│ Median cumulative refresh time across primary shards           │    0.04891666666666667 min │
│ Max cumulative refresh time across primary shards              │               37.39995 min │
│ Cumulative flush time of primary shards                        │     226.82090000000002 min │
│ Cumulative flush count of primary shards                       │                      86269 │
│ Min cumulative flush time across primary shards                │ 1.6666666666666667e-05 min │
│ Median cumulative flush time across primary shards             │   0.004808333333333333 min │
│ Max cumulative flush time across primary shards                │      47.68103333333333 min │
│ Total Young Gen GC time                                        │                    0.048 s │
│ Total Young Gen GC count                                       │                          4 │
│ Total Old Gen GC time                                          │                        0 s │
│ Total Old Gen GC count                                         │                          0 │
│ Store size                                                     │     0.24722508061677217 GB │
│ Translog size                                                  │   0.0002751322463154793 GB │
│ Heap used for segments                                         │                       0 MB │
│ Heap used for doc values                                       │                       0 MB │
│ Heap used for terms                                            │                       0 MB │
│ Heap used for norms                                            │                       0 MB │
│ Heap used for points                                           │                       0 MB │
│ Heap used for stored fields                                    │                       0 MB │
│ Segment count                                                  │                        642 │
│ Total Ingest Pipeline count                                    │                      20037 │
│ Total Ingest Pipeline time                                     │                    1.444 s │
│ Total Ingest Pipeline failed                                   │                          0 │
│ Min Throughput                                                 │            24548.56 docs/s │
│ Mean Throughput                                                │            24548.56 docs/s │
│ Median Throughput                                              │            24548.56 docs/s │
│ Max Throughput                                                 │            24548.56 docs/s │
│ 50th percentile latency                                        │       743.7380615156144 ms │
│ 100th percentile latency                                       │       1083.487429190427 ms │
│ 50th percentile service time                                   │       743.7380615156144 ms │
│ 100th percentile service time                                  │       1083.487429190427 ms │
│ error rate                                                     │                     0.00 % │
╰────────────────────────────────────────────────────────────────┴────────────────────────────╯

--- Benchmark results for package: kubernetes - END   ---
Done

@ali786XI ali786XI added enhancement New feature or request Integration:kubernetes Kubernetes labels Feb 13, 2024
@ali786XI ali786XI self-assigned this Feb 13, 2024
@ali786XI ali786XI mentioned this pull request Feb 13, 2024
Closed
29 tasks
@elasticmachine
Copy link

elasticmachine commented Feb 13, 2024 •
edited
Loading

🚀 Benchmarks report

Package kubernetes 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
container_logs 125000 55555.56 -69444.44 (-55.56%) 💔

To see the full report comment with /test benchmark fullreport

@ali786XI ali786XI marked this pull request as ready for review February 13, 2024 09:39
@ali786XI ali786XI requested a review from a team as a code owner February 13, 2024 09:39
gizas
gizas reviewed Feb 14, 2024
View reviewed changes
packages/kubernetes/_dev/benchmark/rally/statepod-benchmark/template.ndjson
"name": "demo-deployment-{{ $rangeofid }}"
},
"namespace_labels": {
"kubernetes_io/metadata_name": "demo-{{ $rangeofid }}"
Copy link
Contributor

@gizas gizas Feb 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont see a definition of rangeofid in config.go like here https://github.com/elastic/integrations/blob/main/packages/kubernetes/_dev/benchmark/rally/pod-benchmark/config.yml#L36

Is not it better in order to have specific range?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gizas Thanks for pointing out. I missed it.

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @aliabbas-elastic

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No Coverage information No data about Coverage
No Duplication information No data about Duplication

See analysis details on SonarQube

@ali786XI ali786XI merged commit e300934 into elastic:main Feb 15, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gizas gizas gizas approved these changes

@constanca-m constanca-m constanca-m approved these changes

@harnish-crest harnish-crest harnish-crest approved these changes

Assignees

@ali786XI ali786XI

Labels
enhancement New feature or request Integration:kubernetes Kubernetes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ali786XI @elasticmachine @gizas @harnish-crest @constanca-m