Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rally benchmark aws.sqs #8429

Merged
merged 4 commits into from
Nov 16, 2023
Merged

Conversation

aspacca
Copy link
Contributor

@aspacca aspacca commented Nov 8, 2023

Enhancement

Proposed commit message

Add artifacts for elastic-package rally benchmark

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
    - [ ] I have added an entry to my package's changelog.yml file.
    - [ ] I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

From aws package root (remember to bring up the elastic-package stack before):
./elastic-package benchmark rally --benchmark sqs-benchmark -v

Related issues

Screenshots

--- Benchmark results for package: aws - START ---
╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
│ info                                                                                             │
├────────────────────────┬─────────────────────────────────────────────────────────────────────────┤
│ benchmark              │                                                           sqs-benchmark │
│ description            │                                 Benchmark 20000 aws.sqs events ingested │
│ run ID                 │                                    311d2fb3-72aa-416b-a24f-52a5bd2634c5 │
│ package                │                                                                     aws │
│ start ts (s)           │                                                              1699427831 │
│ end ts (s)             │                                                              1699427864 │
│ duration               │                                                                     33s │
│ generated corpora file │ /Users/andreaspacca/.elastic-package/tmp/rally_corpus/corpus-4122319852 │
╰────────────────────────┴─────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────────────╮
│ parameters                                                        │
├─────────────────────────────────┬─────────────────────────────────┤
│ package version                 │                           2.8.5 │
│ data_stream.name                │                             sqs │
│ corpora.generator.total_events  │                           20000 │
│ corpora.generator.template.path │ ./sqs-benchmark/template.ndjson │
│ corpora.generator.template.raw  │                                 │
│ corpora.generator.template.type │                          gotext │
│ corpora.generator.config.path   │      ./sqs-benchmark/config.yml │
│ corpora.generator.config.raw    │                           map[] │
│ corpora.generator.fields.path   │      ./sqs-benchmark/fields.yml │
│ corpora.generator.fields.raw    │                           map[] │
╰─────────────────────────────────┴─────────────────────────────────╯
╭──────────────────────────────────────────╮
│ cluster info                             │
├───────┬──────────────────────────────────┤
│ name  │ af6364c642944c65a6171bea0dcc9e2c │
│ nodes │                                3 │
╰───────┴──────────────────────────────────╯
╭─────────────────────────────────────────────────╮
│ data stream stats                               │
├────────────────────────────┬────────────────────┤
│ data stream                │ metrics-aws.sqs-ep │
│ approx total docs ingested │              20000 │
│ backing indices            │                  1 │
│ store size bytes           │           11467292 │
│ maximum ts (ms)            │      1699431430608 │
╰────────────────────────────┴────────────────────╯
╭───────────────────────────────────────╮
│ disk usage for index .ds-metrics-aws. │
│ sqs-ep-2023.11.08-000001 (for all fie │
│ lds)                                  │
├──────────────────────────────┬────────┤
│ total                        │ 4.0 MB │
│ inverted_index.total         │ 466 kB │
│ inverted_index.stored_fields │ 2.9 MB │
│ inverted_index.doc_values    │ 514 kB │
│ inverted_index.points        │ 198 kB │
│ inverted_index.norms         │    0 B │
│ inverted_index.term_vectors  │    0 B │
│ inverted_index.knn_vectors   │    0 B │
╰──────────────────────────────┴────────╯
╭───────────────────────────────────────────────────────────────────────────╮
│ pipeline metrics-aws.sqs-2.8.5 stats in node DoUgUTZiTyGl4CQZv8HUqQ       │
├───────────────────────────────────┬───────────────────────────────────────┤
│ Totals                            │ Count: 7500 | Failed: 0 | Time: 126ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 14ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 12ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 12ms │
│ remove ()                         │   Count: 7500 | Failed: 0 | Time: 4ms │
│ pipeline (metrics-aws.sqs@custom) │   Count: 7500 | Failed: 0 | Time: 1ms │
╰───────────────────────────────────┴───────────────────────────────────────╯
╭───────────────────────────────────────────────────────────────────────────╮
│ pipeline metrics-aws.sqs-2.8.5 stats in node Gx79GULoQdizRMGIoKEEjQ       │
├───────────────────────────────────┬───────────────────────────────────────┤
│ Totals                            │ Count: 7500 | Failed: 0 | Time: 131ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 16ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 14ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 12ms │
│ rename ()                         │  Count: 7500 | Failed: 0 | Time: 13ms │
│ remove ()                         │   Count: 7500 | Failed: 0 | Time: 4ms │
│ pipeline (metrics-aws.sqs@custom) │    Count: 7500 | Failed: 0 | Time: 0s │
╰───────────────────────────────────┴───────────────────────────────────────╯
╭──────────────────────────────────────────────────────────────────────────╮
│ pipeline metrics-aws.sqs-2.8.5 stats in node WmW84qX4Qo2qWjOO43hIIA      │
├───────────────────────────────────┬──────────────────────────────────────┤
│ Totals                            │ Count: 5000 | Failed: 0 | Time: 83ms │
│ rename ()                         │  Count: 5000 | Failed: 0 | Time: 9ms │
│ rename ()                         │  Count: 5000 | Failed: 0 | Time: 8ms │
│ rename ()                         │  Count: 5000 | Failed: 0 | Time: 9ms │
│ rename ()                         │  Count: 5000 | Failed: 0 | Time: 8ms │
│ rename ()                         │  Count: 5000 | Failed: 0 | Time: 9ms │
│ rename ()                         │  Count: 5000 | Failed: 0 | Time: 8ms │
│ rename ()                         │  Count: 5000 | Failed: 0 | Time: 8ms │
│ rename ()                         │  Count: 5000 | Failed: 0 | Time: 8ms │
│ rename ()                         │  Count: 5000 | Failed: 0 | Time: 8ms │
│ remove ()                         │  Count: 5000 | Failed: 0 | Time: 2ms │
│ pipeline (metrics-aws.sqs@custom) │  Count: 5000 | Failed: 0 | Time: 1ms │
╰───────────────────────────────────┴──────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ rally stats                                                                                 │
├────────────────────────────────────────────────────────────────┬────────────────────────────┤
│ Cumulative indexing time of primary shards                     │   0.022766666666666668 min │
│ Min cumulative indexing time across primary shards             │                      0 min │
│ Median cumulative indexing time across primary shards          │                      0 min │
│ Max cumulative indexing time across primary shards             │   0.022516666666666667 min │
│ Cumulative indexing throttle time of primary shards            │                      0 min │
│ Min cumulative indexing throttle time across primary shards    │                      0 min │
│ Median cumulative indexing throttle time across primary shards │                      0 min │
│ Max cumulative indexing throttle time across primary shards    │                      0 min │
│ Cumulative merge time of primary shards                        │                      0 min │
│ Cumulative merge count of primary shards                       │                          0 │
│ Min cumulative merge time across primary shards                │                      0 min │
│ Median cumulative merge time across primary shards             │                      0 min │
│ Max cumulative merge time across primary shards                │                      0 min │
│ Cumulative merge throttle time of primary shards               │                      0 min │
│ Min cumulative merge throttle time across primary shards       │                      0 min │
│ Median cumulative merge throttle time across primary shards    │                      0 min │
│ Max cumulative merge throttle time across primary shards       │                      0 min │
│ Cumulative refresh time of primary shards                      │                 0.0091 min │
│ Cumulative refresh count of primary shards                     │                        123 │
│ Min cumulative refresh time across primary shards              │                      0 min │
│ Median cumulative refresh time across primary shards           │                      0 min │
│ Max cumulative refresh time across primary shards              │   0.008883333333333333 min │
│ Cumulative flush time of primary shards                        │                0.00185 min │
│ Cumulative flush count of primary shards                       │                         20 │
│ Min cumulative flush time across primary shards                │ 1.6666666666666667e-05 min │
│ Median cumulative flush time across primary shards             │ 1.6666666666666667e-05 min │
│ Max cumulative flush time across primary shards                │  0.0010333333333333334 min │
│ Total Young Gen GC time                                        │                        0 s │
│ Total Young Gen GC count                                       │                          0 │
│ Total Old Gen GC time                                          │                        0 s │
│ Total Old Gen GC count                                         │                          0 │
│ Store size                                                     │    0.010847736150026321 GB │
│ Translog size                                                  │     0.04592067189514637 GB │
│ Heap used for segments                                         │                       0 MB │
│ Heap used for doc values                                       │                       0 MB │
│ Heap used for terms                                            │                       0 MB │
│ Heap used for norms                                            │                       0 MB │
│ Heap used for points                                           │                       0 MB │
│ Heap used for stored fields                                    │                       0 MB │
│ Segment count                                                  │                         15 │
│ Total Ingest Pipeline count                                    │                      20000 │
│ Total Ingest Pipeline time                                     │                      0.9 s │
│ Total Ingest Pipeline failed                                   │                          0 │
│ Min Throughput                                                 │              709.85 docs/s │
│ Mean Throughput                                                │              709.85 docs/s │
│ Median Throughput                                              │              709.85 docs/s │
│ Max Throughput                                                 │              709.85 docs/s │
│ 50th percentile latency                                        │             3289.259417 ms │
│ 100th percentile latency                                       │      4276.0719579999995 ms │
│ 50th percentile service time                                   │             3289.259417 ms │
│ 100th percentile service time                                  │      4276.0719579999995 ms │
│ error rate                                                     │                     0.00 % │
╰────────────────────────────────────────────────────────────────┴────────────────────────────╯

--- Benchmark results for package: aws - END   ---
Done

@aspacca aspacca requested review from a team as code owners November 8, 2023 02:50
@aspacca aspacca self-assigned this Nov 8, 2023
@elasticmachine
Copy link

elasticmachine commented Nov 8, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-11-14T04:38:45.182+0000

  • Duration: 84 min 51 sec

Test stats 🧪

Test Results
Failed 0
Passed 223
Skipped 3
Total 226

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Nov 8, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (17/17) 💚
Files 94.444% (17/18) 👎 -1.746
Classes 94.444% (17/18) 👎 -1.746
Methods 89.701% (270/301) 👎 -2.756
Lines 86.083% (7571/8795) 👎 -2.731
Conditionals 100.0% (0/0) 💚

"ephemeral_id": "cdaaaabb-be7e-432f-816b-bda019fd7c15",
"version": "8.3.2"
},
"elastic_agent": { "id": "2d4b09d0-cdb6-445e-ac3f-6415f87b9864", "version": "8.3.2", "snapshot": false },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having the templates "expanded" with all fields would make it more readable / more reviewable. The problem I hit when I tried to do it myself is that tis is not "valid" JSON so pretty does not do it directly for you without removing the template parts.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. I know, not sure we can tweak this problem completely

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One way of partially solving is putting at least the examples we have at the moment "expanded" by doing the manual work. That will make it easier for other just for example copy the elastic-agent part and arleady have it in the right format :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@endorama a while ago proposed to have some importable templates for this kind of "common boilerplate".
it can be done just with a function indeed, something like

...
"agent": {{ evenAgentMetadata }},
...

or even only

{{ evenAgentMetadata }},

also for event: this would require to override the configs with some runtime values

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like {{ evenAgentMetadata }}, as it makes it really easy to add these blocks. We could have the same for host and k8s and then even have a some events with one, some with the other based on a if rand() % 2 { {{host}} } else { {{k8s}} }}.

I assume this would be some "hardcoded" variables somewhere in the tooling everyone could use?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this would be some "hardcoded" variables somewhere in the tooling everyone could use?

yes, we need to use some variables (not for every field, but a few of them): either we add a way to pass "runtime fields and config" to the generator package, or we could simply "hack" inside elastic-package dynamically adding the fields and config we need for this helpers once we have parsed the yaml files

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a Github issue or similar to track this idea?

{{- $queueName := generate "QueueName" -}}
{{- $timestamp := generate "timestamp" -}}
{
"@timestamp": "{{ $timestamp.Format "2006-01-02T15:04:05.999999Z07:00" }}",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this also work if $timestamp.Format '2006-01-02T15:04:05.999999Z07:00' would be used? Mainly asking in the context of the pretty question below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's gotext template: you can use:

$timestamp.Format `2006-01-02T15:04:05.999999Z07:00`

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this solve the JSON pretty problem?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ignoring the first 2 lines)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this solve the JSON pretty problem?

only for that specific line
json with integer like: {"pc": {{.pct}} }, or the end/if syntax will still break it

Copy link
Member

@ruflin ruflin Nov 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, there is too much other "non json" in there. The parallel discussion we had around validating the outcome of the template might work better to get quick feedback if there is a typo inside.

@aspacca
Copy link
Contributor Author

aspacca commented Nov 14, 2023

/fest

@aspacca aspacca merged commit 3087d39 into elastic:main Nov 16, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants