Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mention Coordinated omission explicitly in performance section #37652

Merged
merged 2 commits into from
Dec 19, 2023
Merged

Mention Coordinated omission explicitly in performance section #37652

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
76471cf
Mention coordinated ommision in performance docs
maxandersen Jun 14, 2022
a445d5d
mention quarkus insights #22
maxandersen Jun 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/src/main/asciidoc/performance-measure.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
* how we measure memory usage
* how we measure startup time
* which additional flags will Quarkus apply to native-image by default
* Coordinated omission Problem in Tools

All of our tests are run on the same hardware for a given batch.
It goes without saying, but it's better when you say it.
Expand Down Expand Up @@ -234,6 +235,27 @@
Generally speaking the ones listed here are those most likely to affect performance metrics, but in the right
circumstances one could observe non-negligible impact from the other flags too.

If you're to investigate some differences in detail make sure to check what Quarkus is invoking exactly: when the build

Check warning on line 238 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.SentenceLength] Try to keep sentences to an average of 32 words or fewer.

Raw Output:
{"message": "[Quarkus.SentenceLength] Try to keep sentences to an average of 32 words or fewer.", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 238, "column": 1}}}, "severity": "INFO"}

Check warning on line 238 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.TermsWarnings] Consider using 'verify' rather than 'make sure' unless updating existing content that uses the term.

Raw Output:
{"message": "[Quarkus.TermsWarnings] Consider using 'verify' rather than 'make sure' unless updating existing content that uses the term.", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 238, "column": 53}}}, "severity": "WARNING"}
plugin is producing a native image, the full command lines are logged.


== Coordinated Omission Problem in Tools

Check warning on line 242 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.Headings] Use sentence-style capitalization in 'Coordinated Omission Problem in Tools'.

Raw Output:
{"message": "[Quarkus.Headings] Use sentence-style capitalization in 'Coordinated Omission Problem in Tools'.", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 242, "column": 4}}}, "severity": "INFO"}

When measuring performance of a framework like Quarkus the latency experience by users are especially interesting and for that there are many different tools. Unfortunately, many fail to measure the latency correctly and instead fall short and create the Coordinate Omission problem. Meaning tools fails to acoomodate for delays to submit new requests when system is under load and aggregate these numbers making the latency and throughput numbers very misleading.

Check warning on line 244 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'acoomodate'?

Raw Output:
{"message": "[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'acoomodate'?", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 244, "column": 308}}}, "severity": "WARNING"}

A good walkthrough of the issue is https://www.youtube.com/watch?v=lJ8ydIuPFeU[this video] where Gil Tene the author of wrk2 explains the issue and https://www.youtube.com/watch?v=xdG8b9iDYbE[Quarkus Insights #22] have John O'Hara from Quarkus performance team show how it can show up.

Check warning on line 246 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'walkthrough'?

Raw Output:
{"message": "[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'walkthrough'?", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 246, "column": 8}}}, "severity": "WARNING"}

Check warning on line 246 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'Tene'?

Raw Output:
{"message": "[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'Tene'?", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 246, "column": 102}}}, "severity": "WARNING"}

Although that video and related papers and articles date all back to 2015 then even today you will find tools that fall short with the coordinated oission problem

Check warning on line 248 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'oission'?

Raw Output:
{"message": "[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'oission'?", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 248, "column": 148}}}, "severity": "WARNING"}

Tools that at current time of writing is known to excert the problem and should NOT be used for measuring latency/throughput (it may be used for other things):

Check warning on line 250 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'excert'?

Raw Output:
{"message": "[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'excert'?", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 250, "column": 51}}}, "severity": "WARNING"}

Check warning on line 250 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.TermsWarnings] Consider using 'might (for possiblity)' or 'can (for ability)' rather than 'may' unless updating existing content that uses the term.

Raw Output:
{"message": "[Quarkus.TermsWarnings] Consider using 'might (for possiblity)' or 'can (for ability)' rather than 'may' unless updating existing content that uses the term.", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 250, "column": 130}}}, "severity": "WARNING"}

* JMeter

Check warning on line 252 in docs/src/main/asciidoc/performance-measure.adoc

View workflow job for this annotation

GitHub Actions / Linting with Vale 

[vale] reported by reviewdog 🐶
[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'JMeter'?

Raw Output:
{"message": "[Quarkus.Spelling] Use correct American English spelling. Did you really mean 'JMeter'?", "location": {"path": "docs/src/main/asciidoc/performance-measure.adoc", "range": {"start": {"line": 252, "column": 4}}}, "severity": "WARNING"}
* wrk

Tools that are known to not be affected are:

* https://github.com/giltene/wrk2[wrk2]
* https://hyperfoil.io[HyperFoil]

Mind you, the tools are not better than your own understanding of what they measure thus even when using `wrk2` or `hyperfoil` make sure you understand what the numbers mean.

Loading