Mention coordinated omission in performance docs #26129
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks, I'll have a close look tomorrow |
geoand
reviewed
Jun 15, 2022
| * how we measure memory usage | ||
| * how we measure startup time | ||
| * which additional flags will Quarkus apply to native-image by default | ||
|
|
||
| All of our tests are run on the same hardware for a given batch. | ||
| It goes without saying but it's better when you say it. | ||
|
|
||
| == Coordinated Omission Problem in Tools | ||
|
|
||
| When measuring performance of a framework like Quarkus the latency experience by users are especially interesting and for that there are many different tools. Unfortunately, many fail to measure the latency correctly and instead fall short and create the Coordinate Omission problem. Meaning tools fails to acoomodate for delays to submit new requests when system is under load and aggregate these numbers making the latency and throughput numbers very misleading. |
There was a problem hiding this comment.
Suggested change
| When measuring performance of a framework like Quarkus the latency experience by users are especially interesting and for that there are many different tools. Unfortunately, many fail to measure the latency correctly and instead fall short and create the Coordinate Omission problem. Meaning tools fails to acoomodate for delays to submit new requests when system is under load and aggregate these numbers making the latency and throughput numbers very misleading. | |
| When measuring performance of a framework like Quarkus, the latency numbers experienced by users are especially interesting and for measuring that, many different tools exist. Unfortunately, many fail to measure the latency correctly and instead suffer from the Coordinate Omission problem - meaning tools fails to accomodate for delays to submit new requests when system is under load and aggregate these numbers, thus rendering their reported latency and throughput numbers misleading. |
geoand
reviewed
Jun 15, 2022
|
|
||
| A good walkthrough of the issue is https://www.youtube.com/watch?v=lJ8ydIuPFeU[this video] where Gil Tene the author of wrk2 explains the issue and https://www.youtube.com/watch?v=xdG8b9iDYbE[Quarkus Insights #22] have John O'Hara from Quarkus performance team show how it can show up. | ||
|
|
||
| Although that video and related papers and articles date all back to 2015 then even today you will find tools that fall short with the coordinated oission problem |
There was a problem hiding this comment.
Suggested change
| Although that video and related papers and articles date all back to 2015 then even today you will find tools that fall short with the coordinated oission problem | |
| Although that video and related papers and articles all date back to 2015, unfortunately many of the widely used tools even today suffer from Coordinated Omission problem. |
gsmet
changed the title
Sanne
reviewed
Jun 22, 2022
| @@ -9,13 +9,34 @@ include::./attributes.adoc[] | |||
|
|
|||
| This guide covers: | |||
|
|
|||
| * Coordinated omission Problem in Tools | |||
There was a problem hiding this comment.
I wouldn't have this as first bullet. In fact I don't think it belongs in this guide - no strong opinion about that, but at least let's not make this aspect too distracting from the primary content.
|
@maxandersen friendly ping, see especially @Sanne 's comment above. |
|
Closing for lack of activity. |
|
🙈 The PR is closed and the preview is expired. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
trying to explain to users that when doing performance measurements be aware of coordinated ommision in various tools but noticed we did not mention it in our own performance guide.
here is a suggestion on a way we could inform/educate. feedback welcome!