From 76471cfeef8c82ebc14452485bd80d17682849b7 Mon Sep 17 00:00:00 2001 From: Max Rydahl Andersen Date: Tue, 14 Jun 2022 17:16:51 +0200 Subject: [PATCH 1/2] Mention coordinated ommision in performance docs --- .../main/asciidoc/performance-measure.adoc | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/src/main/asciidoc/performance-measure.adoc b/docs/src/main/asciidoc/performance-measure.adoc index f18d708c94edd..050cdb3fec0ac 100644 --- a/docs/src/main/asciidoc/performance-measure.adoc +++ b/docs/src/main/asciidoc/performance-measure.adoc @@ -11,6 +11,7 @@ include::_attributes.adoc[] This guide covers: +* Coordinated omission Problem in Tools * how we measure memory usage * how we measure startup time * which additional flags will Quarkus apply to native-image by default @@ -18,6 +19,26 @@ This guide covers: All of our tests are run on the same hardware for a given batch. It goes without saying, but it's better when you say it. +== Coordinated Omission Problem in Tools + +When measuring performance of a framework like Quarkus the latency experience by users are especially interesting and for that there are many different tools. Unfortunately, many fail to measure the latency correctly and instead fall short and create the Coordinate Omission problem. Meaning tools fails to acoomodate for delays to submit new requests when system is under load and aggregate these numbers making the latency and throughput numbers very misleading. + +A good walkthrough of the issue is https://www.youtube.com/watch?v=lJ8ydIuPFeU[this video] where Gil Tene the author of wrk2 explains the issue. + +Although that video and related papers and articles date all back to 2015 then even today you will find tools that fall short with the coordinated oission problem + +Tools that at current time of writing is known to excert the problem and should NOT be used for measuring latency/throughput (it may be used for other things): + + * JMeter + * wrk + +Tools that are known to not be affected are: + + * https://github.com/giltene/wrk2[wrk2] + * https://hyperfoil.io[HyperFoil] + +Mind you, the tools are not better than your own understanding of what they measure thus even when using `wrk2` or `hyperfoil` make sure you understand what the numbers mean. + == How do we measure memory usage When measuring the footprint of a Quarkus application, we measure https://en.wikipedia.org/wiki/Resident_set_size[Resident Set Size (RSS)] From a445d5deffa08fd94a3ef7f75dec97e0287bd98d Mon Sep 17 00:00:00 2001 From: Max Rydahl Andersen Date: Tue, 14 Jun 2022 19:16:42 +0200 Subject: [PATCH 2/2] mention quarkus insights #22 --- .../main/asciidoc/performance-measure.adoc | 43 ++++++++++--------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/docs/src/main/asciidoc/performance-measure.adoc b/docs/src/main/asciidoc/performance-measure.adoc index 050cdb3fec0ac..cd05214955e56 100644 --- a/docs/src/main/asciidoc/performance-measure.adoc +++ b/docs/src/main/asciidoc/performance-measure.adoc @@ -11,34 +11,14 @@ include::_attributes.adoc[] This guide covers: -* Coordinated omission Problem in Tools * how we measure memory usage * how we measure startup time * which additional flags will Quarkus apply to native-image by default +* Coordinated omission Problem in Tools All of our tests are run on the same hardware for a given batch. It goes without saying, but it's better when you say it. -== Coordinated Omission Problem in Tools - -When measuring performance of a framework like Quarkus the latency experience by users are especially interesting and for that there are many different tools. Unfortunately, many fail to measure the latency correctly and instead fall short and create the Coordinate Omission problem. Meaning tools fails to acoomodate for delays to submit new requests when system is under load and aggregate these numbers making the latency and throughput numbers very misleading. - -A good walkthrough of the issue is https://www.youtube.com/watch?v=lJ8ydIuPFeU[this video] where Gil Tene the author of wrk2 explains the issue. - -Although that video and related papers and articles date all back to 2015 then even today you will find tools that fall short with the coordinated oission problem - -Tools that at current time of writing is known to excert the problem and should NOT be used for measuring latency/throughput (it may be used for other things): - - * JMeter - * wrk - -Tools that are known to not be affected are: - - * https://github.com/giltene/wrk2[wrk2] - * https://hyperfoil.io[HyperFoil] - -Mind you, the tools are not better than your own understanding of what they measure thus even when using `wrk2` or `hyperfoil` make sure you understand what the numbers mean. - == How do we measure memory usage When measuring the footprint of a Quarkus application, we measure https://en.wikipedia.org/wiki/Resident_set_size[Resident Set Size (RSS)] @@ -258,3 +238,24 @@ circumstances one could observe non-negligible impact from the other flags too. If you're to investigate some differences in detail make sure to check what Quarkus is invoking exactly: when the build plugin is producing a native image, the full command lines are logged. + +== Coordinated Omission Problem in Tools + +When measuring performance of a framework like Quarkus the latency experience by users are especially interesting and for that there are many different tools. Unfortunately, many fail to measure the latency correctly and instead fall short and create the Coordinate Omission problem. Meaning tools fails to acoomodate for delays to submit new requests when system is under load and aggregate these numbers making the latency and throughput numbers very misleading. + +A good walkthrough of the issue is https://www.youtube.com/watch?v=lJ8ydIuPFeU[this video] where Gil Tene the author of wrk2 explains the issue and https://www.youtube.com/watch?v=xdG8b9iDYbE[Quarkus Insights #22] have John O'Hara from Quarkus performance team show how it can show up. + +Although that video and related papers and articles date all back to 2015 then even today you will find tools that fall short with the coordinated oission problem + +Tools that at current time of writing is known to excert the problem and should NOT be used for measuring latency/throughput (it may be used for other things): + + * JMeter + * wrk + +Tools that are known to not be affected are: + + * https://github.com/giltene/wrk2[wrk2] + * https://hyperfoil.io[HyperFoil] + +Mind you, the tools are not better than your own understanding of what they measure thus even when using `wrk2` or `hyperfoil` make sure you understand what the numbers mean. +