Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Histogram Record Method Benchmark #2754

Merged
merged 13 commits into from
Feb 4, 2022
Merged

Histogram Record Method Benchmark #2754

merged 13 commits into from
Feb 4, 2022

Conversation

mic-max
Copy link
Contributor

@mic-max mic-max commented Dec 17, 2021

Changes

Benchmarking the Histogram.Record function for various number of explicitly provided bounds for both long and double values.

Method BoundCount Mean Error StdDev Allocated
HistogramLongHotPath 10 55.44 ns 0.211 ns 0.187 ns -
HistogramDoubleHotPath 10 55.69 ns 0.129 ns 0.107 ns -
HistogramLongHotPath 20 57.71 ns 0.297 ns 0.278 ns -
HistogramDoubleHotPath 20 58.10 ns 0.117 ns 0.110 ns -
HistogramLongHotPath 50 65.21 ns 0.356 ns 0.333 ns -
HistogramDoubleHotPath 50 66.34 ns 0.381 ns 0.356 ns -
HistogramLongHotPath 100 79.49 ns 0.804 ns 0.753 ns -
HistogramDoubleHotPath 100 85.77 ns 0.947 ns 0.840 ns -

Aside: The reason this was added was since I was investigating whether changing the bucket finding algorithm from a linear one to a binary search one would improve the speed. It only did at ~100 boundaries. So opted not to make the change. Here is the results of that benchmark which is not included in this PR.

Current: Linear

Method BoundCount Mean Error StdDev Allocated
HistogramLongCustomBoundsHotPath 10 42.78 ns 0.159 ns 0.141 ns -
HistogramLongCustomBoundsHotPath 20 45.02 ns 0.158 ns 0.148 ns -
HistogramLongCustomBoundsHotPath 50 53.00 ns 0.242 ns 0.215 ns -
HistogramLongCustomBoundsHotPath 100 69.03 ns 1.399 ns 1.555 ns -

Proposed: Binary

Method BoundCount Mean Error StdDev Allocated
HistogramLongCustomBoundsHotPath 10 51.56 ns 0.161 ns 0.151 ns -
HistogramLongCustomBoundsHotPath 20 57.33 ns 0.156 ns 0.146 ns -
HistogramLongCustomBoundsHotPath 50 66.93 ns 0.219 ns 0.205 ns -
HistogramLongCustomBoundsHotPath 100 72.99 ns 0.219 ns 0.183 ns -
  • Appropriate CHANGELOG.md updated for non-trivial changes
  • Design discussion issue #
  • Changes in public API reviewed

Sorry, something went wrong.

@codecov
Copy link

codecov bot commented Dec 28, 2021

Codecov Report

Merging #2754 (33c402d) into main (aca7efc) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #2754   +/-   ##
=======================================
  Coverage   83.75%   83.75%           
=======================================
  Files         250      250           
  Lines        8882     8882           
=======================================
  Hits         7439     7439           
  Misses       1443     1443           
Impacted Files Coverage Δ
...nTelemetry/Internal/OpenTelemetrySdkEventSource.cs 71.29% <0.00%> (-0.93%) ⬇️
...emetry.Api/Internal/OpenTelemetryApiEventSource.cs 82.35% <0.00%> (+2.94%) ⬆️

@mic-max mic-max marked this pull request as ready for review December 28, 2021 20:56
@mic-max mic-max requested a review from a team December 28, 2021 20:56
@@ -247,12 +247,18 @@ internal void Update(double number)
case AggregationType.Histogram:
{
int i;
for (i = 0; i < this.histogramBuckets.ExplicitBounds.Length; i++)
if (double.IsNaN(number))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting... 🤔 makes me wonder if NaN should be an error case that we just log instead of recording.

Copy link
Contributor Author

@mic-max mic-max Dec 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, I'm not sure. I was trying to fix the Prometheus exporter test for PrometheusSerializerTests.HistogramNaN since I was trying to compare by using the == operator and also double.NaN returns False for it appears all uses of <, >, ==, etc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, doesn't make sense to me that we should increment the largest bucket... why not the smallest? or even a random one?

Interestingly, I checked the Java SDK, and it turns out they too increment the last bucket. Wonder if this is just accidental because I don't see anything in the spec.

Though I looked at that prometheus test and noted that NaN is a valid value in the prometheus format.

Maybe something @reyang could shed some light on.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SDK Specification gives certain flexibility to the SDK implementations for perf consideration.

Prometheus Exporter spec (which is still Experimental) hasn't covered NaN histogram yet, further clarifications are needed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If perf is what drives our decision then it would make sense to try and eliminate the if (double.IsNaN(number)) check.

If we settle on using Array.BinarySearch then ~Array.BinarySearch(buckets, double.NaN) equals 0. Seems it would be slightly more efficient to increment the first bucket in this case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though, the bummer of aggregating a NaN value is that it not only messes up one of the bucket counts, it also clobbers the sum.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If perf is what drives our decision then it would make sense to try and eliminate the if (double.IsNaN(number)) check.

+1

This is also called out in the spec:

It is unspecified how the SDK should handle the input limits. The SDK authors MAY leverage/follow the language runtime behavior for better performance, rather than perform a check on each value coming from the API.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though, the bummer of aggregating a NaN value is that it not only messes up one of the bucket counts, it also clobbers the sum.

In addition to NaN, other values could result in NaN as well, for example (note that currently histogram does not support negative value):

histogram.Record(double.PositiveInfinity);
histogram.Record(double.NegativeInfinity);

{
// If `number` is not a histogram bound, the result will be negative
// The bitwise complement of the returned value is the insertion index
i = Array.BinarySearch(this.histogramBuckets.ExplicitBounds, number);
Copy link
Member

@reyang reyang Jan 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For histogram with small number of bounds, this extra invocation could turn out to be slower.

We need to have a perf benchmark to show the numbers (especially for low buckets).

Refer to the prom-client and see how low/high buckets are handled differently:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll write a self-contained benchmark that compares the existing linear search implementation with the binary search one and run it with a range of boundary counts.

I agree that linear is likely faster for a small number of boundaries, but eventually the binary search will outperform it. Then I can a boundary number where anything less uses linear, and anything more uses binary.

I'll generate the random numbers so that each index gets an equal amount of finds.
Ex: If the sorted array is [ 20, 30, 40 ] then the random numbers will be generated like random.next(10, 50)
An array of all the results from performing 10,000 checks should looks similar to:

Value Count
0 2500
1 2500
2 2500
3 2500

Copy link
Member

@reyang reyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mic-max mic-max changed the title Histogram bucket find to use binary search Histogram bucket find benchmark Jan 8, 2022
@mic-max mic-max requested a review from reyang January 10, 2022 16:47
@utpilla
Copy link
Contributor

utpilla commented Jan 27, 2022

We should add benchmarks for recording histogram values with different number of dimensions similar to how we have benchmarks for Counter: https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/test/Benchmarks/Metrics/MetricsBenchmarks.cs

If it becomes too complicated, we could either remove the double Histogram benchmarks or have them in a separate class.

mic-max and others added 3 commits January 27, 2022 13:59

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
@mic-max mic-max changed the title Histogram bucket find benchmark Histogram Record Method Benchmark Feb 3, 2022
@cijothomas cijothomas merged commit fe7b917 into open-telemetry:main Feb 4, 2022
@mic-max mic-max deleted the hist-binary-search branch February 10, 2022 18:26
@reyang
Copy link
Member

reyang commented May 3, 2022

@mic-max I learned from some users of OpenTelemetry .NET that in some extreme cases they might be using more than 1k buckets. I guess we might want to revisit this and see if certain #2754 (comment) (e.g. I guess 140 might be a good magical number to trigger binary search).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants