PrometheusExporter: New concurrency handling for scrape middleware + http server #2610

CodeBlanch · 2021-11-15T07:50:05Z

Changes

Removed the semaphore and added logic that returns the same response to concurrent scrape requests and then caches the response for a configurable period of time (10 seconds by default).

Public API Changes

public class PrometheusExporterOptions
{
   public int ScrapeResponseCacheDurationMilliseconds { get; set; }
}

TODOs

CHANGELOG.md updated for non-trivial changes
README.md update for new setting
Changes in public API reviewed

…exporter.

codecov · 2021-11-15T07:59:57Z

Codecov Report

Merging #2610 (33b341e) into main (770a367) will decrease coverage by 0.05%.
The diff coverage is 75.43%.

❗ Current head 33b341e differs from pull request most recent head f94bc5f. Consider uploading reports for the commit f94bc5f to get more accurate results

@@            Coverage Diff             @@
##             main    #2610      +/-   ##
==========================================
- Coverage   80.27%   80.22%   -0.06%     
==========================================
  Files         256      257       +1     
  Lines        8826     8844      +18     
==========================================
+ Hits         7085     7095      +10     
- Misses       1741     1749       +8

Impacted Files	Coverage Δ
...y.Exporter.Prometheus/PrometheusExporterOptions.cs	`60.00% <33.33%> (-40.00%)`	⬇️
...eus/Implementation/PrometheusExporterMiddleware.cs	`61.53% <60.00%> (-16.52%)`	⬇️
...eus/Implementation/PrometheusExporterHttpServer.cs	`54.05% <65.00%> (-17.00%)`	⬇️
...heus/Implementation/PrometheusCollectionManager.cs	`80.88% <80.88%> (ø)`
...ometheus/Implementation/PrometheusSerializerExt.cs	`33.33% <100.00%> (-3.38%)`	⬇️
...elemetry.Exporter.Prometheus/PrometheusExporter.cs	`90.90% <100.00%> (-1.10%)`	⬇️
...ter.ZPages/Implementation/ZPagesActivityTracker.cs	`100.00% <0.00%> (+2.85%)`	⬆️
....Prometheus/Implementation/PrometheusSerializer.cs	`62.88% <0.00%> (+14.43%)`	⬆️

src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusCollectionManager.cs

reyang · 2021-11-15T17:22:58Z

src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusExporterHttpServer.cs

@@ -218,5 +148,46 @@ private void WorkerProc()
                }
            }
        }
+
+        private async Task ProcessRequestAsync(HttpListenerContext context)


This looks much cleaner 👍.

src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusCollectionManager.cs

cijothomas · 2021-11-15T17:56:43Z

src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusCollectionManager.cs

+                Interlocked.Increment(ref this.readerCount);
+                this.ExitGlobalLock();
+#if NETCOREAPP3_1_OR_GREATER
+                return new ValueTask<ArraySegment<byte>>(this.previousDataView);


nit: for troubleshooting purposes, i think it'd be good to add some logging to indicate whether we re-used previous view, or triggered new collection.

Or maybe use the Last-Modified response header. I think we should exclude the change from this PR (to keep it small).

OK I was working on a log message but I'll hold off and do this as a follow-up. I like the idea of using the header.

…/opentelemetry-dotnet into prometheus-thread-poc

CodeBlanch · 2021-11-15T19:28:48Z

src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusSerializerExt.cs


-                    for (var i = 0; i < keys.Length; i++)
+                    int numberOfKeys = keys?.Length ?? 0;
+                    if (numberOfKeys > 0)


@reyang I think this was a bug. We were getting nullref exception when keys/labels were not specified.

Yeah, I noticed this.

Let me add a test case for it once this PR is merged.

mic-max · 2021-12-13T19:22:08Z

src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusSerializerExt.cs

+                            if (i > 0)
+                            {
+                                buffer[cursor++] = unchecked((byte)',');
+                            }


@CodeBlanch Would there be benefit to moving this to above the for loop, starting i = 1 and also adding cursor = WriteLabel(buffer, cursor, keys[i], values[i]); or would the compiler already do this loop unrolling?

Oops I misread the code. another idea would be to always add the comma character after the WriteLabel call and then decrement the cursor after the loop so the next statement that modifies the buffer will overwrite it.

Cool idea! Updated on #2749

mic-max · 2021-12-13T19:31:13Z

src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusCollectionManager.cs

+        [MethodImpl(MethodImplOptions.AggressiveInlining)]
+        private bool ExecuteCollect()
+        {
+            this.exporter.OnExport = this.onCollectRef;


this.exporter.OnExport = this.OnCollect;
Would this not have worked?

So what this is doing is preventing the allocation of a delegate. The only sure-fire way I know how to do that is using a field. If you do it locally, sometimes the compiler will generate a field and initialize it on the first invocation, other times it will just allocate a delegate for each call. Not totally sure how it decides. I can't remember if this particular case fell into the delegate path or if I was just being cautious. I did do some benchmarks over this when I was working on it so I probably saw allocations and used the field, but I can't remember exactly.

Thanks for the explanation, I had a hunch it was for perf reasons

mic-max · 2021-12-13T19:51:37Z

src/OpenTelemetry.Exporter.Prometheus/PrometheusExporterOptions.cs

+                if (value < 0)
+                {
+                    throw new ArgumentOutOfRangeException(nameof(value), "Value should be greater than or equal to zero.");
+                }


use Guard.Range(value, nameof(value), min: 0)

CodeBlanch added 10 commits November 12, 2021 11:07

PoC for coordinating prometheus pull threads.

ffedbb2

Allocation free on .NET Core 3.1+ & locks are now tied to a specific …

482e390

…exporter.

Merge remote-tracking branch 'upstream/main' into prometheus-thread-poc

7cbbd96

Use CollectionManager in HttpServer. Remove delegate allocation.

017e481

Bug fixes and benchmark updates.

207453e

Merge from main.

229bc50

Configurable cache duration & bug fixes.

62bf08b

Unit tests and bug fixes.

7c17519

More interesting unit test.

c23fda4

Sealed PrometheusCollectionManager.

52591eb

CodeBlanch requested a review from a team November 15, 2021 07:50

reyang reviewed Nov 15, 2021

View reviewed changes

src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusCollectionManager.cs Show resolved Hide resolved

reyang reviewed Nov 15, 2021

View reviewed changes

reyang approved these changes Nov 15, 2021

View reviewed changes

Merge branch 'main' into prometheus-thread-poc

7dc4719

cijothomas reviewed Nov 15, 2021

View reviewed changes

src/OpenTelemetry.Exporter.Prometheus/Implementation/PrometheusCollectionManager.cs Outdated Show resolved Hide resolved

cijothomas reviewed Nov 15, 2021

View reviewed changes

cijothomas approved these changes Nov 15, 2021

View reviewed changes

reyang mentioned this pull request Nov 15, 2021

Prototype how to solve PrometheusExporter race condition #2528

Closed

CodeBlanch added 5 commits November 15, 2021 11:15

Nits.

4362400

CHANGELOG update.

9715a7f

Doc tweaks.

b75e848

README update.

fb7e2de

Merge branch 'prometheus-thread-poc' of https://github.com/CodeBlanch…

33b341e

…/opentelemetry-dotnet into prometheus-thread-poc

CodeBlanch commented Nov 15, 2021

View reviewed changes

Merge branch 'main' into prometheus-thread-poc

f94bc5f

CodeBlanch merged commit e871c27 into open-telemetry:main Nov 15, 2021

CodeBlanch deleted the prometheus-thread-poc branch November 15, 2021 20:02

CodeBlanch mentioned this pull request Nov 16, 2021

PrometheusExporter: Return last-modified header on scrape endpoint responses #2613

Merged

This was referenced Nov 17, 2021

[Do Not Merge] Metrics Async Pull API #2517

Closed

Fix Prometheus serialization bug #2636

Merged

mic-max reviewed Dec 13, 2021

View reviewed changes

CodeBlanch mentioned this pull request Dec 14, 2021

Prometheus Exporter: Misc code review feedback #2749

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PrometheusExporter: New concurrency handling for scrape middleware + http server #2610

PrometheusExporter: New concurrency handling for scrape middleware + http server #2610

CodeBlanch commented Nov 15, 2021 •

edited

Loading

codecov bot commented Nov 15, 2021 •

edited

Loading

reyang Nov 15, 2021

cijothomas Nov 15, 2021

reyang Nov 15, 2021

CodeBlanch Nov 15, 2021

CodeBlanch Nov 15, 2021

reyang Nov 15, 2021

reyang Nov 15, 2021

reyang Nov 16, 2021

mic-max Dec 13, 2021

mic-max Dec 13, 2021

CodeBlanch Dec 14, 2021

mic-max Dec 13, 2021

CodeBlanch Dec 14, 2021

mic-max Dec 14, 2021

mic-max Dec 13, 2021

PrometheusExporter: New concurrency handling for scrape middleware + http server #2610

PrometheusExporter: New concurrency handling for scrape middleware + http server #2610

Conversation

CodeBlanch commented Nov 15, 2021 • edited Loading

Changes

Public API Changes

TODOs

codecov bot commented Nov 15, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CodeBlanch commented Nov 15, 2021 •

edited

Loading

codecov bot commented Nov 15, 2021 •

edited

Loading