Proof of concept: OpenTelemetry metrics

[edwardneal]

Contributes to #2211.

This adds an initial framework for updated telemetry using .NET's new metrics API. Leaving it in draft because I'm conscious that the issue hasn't yet had design approval from the SqlClient team, but this should be a reasonably generic and performant base to feedback on build on for the metrics design. It also provides a spot to put the eventual tracing APIs (when those are eventually standardised) and allows us to remove the project-specific connection pool counters.

I've added instrumentation for the metrics laid out in the OTel specifications, and for the existing .NET Core event counters. In most cases, this isn't wired up to anything.

From some very basic benchmarking, this has no measurable GC or CPU overhead. I don't think it has any explicit lock contention either - just a bundle of Interlocked.Increment/.Decrement calls. The OpenTelemetry SDK is another matter!

Although there's going to be some memory overhead here, it should be minimal. Since most of the tags only vary at the connection pool group level, I'm planning to cache them there and allow SqlConnection and SqlCommand to draw on that reference (unless pooling is disabled, or a specific SqlConnection wants to specify a custom tag.)

There are two trivial points, one piece of additional work, and one specification design decision which I'm not sure I agree with.

• The documentation for Counter<T> states that it's only supposed to increase. I'm decrementing this type of counter because we currently have to target .NET Standard and .NET 6.0, and UpDownCounter<T> is only available in .NET 7+.
• I've left integration points for custom tags for a connection, and for the disablement of the legacy metrics. There's space for a feature request there, but I'm leaving it alone for now.
• Measuring the connection's usage time, open time, wait time might need some rework of SqlStatistics. Since the collection overhead of SqlStatistics is practically nil, I'd like to always enable such and push the data into the corresponding metrics. The precision of DateTime.UtcNow (used by SqlStatistics) is 0.5ms-15ms, so this might need to start using stopwatches instead, particularly for local or LAN SQL Server instances.

The specification design decision surrounds the use of the pool.name tag. In cases where the connection string uses Windows authentication (or, in future, perhaps some other kind of process-/thread-level identity impersonation mechanism) it's possible for two connections to have the same connection string, but to be in different connection pools because they've been opened under an impersonation context. The specification rolls both of these up by their connection string. I'm thinking about whether or not to add a db.user tag to handle this - it'd make it clearer that multiple pools are involved, but might break compatibility with downstream clients which expect one metric per pool.name.

Tagging @roji and @JRahnama for any comments on the topic, direction or methodology. There aren't that many ways to implement these metrics, but I'd prefer to have some feedback from the SqlClient team so that I'm not surprising anyone with an x000-line PR for merge.

[Wraith2]

since the options can come from user input is it worth guarding against overflow here and below?

[edwardneal]

I don't think one could be triggered. MaxPoolSize ultimately comes via SqlConnectionString.MaxPoolSize, which is defined as an int and has protection against negative values.

An overflow would be triggered if there were uint.MaxValue connection pools in the connection pool group with a MaxPoolSize of int.MaxValue, but the list of connection pools is stored as a ConcurrentDictionary in DbConnectionPoolGroup (which is limited to storing int.MaxValue entries.)

[NatMarchand]

The documentation for Counter states that it's only supposed to increase. I'm decrementing this type of counter because we currently have to target .NET Standard and .NET 6.0, and UpDownCounter is only available in .NET 7+.

Although it came with NET7, updating System.Diagnostics.DiagnosticSource to a version > 7 will do the trick and preserve your target to netstandard2.0. There's no preprocessor directives preventing UpDownCounter to be compiled for targets other than NET7+

[edwardneal]

Thanks - I've just updated this to allow the use of UpDownCounter.

I've added enough code to show how I'd expect the .NET Framework version of the library to work and to ensure it compiles, but I'm going to leave this in draft until the overall design can be reviewed.