Make elastic-transport-net OpenTelemetry compatible

[gregkalapos]

This initiative is an outcome from our syncs across the client and the APM team.

Current status

• The Elastic APM .NET Agent is able to trace the "old" elasticsearch-net client. This relies on 2 things: 1) the library emits DiagnosticSource events, and 2) the agent has a specific library which subscribes to those events and based on the events it creates traces.
• That APM package does not work with the new V8 client. Reasons:
  1. events got renamed
  2. types got moved, so those now live in different assemblies and namespaces, so the library can't cast them. This is the main challenge.

Idea

• During discussions we came up with the idea that instead of relying on an extra package in the APM .NET Agent, we could just turn elastic-transport-net into an OpenTelemetry compatible API and use the OpenTelemetry Bridge in the .NET Agent to capture the elasticsearch spans.
• I made a POC and it seems doable. We agreed to proceed with this.

Goal

• The goal is to turn elastic-transport-net into an OpenTelemetry compatible library.
• This means it should emit Activity instances that follow the OpenTelemetry database spec.

Steps

• Fill attributes (Activity.Tags) instead of passing objects - Emit OpenTelemetry compatible Activity on CallProductEndpoint #55
• Decide what to do with the existing DiagnosticSource and Activity code - currently the library emits more events than needed by OpenTelemetry. Currently my understanding is that we (the APM Agent, and OTel agents in general) are interested in the CallProductEndpoint calls, other things (like ping, sniff, serialization, etc.) aren't relevant and typically should not create spans.
• Activity names should have proper cardinality. Currently our best idea is to use the API endpoints. The challenge here is that, the API endpoint isn't available in elastic-transport-net, so the higher layer should pass it somehow. @stevejgordon
• Testing: we can write tests in this repository, but I feel we should also have tests with the proper client. Additionally we'll likely also have tests in the APM .NET Agent repo. - Emit OpenTelemetry compatible Activity on CallProductEndpoint #55 adds a basic test that can run in this repo.

cc @z1c0

[gregkalapos]

Implemented in elastic/elasticsearch-net#6990 and elastic/apm-agent-dotnet#1935