[CIAPP-2959] CI visibility protocol and agentless mode support

[anmarchenko]

What does this PR do?

Decouples Contrib module and configuration from ddtrace

• datadog-ci now implements its own instrument method
• decouples Contrib from ddtrace: adding Integration and Contrib::Settings modules instead of using Tracing contribs
• renames CI::Test to CI::Recorder to better communicate its purpose

Adds agentless mode settings

• adds agentless_mode_enabled and agentless_url settings

Adds HTTP layer with gzip support

• Datadog::CI::Transport::HTTP is a simple HTTP transport that we use to submit test traces to citestcycle intake (TODO: when ddtrace 1.15.0 is released, migrate the Net::HTTP logic to use Datadog::Core::Transport::HTTP::Adapters::Net)
• Datadog::CI::Transport::Gzip uses standard Ruby zlib library to encode request payloads to gzip

Adds TestVisibility transport and configures ddtrace Tracer to use it

• Datadog::CI::TestVisibility::Transport is a drop-in replacement for tracer's transport that implements send_traces method that receives a list of traces, serializes them to msgpack, chunks by size (at most 4Mb per request) and sends to CI visibility intake
• Components#activate_ci! takes care of configuring agentless mode and tracer's writer options when application starts. NOTE: shutdown_timeout setting is needed to ensure that writer process is not going to be killed when test run ends and will have enough time (60 seconds) to send all leftover traces

Adds a layer of TestVisbility serializers and factories

• TestLevel constructs serializers depending on Span#type (another type of factory that we'll introduce in the future will be TestSuiteLevel)
• Most of the serialization and validation logic lives in Base serializer. TestV1 and Span serializers configure the fields needed for content tag, required fields and some field level logic

Motivation

Implementing long-awaited agentless mode for Ruby test visibility.

How to test the change?

Used https://github.com/anmarchenko/sidekiq for integration testing.
git clone this repo and then DD_API_KEY=<your key> bundle exec rake

[codecov-commenter]

Codecov Report

Merging #33 (98b0f06) into main (ef60285) will increase coverage by 0.23%.
The diff coverage is 98.09%.

@@            Coverage Diff             @@
##             main      #33      +/-   ##
==========================================
+ Coverage   98.94%   99.18%   +0.23%     
==========================================
  Files          82       95      +13     
  Lines        2563     3437     +874     
  Branches       98      125      +27     
==========================================
+ Hits         2536     3409     +873     
- Misses         27       28       +1     

Files	Coverage Δ
lib/datadog/ci/configuration/components.rb	100.00% <100.00%> (ø)
lib/datadog/ci/configuration/settings.rb	100.00% <100.00%> (ø)
...adog/ci/contrib/cucumber/configuration/settings.rb	100.00% <100.00%> (ø)
lib/datadog/ci/contrib/cucumber/integration.rb	100.00% <100.00%> (ø)
...adog/ci/contrib/minitest/configuration/settings.rb	100.00% <100.00%> (ø)
lib/datadog/ci/contrib/minitest/hooks.rb	96.96% <100.00%> (+0.30%)	⬆️
lib/datadog/ci/contrib/minitest/integration.rb	100.00% <100.00%> (ø)
...datadog/ci/contrib/rspec/configuration/settings.rb	100.00% <100.00%> (ø)
lib/datadog/ci/contrib/rspec/integration.rb	100.00% <100.00%> (ø)
lib/datadog/ci/contrib/settings.rb	100.00% <100.00%> (ø)
... and 33 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[delner]

There's a lot to digest here. Many of the changes you've been making are great! (e.g. I like driving the separation between CI and Tracing contrib).

I do have one area of concern though:

Datadog::CI::TestVisibility::Transport is a drop-in replacement for tracer's transport

This leverages the old transport. In the future, tracing will NOT support this as a drop-in replacement (likely in 2.0.) Is your plan to re-implement CI's transport on the new trace transport when that happens?

I think this speaks to my concern that CI should not be replacing Tracing internals: you might destabilize Tracing. We do not give any public API guarantees around the use of Transport code within the Tracing namespace (regardless of settings.tracing.test_mode.writer_options... this is a deprecated legacy API.)

[anmarchenko]

@delner thanks a lot for the additional context!
I was not aware that tracing is going to replace transport layer (currently it is not in ddtrace 2.0 scope document). I think it is still important to merge this PR because the scope of ddtrace 2.0 is still being defined but this agentless feature is very important for CI visibility customers right now.

I explained more of my motivation to go this way in my PR to ddtrace that was accepted last week: DataDog/dd-trace-rb#3158
In this PR I made additonal changes to the test mode configuration and added configurable shutdown timeout for both the old and new trace writers (new trace writer supports pluggable transport as of now).

re-implement CI's transport on the new trace transport when that happens

I think adapting CI transport would be still less work than copypasting and maintaining TraceWriter and TraceBuffer in CI gem, so this would be my plan, yes

tracing will NOT support this as a drop-in replacement

I would happily help us to define a new concept of tracing's extensibility then! I am very optimistic about our prospects to find a solution to make Tracer to submit traces to CI endpoint with CI serializers.

you might destabilize Tracing

could you explain more? I am clearly missing some context here: CI visibility will never run in production and will be included by CI customers as a separate library, so right now I don't see how could we destabilize tracing.

If the concern here is that providing configurable transport will open path for external developers to replace this part of tracing without understanding the underlying complexity then removing configuration will help not that much: it is still possible to replace tracer's logic with monkey patching.

regardless of settings.tracing.test_mode.writer_options... this is a deprecated legacy API

if writer options will be removed in the future and we won't create any replacement then I have a backup plan to implement TraceWriter and Tracing::Buffer in the CI gem. It will essentially duplicate the logic that already exists in Tracing module.

Then I will use CI::Flush to consume traces and send them to CI::TraceWriter that will store them in CI::TraceBuffer and send via CI::Transport.

I would not want to go this way now because it means to copypaste the code that already exists in Tracing to this library and maintaining it: currently it would not help us to achieve our goal of releasing this library to GA. This is still an option for the future though.

[juan-fernandez]

🎉 nice! Left some minor comments but LGTM

[delner]

@anmarchenko and I discussed the "Datadog::CI::TestVisibility::Transport is a drop-in replacement for tracer's transport" change.

• I shared why the current behavior is undesirable (as it touches a lot of internals of how Tracing works)
• I suggested an alternative strategy where CI can subscribe to trace_completed, then reuse the Core::Workers and Core::Buffer components to build its own worker to ingest traces, transform them to CI events, and submit them to the API. @anmarchenko and I agreed this would work for CI's needs.
• @anmarchenko and I agreed to implement the above strategy in 2.x: Tracing will provide an API so that CI can subscribe & receive completed traces, then Tracing will remove support for modifying its Writer. CI will re-implement this feature on this new public API.
• In the interim, in 1.x, Tracing will continue to support the existing transport_options and writer configurability on which this PR is built.

Therefore, there's no blocker for merging this today.