[CIAPP-2959] CI visibility protocol and agentless mode support

Gemfile

@@ -10,14 +10,16 @@ gem "ddtrace"
 
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 
 gem "climate_control"
 
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
+
 gem "standard", "~> 1.31.0"
 
 gem "yard"

LICENSE-3rdparty.csv

@@ -1,2 +1,3 @@
 Component,Origin,License,Copyright
 dd-trace-rb,https://github.com/DataDog/dd-trace-rb,Apache 2.0,"Copyright 2016-Present Datadog, Inc."
+msgpack,https://rubygems.org/gems/msgpack,Apache-2.0,"Copyright (c) 2008-2015 Sadayuki Furuhashi"

README.md

@@ -117,6 +117,72 @@ end
 | `service_name` | Service name used for `cucumber` instrumentation. | `'cucumber'` |
 | `operation_name` | Operation name used for `cucumber` instrumentation. Useful if you want rename automatic trace metrics e.g. `trace.#{operation_name}.errors`. | `'cucumber.test'` |
 
+## Agentless mode
+
+If you are using a cloud CI provider without access to the underlying worker nodes, such as GitHub Actions or CircleCI, configure the library to use the Agentless mode. For this, set the following environment variables:
+`DD_CIVISIBILITY_AGENTLESS_ENABLED=true (Required)` and `DD_API_KEY=your_secret_api_key (Required)`.
+
+Additionally, configure which [Datadog site](https://docs.datadoghq.com/getting_started/site/) you want to send data to:
+`DD_SITE=your.datadoghq.com` (datadoghq.com by default).
+
+Agentless mode can be enabled via `Datadog.configure` (but don't forget to set DD_API_KEY environment variable):
+
+```ruby
+Datadog.configure { |c| c.ci.agentless_mode_enabled = true }
+```
+
+## Additional configuration
+
+### Add tracing instrumentations
+
+It can be useful to have rich tracing information about your tests that includes time spent performing database operations
+or other external calls like here:
+
+![Test trace with redis instrumented](./docs/screenshots/test-trace-with-redis.png)
+
+In order to achieve this you can configure ddtrace instrumentations in your configure block:
+
+```ruby
+Datadog.configure do |c|
+  #  ... ci configs and instrumentation here ...
+  c.instrument :redis
+  c.instrument :pg
+end
+```
+
+...or enable auto instrumentation in your test_helper/spec_helper:
+
+```ruby
+require "ddtrace/auto_instrument"
+```
+
+Note: in CI mode these traces are going to be submitted to CI visibility backend,
+they will **not** be submitted to Datadog APM.
+
+For the full list of available instrumentations see [ddtrace documentation](https://github.com/DataDog/dd-trace-rb/blob/master/docs/GettingStarted.md)
+
+### Disabling startup logs
+
+Startup logs produce a report of tracing state when the application is initially configured.
+These logs are activated by default in test mode, if you don't want them you can disable this
+via `diagnostics.startup_logs.enabled = false` or `DD_TRACE_STARTUP_LOGS=0`.
+
+```ruby
+Datadog.configure { |c| c.diagnostics.startup_logs.enabled = false }
+```
+
+### Enabling debug mode
+
+Switching the library into debug mode will produce verbose, detailed logs about tracing activity, including any suppressed errors. This output can be helpful in identifying errors, confirming trace output, or catching HTTP transport issues.
+
+You can enable this via `diagnostics.debug = true` or `DD_TRACE_DEBUG`.
+
+```ruby
+Datadog.configure { |c| c.diagnostics.debug = true }
+```
+
+**We do NOT recommend use of this feature in production or other sensitive environments**, as it can be very verbose under load. It's best to use this in a controlled environment where you can control application load.
+
 ## Contributing
 
 See [development guide](/docs/DevelopmentGuide.md), [static typing guide](docs/StaticTypingGuide.md) and [contributing guidelines](/CONTRIBUTING.md).

Steepfile

@@ -4,6 +4,7 @@ target :lib do
   check "lib"
 
   ignore "lib/datadog/ci/configuration/settings.rb"
+  ignore "lib/datadog/ci/transport/gzip.rb"
 
   library "pathname"
   library "json"
@@ -18,4 +19,5 @@ target :lib do
   library "open3"
   library "rspec"
   library "cucumber"
+  library "msgpack"
 end

datadog-ci.gemspec

@@ -39,4 +39,6 @@ Gem::Specification.new do |spec|
     ]].select { |fn| File.file?(fn) } # We don't want directories, only files
 
   spec.require_paths = ["lib"]
+
+  spec.add_dependency "msgpack"
 end

gemfiles/jruby_9.4.0.0_cucumber_3.gemfile

@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

gemfiles/jruby_9.4.0.0_cucumber_3.gemfile.lock

@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -126,6 +127,7 @@ GEM
       lint_roller (~> 1.1)
       rubocop-performance (~> 1.19.0)
     thor (1.2.2)
+    timecop (0.9.8)
     unicode-display_width (2.4.2)
     webrick (1.8.1)
     yard (0.9.34)
@@ -149,6 +151,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

gemfiles/jruby_9.4.0.0_cucumber_4.gemfile

@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

gemfiles/jruby_9.4.0.0_cucumber_4.gemfile.lock

@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -156,6 +157,7 @@ GEM
       ffi (~> 1.1)
     thor (1.2.2)
     thread_safe (0.3.6-java)
+    timecop (0.9.8)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
     unicode-display_width (2.4.2)
@@ -181,6 +183,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

gemfiles/jruby_9.4.0.0_cucumber_5.gemfile

@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

gemfiles/jruby_9.4.0.0_cucumber_5.gemfile.lock

@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -156,6 +157,7 @@ GEM
       ffi (~> 1.1)
     thor (1.2.2)
     thread_safe (0.3.6-java)
+    timecop (0.9.8)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
     unicode-display_width (2.4.2)
@@ -181,6 +183,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

gemfiles/jruby_9.4.0.0_cucumber_6.gemfile

@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

gemfiles/jruby_9.4.0.0_cucumber_6.gemfile.lock

@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -160,6 +161,7 @@ GEM
       ffi (~> 1.1)
     thor (1.2.2)
     thread_safe (0.3.6-java)
+    timecop (0.9.8)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
     unicode-display_width (2.4.2)
@@ -185,6 +187,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

gemfiles/jruby_9.4.0.0_cucumber_7.gemfile

@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

gemfiles/jruby_9.4.0.0_cucumber_7.gemfile.lock

@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -142,6 +143,7 @@ GEM
     sys-uname (1.2.3)
       ffi (~> 1.1)
     thor (1.2.2)
+    timecop (0.9.8)
     unicode-display_width (2.4.2)
     webrick (1.8.1)
     yard (0.9.34)
@@ -165,6 +167,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

gemfiles/jruby_9.4.0.0_cucumber_8.gemfile

@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

gemfiles/jruby_9.4.0.0_cucumber_8.gemfile.lock

@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -136,6 +137,7 @@ GEM
     sys-uname (1.2.3)
       ffi (~> 1.1)
     thor (1.2.2)
+    timecop (0.9.8)
     unicode-display_width (2.4.2)
     webrick (1.8.1)
     yard (0.9.34)
@@ -159,6 +161,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

gemfiles/jruby_9.4.0.0_minitest_5.gemfile

@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

gemfiles/jruby_9.4.0.0_minitest_5.gemfile.lock

@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -106,6 +107,7 @@ GEM
       lint_roller (~> 1.1)
       rubocop-performance (~> 1.19.0)
     thor (1.2.2)
+    timecop (0.9.8)
     unicode-display_width (2.4.2)
     webrick (1.8.1)
     yard (0.9.34)
@@ -129,6 +131,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard