1.19.0

Highlights

Alpha support for Memory Profiling (Allocations and Heap)

As of ddtrace 1.19.0, the Profiler supports three new profile types:

• Allocations
• Heap Live Objects
• Heap Live Size

You can use these to optimize your application to allocate and use less memory, as well as to track down memory leaks.

We're still working on tuning the overhead impact and sampling frequency for these new profile types, and that's why this feature is marked as experimental.

You can enable these features:

• Using the environment variables DD_PROFILING_EXPERIMENTAL_ALLOCATION_ENABLED=true,
  DD_PROFILING_EXPERIMENTAL_HEAP_ENABLED=true and DD_PROFILING_EXPERIMENTAL_HEAP_SIZE_ENABLED=true
• Or via code by adding to your Datadog.configure block:

Datadog.configure do |c|
  # ... existing configuration ...
  c.profiling.advanced.experimental_allocation_enabled = true
  c.profiling.advanced.experimental_heap_enabled = true
  c.profiling.advanced.experimental_heap_size_enabled = true
end

Known issues and limitations:

• This feature only works on Ruby 2.7 and above.
• Ruby versions 3.2.0, 3.2.1 and 3.2.2 are not supported due to a Ruby VM bug. The bug has been fixed upstream and the upcoming Ruby 3.2.3 will be supported.
• This feature is currently incompatible with applications using Ractors due to a Ruby VM bug.
• The three options above are not fully independent; experimental_heap_size_enabled requires experimental_heap_enabled,
  and in turn experimental_heap_enabled requires experimental_allocation_enabled.
• We have added experimental tunables for lowering the overhead of these features. For more details, check the documentation for the experimental_allocation_sample_rate and experimental_heap_sample_rate settings.
• The experimental_allocation_enabled feature replaces the allocation_counting_enabled setting, which is now deprecated.

We're very interested in feedback on how this feature works for you. Give it a try, let us know how it's working!

Performance improvements to Garbage Collection Profiling + Timeline view support

The off-by-default Garbage Collection profiling feature has been redesigned to lower its overhead, and to support showing time spent doing Garbage Collection in the Profiler Timeline view.

See #3313 for some nice screenshots of how it looks!

You can enable this feature using the DD_PROFILING_FORCE_ENABLE_GC=true environment variable, or using the
c.profiling.advanced.force_enable_gc_profiling = true setting via code.

Added

• Tracing: Add on_error settings for mysql2 (#3316)
• Core: Add install_signature to app-started telemetry event (#3349)
• Profiling: Heap Profiling (#3281) (#3287) (#3328) (#3329) (#3333) (#3360)
• Profiling: Redesign GC profiling to add timeline support and reduce overhead (#3313)
• Core: Use Ruby 3.3 stable for CI testing (#3354)

Changed

• Core: Bump datadog-ci dependency to 0.6.0 (#3361)
• Core: Bump debase-ruby_core_source dependency to 3.3.1 (#3373)
• Docs: Backport "List Ruby 3.3 as supported in the docs" to master branch (#3374)
• Profiling: Import upstream rb_profile_frames fix (#3352)
• Profiling: Allow the dynamic sampling rate overhead target to be set (#3310)
• Profiling: Split profiling tests into ractor and non-ractor suites. (#3320)

Fixed

• Docs: Fix pg doc markdown format (#3317)
• Tracing: Fix recursive require in Railtie (#3365)
• Profiling: Fix issues stemming from rb_gc_force_recycle (#3366)
• Profiling: Fix Ruby 3.3 CI being broken in master due to profiler (#3356)
• Profiling: Fix "no signals" workaround detection when mariadb is in use (#3362)

Read the full changeset and the release milestone.

1.18.0

Added

• Tracing: Support lib injection for ARM64 architecture (#3307)
• Tracing: Add error_handler for pg instrumentation (#3303)
• Appsec: Enable "Trusted IPs", a.k.a passlist with optional monitoring (#3229)

Changed

• Mark ddtrace threads as fork-safe (#3279)
• Bump datadog-ci dependency to 0.5.0 (#3308)
• Bump debase-ruby_core_source dependency to 3.2.3 (#3284)
• Profiling: Disable profiler on Ruby 3.3 when running with RUBY_MN_THREADS=1 (#3259)
• Profiling: Run without "no signals" workaround on passenger 6.0.19+ (#3280)

Fixed

• Tracing: Fix pg instrumentation enabled settings (#3271)
• Profiling: Fix potential crash by importing upstream rb_profile_frames fix (#3289)
• Appsec: Call devise RegistrationsController block (#3286)

Read the full changeset and the release milestone.

1.17.0

For W3C Trace Context, this release adds tracecontext to the default trace propagation extraction and injection styles. The new defaults are:

• Extraction: Datadog,b3multi,b3,tracecontext
• Injection: Datadog,tracecontext

And to increase interoperability with tracecontext, 128-bit Trace ID generation is now the default.

For OpenTelemetry, this release adds support for converting OpenTelemetry Trace Semantic Conventions into equivalent Datadog trace semantics. Also, it's now possible to configure top-level Datadog span fields using OpenTelemetry span attributes (#3262).

For CI Visibility, you can now manually create CI traces and spans with the newly released API.

Added

• OpenTelemetry: Parse OpenTelemetry semantic conventions to Datadog's (#3273)
• OpenTelemetry: Support span reserved attribute overrides (#3262)
• Tracing: Ensure W3C tracestate is always propagated (#3255)

Changed

• Tracing: Set 128-bit trace_id to true by default (#3266)
• Tracing: Default trace propagation styles to Datadog,b3multi,b3,tracecontext (#3248,#3267)
• Ci-App: Upgraded datadog-ci dependency to 0.4 (#3270)

Read the full changeset and the release milestone.

1.16.2

This release reverts a change to appsec response body parsing that was introduced in 1.16.0 that may cause memory leaks.

Fixed

• Appsec: Revert parse response body fix introduced in 1.16.0 (#3252)

Read the full changeset and the release milestone.

1.16.1

Fixed

• Tracing: Fix concurrent-ruby future propagation without active_trace (#3242)
• Tracing: Fix host injection error handling (#3240)

Read the full changeset and the release milestone.

1.16.0

This release includes a security change for the Tracing Redis integration:

Currently, the Datadog Agent removes command arguments from the resource name. However there are cases, like Redis compressed keys, where this obfuscation cannot correctly remove command arguments. To safeguard that situation, the resource name set by the tracer will only be the command (e.g. SET) with no arguments. To retain the previous behavior and keep arguments in the span resource, with the potential risk of some command arguments not being fully obfuscated, set DD_REDIS_COMMAND_ARGS=true or set the option c.instrument :redis, command_args: true.

Added

• Tracing: Propagate trace through Concurrent::Promises.future (#1522)
• Core: Name Datadog::Core::Remote::Worker thread (#3207)

Changed

• Tracing: Redis - Omit command arguments from span.resource by default (#3235)
• Ci-app: Bump datadog-ci dependency from 0.2.0 to 0.3.0 (#3223)

Fixed

• Appsec: ASM parse response body (#3153)
• Appsec: ASM make sure to append content type and length information (#3204)
• Appsec: Make sure function that checks content-type header value accepts nil content-type header value (#3234)
• Profiling: Shut down profiler if any components failed (#3197)
• Tracing: Fix ActiveSupport instrumentation of custom cache stores (#3206)

Read the full changeset and the release milestone.

1.15.0

Highlights

Timeline view for Profiler beta

As of ddtrace 1.15.0, the Profiler now supports gathering data for the new Timeline view.

The Timeline view allows you to look at time-based patterns and work distribution over the period of a single profile: you can look at what individual threads were doing, and when 🎉

You can use the timeline view both when looking at individual profiles, as well as when scoped to a given trace.

You can enable it:

• Using an environment variable by setting DD_PROFILING_EXPERIMENTAL_TIMELINE_ENABLED=true
• Or via code by adding to your Datadog.configure block:

Datadog.configure do |c|
  # … existing configuration …
  c.profiling.advanced.experimental_timeline_enabled = true
end

Give it a try, let us know what you think!

(Note: We do not recommend enabling this feature prior to 1.15.0!)

google-protobuf dependency is no longer needed by the Profiler

As of ddtrace version 1.15.0, the google-protobuf gem is no longer needed to enable the Profiler.

If you've added this gem to your Gemfile/gems.rb file as part of enabling the Profiler, you can
remove it now. (If you're curious, we've internally replaced this dependency with the libdatadog gem.)

Configure blocking responses for AppSec via configuration or Remote Configuration

As of dd-trace-rb 1.15.0, AppSec supports configuring the blocking response.

You can configure the blocking response via:

• Using the ENV variables: DD_APPSEC_HTTP_BLOCKED_TEMPLATE_HTML=#{file_name}, and DD_APPSEC_HTTP_BLOCKED_TEMPLATE_JSON=#{file_name}
• Via code by adding to your Datadog.configure block:

Datadog.configure do |c|
  # … existing configuration …
  c.appsec.block.templates.html = "#{file_name}"
  c.appsec.block.templates.json = "#{file_name}"
end

• Using the Remote configuration UI. This option allows you to configure the status code and the blocking behaviour. You can redirect malicious attackers to custom pages.
  You can find more information on the official documentation

Configure agentless mode for CI visibility

If you are using CI visibility with a cloud CI provider without access to the underlying worker nodes, such as GitHub Actions or CircleCI, configure the library to use the Agentless mode.

For this, set the following environment variables:

• DD_CIVISIBILITY_AGENTLESS_ENABLED=true
• DD_API_KEY=<your_api_key>

Additionally, configure which Datadog site you want to send your data to:

• DD_SITE (default: datadoghq.com)

You can also enable agentless mode with Datadog.configure block:

Datadog.configure do |c|
  # … existing configuration …
  c.ci.agentless_mode_enabled = true
  # don't forget to set DD_API_KEY env variable!
end

Added

• Enable allocation counting feature by default for some Ruby 3 versions (#3176)
• Detect WebMock Cucumber and Rails.env to disable telemetry and remote configuration for development environment (#3065 , #3062 , #3145)
• Profiling: Import java-profiler PID controller and port it to C (#3190)
• Profiling: Record allocation type when sampling objects (#3096)
• Profiling: Include ruby vm type in profiler allocation samples (#3074)
• Tracing: Support Rack 3 (#3132 )
• Tracing: Support Opensearch 3 (#3189)
• Tracing: grpc adds client_error_handler option (#3095)
• Tracing: Add async option for test_mode configuration (#3158)
• Tracing: Implements _dd.base_service tag (#3018)
• Appsec: Allow blocking response template configuration via ENV variables (#2975)
• Appsec: ASM API security. Schema extraction (#3131, #3166, #3177)
• Appsec: Enable configuring blocking response via Remote Configuration (#3099)
• Ci-app: Validate git tags (#3100)
• Ci-app: Add agentless mode (#3186 )

Changed

• Appsec: Skip passing waf addresses when the value is empty (#3188)
• Profiling: Restore support for Ruby 3.3 (#3167)
• Profiling: Add approximate thread state categorization for timeline (#3162)
• Profiling: Wire up allocation sampling into CpuAndWallTimeWorker (#3103)
• Tracing: dalli disable memcached command tag by default (#3171)
• Tracing: Use first valid extracted style for distributed tracing (#2879 )
• Tracing: Rename configuration option on_set to after_set (#3107)
• Tracing: Rename experimental_default_proc to default_proc (#3091)
• Tracing: Use peer.service for sql comment propagation (#3127)
• Ci-app: Fix Datadog::CI::Environment to support the new CI specs (#3080)
• Bump datadog-ci dependency to 0.2 (#3186)
• Bump debase-ruby_core_source dependency to 3.2.2 (#3163)
• Upgrade libdatadog 5 (#3169, #3104)
• Upgrade libddwaf-rb 1.11.0 (#3087 )
• Update AppSec rules to 1.8.0 (#3140, #3139)

Fixed

• Profiling: Add workaround for incorrect invoke location when logging gem is in use (#3183)
• Profiling: Fix missing endpoint profiling when request_queuing is enabled in rack instrumentation (#3109)
• Appsec: Span tags reporting the number of WAF failed loaded rules (#3106)
• Tracing: Fix tagging with empty data (#3102)
• Tracing: Fix rails.cache.backend span tag with multiple stores (#3060)

Removed

• Profiling: Remove legacy profiler codepath (#3172)
• Ci-app: Remove CI module and add a dependency on datadog-ci gem (#3128)
• Tracing: Remove depends_on option from configuration DSL (#3085)
• Tracing: Remove delegate_to option from configuration DSL (#3086)

Read the full changeset and the release milestone.

1.14.0

Added

• Cucumber 8.0.0 support, test CI visibility with cucumber versions 6-8 (#3061)
• Tracing: Add ddsource to #to_log_format (#3025)
• Core: include peer service configurations in telemetry payload (#3056)
• Tracing: Improve quantization (#3041)

Changed

• Profiling: Disable profiler on Ruby 3.3 due to incompatibility (#3054)
• Core: EnvironmentLogger adjustments (#3020, #3057)

Fixed

• Appsec: Fix ASM setting for automated user events. (#3070)
• Tracing: Fix ActiveRecord adapter name for Rails 7 (#3051)

Read the full changeset and the release milestone.

1.13.1

Fixed

• Tracing: net/http instrumentation excludes query string for http.url tag (#3045)
• Tracing: Remove log_tags warning when given hash for log injection (#3022)
• Tracing: Fix OpenSearch integration loading (#3019)
• Core: Fix default hostname/port when mixing http and uds configuration (#3037)
• Core: Disable Telemetry and Remote Configuration in development environments (#3039)
• Profiling: Improve Datadog::Profiling::HttpTransport error logging (#3038)
• Docs: Document known issues with hanging Resque workers (#3033)

Read the full changeset and the release milestone.

1.13.0

Automated User Login Events for AppSec (Click to expand for more information)

Starting from ddtrace version 1.13.0, Automated User Login Events are enabled by default and currently only available for Devise.

What to expect from Automated User Login Events?

When enabled, any login or signup events will be automatically monitored. We will fill in the span with information extracted from the event.

Modes

The Automated User Login Events are configured to run in "safe" mode by default.

Safe mode focuses on preventing personally identifiable information (PII) from being reported. During safe mode, we only extract the ID from the user object. We report only IDs in the GUID format to avoid disclosing private information. In cases where IDs are not GUID, events will be reported without metadata.

Here is an example of a login event in safe mode:

{
  "appsec.events.users.login.success.track" => true,
  "usr.id" => "5ff35d04-445a-4d00-b6f1-31960153eaf6"
}

Automated User Login Events also provide an "extended" mode.

The extended mode aims to extract the maximum amount of information from the user. In extended mode, we report the ID regardless of its format and try to extract the user's email and username. Depending on the user application, we may be able to extract the user's email and username.

Here is an example of a login event in extended mode:

{
  "appsec.events.users.login.success.track" => true,
  "appsec.events.users.login.success.email" => "[email protected]",
  "appsec.events.users.login.success.username" => "John Doe",
  "usr.id" => "5ff35d04-445a-4d00-b6f1-31960153eaf6"
}

In cases where safe or extended mode is unable to extract information, we encourage users to manually monitor user login events using our public API.

How to configure Automated User Login Events:

• Use DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING to disable or change the mode.
  • To disable, use DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING=disable.
  • To change the mode, use DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING=safe|extended.
• Add to your Datadog.configure block to configure via code:

Datadog.configure do |c|
  c.appsec.track_user_events.enabled = true | false
  c.appsec.track_user_events.mode = "safe" | "extended"
end

Configuration options are now type checked (Click to expand for details)

Starting with ddtrace 1.13, configuration values are type checked when ddtrace is starting up.

In older versions, when setting configuration via Datadog.configure, no validation was performed on values. We've found this could lead to misleading or incorrect configurations, where something seems to be accepted but then did not work as expected. To address this, type checking is now applied to make sure settings are correct.

We've tested this extensively and don't expect this to impact correct configuration setups.

But, if you do have an incorrect configuration, this will now be flagged using an ArgumentError that includes a descriptive message for the setting name and the expected type.

Alternatively, we've provided a way to skip the type checking, by setting the Environment variable DD_EXPERIMENTAL_SKIP_CONFIGURATION_VALIDATION=true. We don't recommend using this option, since it hides configuration problems and thus may lead to ddtrace misbehaving.

As usual, your feedback is welcome -- let us know if we missed anything.

(Feature added in #2983, #2987, #2998 and #2999)

Added

• Core: Add support for Option precedence (#2915)
• Core: Add support for unsetting options (#2972)
• Core: Gauge primitive RubyVM::YJIT.runtime_stats, if YJIT is enabled (#2711, #2959)
• Core: Add Telemetry app-client-configuration-change event (#2977)
• Core: Improve our SafeDup module (#2960)
• Tracing: Add OpenSearch Integration (#2940)
• Tracing: Implement peer.service tag to integrations (#2982)
• Tracing: Add mandatory rpc and grpc tags for grpc integration (#2620, #2962)
• Tracing: Include _dd.profiling.enabled tag (#2913)
• Tracing: Support host injection (#2941, #3007)
• Tracing: Implement Dynamic Configuration for tracing (#2848, #2973)
• Tracing: Add for dynamic log injection configuration (#2992)
• Tracing: Add sampling configuration with DD_TRACE_SAMPLING_RULES (#2968)
• Tracing: Add HTTP header tagging with DD_TRACE_HEADER_TAGS for clients and servers (#2946, #2935)
• Profiling: Add fallback name/invoke location for unnamed threads started in native code (#2993)
• Profiling: Use invoke location as a fallback for nameless threads in the profiler (#2950)
• Profiling: Add fallback name for main thread in profiling data (#2939)
• Ci-app: Add Minitest CI integration (#2932)
• Appsec: Devise integration and automatic user events (#2877)
• Appsec: Handle disabled tracing and appsec events (#2572)
• Appsec: Automate user events check for UUID in safe mode (#2952)
• Docs: Add Ruby 3.2 support to compatibility matrix (#2971)

Changed

• Core: Set maximum supported Ruby version (#2497)
• Core: Prevent telemetry requests from being traced (#2961)
• Core: Add env and type to Core configuration option (#2983, #2988, #2994)
• Core: Remove lazy from Core configuration option (#2931, #2999)
• Profiling: Bump libdatadog dependency to version 3 (#2948)
• Profiling: Improve error message when ddtrace_profiling_loader fails to load (#2957)
• Tracing: Improve log injection runtime conditionals (#2926, #2882)

Fixed

• Core: Fix polynomial-time regular expressions (#2814)
• Core: Fix environment variable for dynamic configuration polling interval (#2967)
• Core: Reduce remote configuration error logging noise (#3011)
• Tracing: Fix manual log injection for 128 bit trace_id (#2974)
• Tracing: Ensure the GRPC client interceptor return the response (#2928)
• Tracing: Remove dynamic input used in regular expression (#2867)
• Tracing: Fix distributed tracing header formats (#3005 )
• Profiling: Fix profiler libmysqlclient version detection with mysql2-aurora gem (#2956)
• Profiling: Automatically enable profiler "no signals" workaround for passenger web server (#2978)

Read the full changeset and the release milestone.