2.18.0rc1

Upgrade Notes

• ASM

  • With this upgrade, you can now control how the stack trace report are cropped when reported for exploit prevention or IAST.

    • DD_APPSEC_MAX_STACK_TRACE_DEPTH allowed to control the maximum stack trace size reported (default 32)
    • DD_APPSEC_MAX_STACK_TRACE_DEPTH_TOP_PERCENT allows now to specify how the stack trace is cropped as a percentage.

    For example, a value of 100 will report the top DD_APPSEC_MAX_STACK_TRACE_DEPTH frames from the stack, while a value of 0 will report the bottom DD_APPSEC_MAX_STACK_TRACE_DEPTH frames of the trace. A value of 50 will report half of DD_APPSEC_MAX_STACK_TRACE_DEPTH (rounded down) frames from the top of the stack and the rest from bottom. Default value is 75.

  • Upgrades libddwaf to 1.22.0

  • Upgrades libddwaf to 1.21.0 and security rule file to 1.13.3

Deprecation Notes

• Python 3.7 support is deprecated and will be removed in 3.0

New Features

• CI Visibility

  • Beta release of the new version of the pytest plugin, introducing the following features:

    • Auto Test Retries
    • Early Flake Detection
    • Improved coverage collection for Test Impact Analysis (formerly Intelligent Test Runner) now uses an internal collection method instead of coverage.py, with improved dependency discovery.

    Set the DD_PYTEST_USE_NEW_PLUGIN_BETA environment variable to true to use this new version.

    NOTE: this new version of the plugin introduces breaking changes:

    • module, suite, and test names are now parsed from the item.nodeid attribute
    • test names now include the class for class-based tests
    • Test skipping by Test Impact Analysis (formerly Intelligent Test Runner) is now done at the suite level, instead of at the test level

• Adds support for Selenium and RUM integration

• Code Security
  -Introduces "Standalone Code Security", a feature that disables APM in the tracer but keeps Code Security (IAST) enabled. In order to enable it, set the environment variables DD_IAST_ENABLED=1 and DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED=1.

• LLM Observability

  • Adds support to automatically submit Vertex AI Python calls to LLM Observability.
  • vertexai: Introduces tracing support for Google's Vertex AI SDK for Python's generate_content and send_message calls. See the docs for more information.

• Profiling

  • Profiler uses agent url configured via tracer.configure()

Bug Fixes

• ASM

  • Rnsures that common patches for exploit prevention and sca are only loaded if required, and only loaded once.
  • Resolves an issue where AppSec was using a patched JSON loads, creating telemetry errors.
  • Resolves an issue where some root span where not appropriately tagged for ASM standalone.

• CI Visibility

  • Fixes an issue where the CIVisbility service would incorrectly default the tracer env to None in EVP proxy mode if DD_ENV was not specified but the agent had a default environment set to a value other than none (eg: using DD_APM_ENV in the agent's environment).
  • Updates the inferred base service name algorithm to ensure that arguments following --ddtrace are no longer skipped when executing tests with pytest. Previously, the algorithm misinterpreted these arguments as standard flags, overlooking possible test paths that may contribute to the inferred service name.

• Code Security

  • Patches the module dir function so original pre-patch results are not changed.
  • Resolves a patching issue with psycopg3.
  • This fix resolves an issue where the modulo (%) operator would not be replaced correctly for bytes and bytesarray if IAST is enabled.
  • Ensures IAST SSRF vulnerability redacts the url query parameters correctly.
  • Adds umap, numba and pynndescent to the Code Security denylist.

• Crashtracking

  • Resolves issue where the crashtracker receiver may leave a zombie process behind after a crash.

• Lib-Injection

  • Ensures any user defined sitecustomize.py are preserved when auto-injecting.
  • Supports Python 2.7+ for injection compatibility check.
  • Resolves an issue where the default versions of click and jinja2 installed on 3.8 were outside of the allowed minimum versions for autoinstrumentation.

• LLM Observability

  • Ensures bedrock spans are finished even when streamed responses are not fully consumed.
  • langchain: Resolves a JSON decoding issue resulting from tagging streamed outputs from chains ending with a PydanticOutputParser.
  • Fixes an issue where decorators were not tracing generator functions properly.

• Profiling

  • Updates setup.py to ignore int-ptr conversion warnings for the profiler stack.pyx file. This is important because gcc 14 makes these conversions an error, alpine 3.21.0 ships with gcc 14, and any patch version of a Python alpine image cut after December 5th, 2024, will have this issue.
  • Fixes unbounded memory usage growth caused by keeping arbitrary user-generated strings (e.g. asyncio Task names) in an internal table and never removing them.
  • Fixes an issue where asyncio task names are not properly propagated when using stack v2, i.e. when DD_PROFILING_STACK_V2_ENABLED is set. Fixes an issue where asyncio tasks are not associated with spans when using stack v2, i.e. when DD_PROFILING_STACK_V2_ENABLED is set.

• Telemetry

• Ensures that Telemetry heartbeats are not skipped for forked processes, as doing so could result in the dependency list being lost over time.

• Tracing

  • botocore: This fix resolves an issue in the Bedrock integration where not consuming the full response stream would prevent spans from finishing.
  • botocore: This fix resolves the issue where the span pointer for deserialized DynamoDB requests (through the resource-based API) were not being generated.
  • botocore: This fix resolves an issue where our span pointer calculation code added recently logged unactionable messages.
  • celery: This fix resolves two issues with context propagation in celery
    1. Invalid span parentage when task A calls task B async and task A errors out, causing A's queuing of B, and B itself to not be parented under A.
    2. Invalid context propagation from client to workers, and across retries, causing multiple traces instead of a single trace
  • celery: Changes celery out.host span tag to point towards broker host url instead of local celery process hostname. Fixes inferred service representation issues when using celery.
  • grpcaio: Resolves a concurrency bug where distributed tracing headers were overwritten resulting in spans being assigned to the wrong trace.
  • kafka: Fixes an issue with Kafka consumer spans not using the active trace context when distributed tracing was enabled and no valid distributed context found was found within a consumed message.

Other Changes

• Tracing
  • Removed x-forwarded from headers used for client IP resolution (but not from collected headers). We lack evidence of actual usage, and whether this should follow RFC 7239 or regular XFF list format.

2.16.6

Bug Fixes

• SCA:

  • Ensure that Telemetry heartbeats are not skipped for forked processes, as doing so could result in the dependency list being lost over time.

• Code Security:

  • Resolve a patching issue with psycopg3.
  • Resolve an issue where the modulo (%) operator would not be replaced correctly for bytes and bytesarray if IAST is enabled.
  • Ensure IAST SSRF vulnerability redacts the url query parameters correctly.

• Lib-Injection:

  • Fix injection guardrail check when sys.argv is not available.

• Profiling

  • Updates setup.py to ignore int-ptr conversion warnings for the profiler stack.pyx file. This is important because gcc 14 makes these conversions an error, alpine 3.21.0 ships with gcc 14, and any patch version of a Python alpine image cut after December 5th, 2024, will have this issue.

2.17.3

Bug Fixes

• SCA:

  • Ensure that Telemetry heartbeats are not skipped for forked processes, as doing so could result in the dependency list being lost over time.

• Celery:

  • This fix resolves two issues with context propagation in celery
    • 1. Invalid span parentage when task A calls task B async and task A errors out, causing A's queuing of B, and B itself to not be parented under A.
    • 2. Invalid context propagation from client to workers, and across retries, causing multiple traces instead of a single trace

• Code Security:

  • This fix resolves a patching issue with psycopg3.
  • This fix resolves an issue where the modulo (%) operator would not be replaced correctly for bytes and bytesarray if IAST is enabled.
  • Ensure IAST SSRF vulnerability redacts the url query parameters correctly.

• Profiling:

  • Updates setup.py to ignore int-ptr conversion warnings for the profiler stack.pyx file. This is important because gcc 14 makes these conversions an error, alpine 3.21.0 ships with gcc 14, and any patch version of a Python alpine image cut after December 5th, 2024, will have this issue.

2.17.2

Bug Fixes

• ASM

  • Ensures that common patches for exploit prevention and SCA are only loaded if required, and only loaded once.

• LLM Observability

  • Ensures bedrock spans are finished even when streamed responses are not fully consumed.
  • Fixes an issue where decorators were not tracing generator functions properly.

• Tracing

  • botocore: Resolves an issue in the Bedrock integration where not consuming the full response stream would prevent spans from finishing.
  • celery: Changes celery out.host span tag to point towards broker host url instead of local celery process hostname. Fixes inferred service representation issues when using celery.
  • grpcaio: Resolves a concurrency bug where distributed tracing headers were overwritten resulting in spans being assigned to the wrong trace.

2.16.5

Bug Fixes

• ASM

  • Ensures that common patches for exploit prevention and sca are only loaded if required, and only loaded once.
  • Resolves an issue where some root span where not appropriately tagged for ASM standalone.

• Auto-Instrumentation

  • Resolves an issue where the default versions of click and jinja2 installed on python3.8 were outside of the allowed minimum versions for auto-instrumentation.

• Code Security

  • Patches the module dir function so original pre-patch results are not changed.

• LLM Observability

  • Ensures bedrock spans are finished even when streamed responses are not fully consumed.

• Tracing

  • botocore: Resolves an issue in the Bedrock integration where not consuming the full response stream would prevent spans from finishing.

2.15.4

Bug Fixes

• ASM

  • Ensures that common patches for exploit prevention and sca are only loaded if required, and only loaded once.
  • Resolves an issue where some root span where not appropriately tagged for ASM standalone.

• Auto-Instrumentation

  • Resolves an issue where the default versions of click and jinja2 installed on python3.8 were outside of the allowed minimum versions for auto-instrumentation.

• Code Security

  • Patches the module dir function so original pre-patch results are not changed.

• LLM Observability

  • Ensures bedrock spans are finished even when streamed responses are not fully consumed.

• Tracing

  • botocore: Resolves an issue in the Bedrock integration where not consuming the full response stream would prevent spans from finishing.

2.17.1

Bug Fixes

• ASM
  • Resolves an issue where some root spans were not appropriately tagged for ASM standalone.
• Code Security
  • Patches the module dir function so original pre-patch results are not changed.
• Tracing
  • Resolves an issue where the default versions of click and jinja2 installed on 3.8 were outside of the allowed minimum versions for autoinstrumentation.

2.17.0

New Features

• ASM

  • Support added for session fingerprints.

• LLM Observability

  • When not using a provider integration (OpenAI, Anthropic, or Bedrock) with the LangChain integration, token metrics will be appended to the LLM Observability llm span.
  • LLM Observability: When langchain's chat_model.with_structured_output(..., method="json_mode") is used, or response_format={"type": "json_object"} is passed into a langchain chat model invocation, the LLM Observability span will be an llm span instead of a workflow span.

• SSI

  • Adds requirements.json to SSI artifact for bailing out on unsupported systems.

• Tracing

  • Adds support for expanding AWS request/response Payloads into flattened span tags.
  • Updates the service naming algorithm to infer the base service name when DD_SERVICE is not set, replacing instances of 'unnamed-python-service'. Ensures that a more meaningful service name is used whenever possible, enhancing clarity in service identification.

Bug Fixes

• ASM

  • The new user events policy is preventing users PII to be added by default as span tags. To allow customers using the Django auto instrumentation to still have those information, new environment variables have been added. In particular DD_DJANGO_INCLUDE_EMAIL (false by default), will tag user events with user email as before.

• Code Security/IAST

  • Adds umap, numba and pynndescent to the Code Security denylist.
  • Adds googlecloudsdk and google auth to the Code Security deny list.
  • Resolves an issue where importing the google.cloud.storage.batch module would fail raising an ImportError

• Crashtracking

  • Fixes an issue where the use of the Crashtracking component could result in zombie processes.

• Lib-Injection

  • Supports Python 2.7+ for injection compatibility check.
  • Adds more commands to the auto-injection denylist.
  • Ensures we do not import the user installed ddtrace if it is present.
  • Fixes injection guardrail check when sys.argv is not available.

• LLM Observability

  • Resolves an issue where annotating spans with non-ASCII language input/output values resulted in encoded unicode being submitted.

• Profiling

  • Fixes a data race where span information associated with a thread was read and updated concurrently, leading to segfaults
  • Fixes an issue where cpu-time was not profiled for services using gunicorn, when DD_PROFILING_STACK_V2_ENABLED was set.
  • Fixes an issue where enabling native exporter via DD_PROFILING_EXPORT_LIBDD_ENABLED, DD_PROFILING_TIMELINE_ENABLED or DD_PROFILING_STACK_V2_ENABLED turned off live heap profiling.
  • The lock profiler would log a warning if it couldn't determine a name for a lock, and it would try determining a name multiple times for the same lock. This lead to excessive log spam. Downgrade this to a debug log and only try to determine the name once.
  • Fixes an issue where the profiler was allocating too much memory from ensure_binary_or_empty() function, on Python versions before 3.12, with DD_PROFILING_EXPORT_LIBDD_ENABLED or DD_PROFILING_TIMELINE_ENABLED.
  • Fixes an issue where the sample pool could deadlock after fork() by clearing it in the child process.
  • When a Python thread finishes, this change frees memory used for mapping its thread id to Span. The mapping is populated and used when DD_PROFILING_ENDPOINT_COLLECTION_ENABLED and DD_PROFILING_STACK_V2_ENABLED were set to enable grouping of profiles for endpoints.

• Tracing

  • Updates the inferred base service name algorithm to ensure that arguments following --ddtrace are no longer skipped when executing tests with pytest. Previously, the algorithm misinterpreted these arguments as standard flags, overlooking possible test paths that may contribute to the inferred service name.
  • botocore: Resolves the issue where the span pointer for deserialized DynamoDB requests (through the resource-based API) were not being generated.
  • botocore: Resolves an issue where our span pointer calculation code added recently logged unactionable messages.
  • pymongo: add type checking to solve an issue where NoneType instead of expected Pin object would throw an error in TracedTopology method.

2.17.0rc2

Bug Fixes

• Code Security

  • Adds umap, numba and pynndescent to the Code Security denylist.

• Lib-Injection

  • Supports Python 2.7+ for injection compatibility check.

• Tracing

  • Updates the inferred base service name algorithm to ensure that arguments following --ddtrace are no longer skipped when executing tests with pytest. Previously, the algorithm misinterpreted these arguments as standard flags, overlooking possible test paths that may contribute to the inferred service name.
  • botocore: Resolves an issue where our span pointer calculation code added recently logged unactionable messages.

2.16.4

Bug Fixes

• Tracing
  • botocore: Resolves the issue where the span pointer for deserialized DynamoDB requests (through the resource-based API) were not being generated.
  • botocore: Resolves an issue where our span pointer calculation code added recently logged unactionable messages.