Skip to content

Commit

Permalink
Merge branch 'master' into di-probe-notification-builder
Browse files Browse the repository at this point in the history
  • Loading branch information
p-datadog authored Oct 24, 2024
2 parents 9637f2b + 91d883f commit 3cff768
Show file tree
Hide file tree
Showing 649 changed files with 3,730 additions and 1,826 deletions.
11 changes: 7 additions & 4 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,18 @@ for guidance on how to set up your development environment,
run the test suite, write new integrations, and more.
-->

**Change log entry**
<!-- If this is a customer-visible change, a brief summary to be placed into the change log. -->

**What does this PR do?**
<!-- A brief description of the change being made with this pull request. -->

**Motivation:**
<!-- What inspired you to submit this pull request? -->

**Change log entry**
<!-- If this is a customer-visible change, a brief summary to be placed
into the change log.
If you are not a Datadog employee, you can skip this section
and it will be filled or deleted during PR review. -->

**Additional Notes:**
<!-- Anything else we should know when reviewing? -->

Expand All @@ -25,4 +28,4 @@ If this change cannot be feasibly tested, please explain why,
unless the change does not modify code (e.g. only modifies docs, comments).
-->

Unsure? Have a question? Request a review!
<!-- Unsure? Have a question? Request a review! -->
55 changes: 55 additions & 0 deletions .github/workflows/nix.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
name: Test Nix

on:
push:
branches:
- "**"

jobs:
test:
strategy:
fail-fast: false
matrix:
platform:
- os: darwin
cpu: x86_64
base: macos-13 # always x86_64-darwin
- os: darwin
cpu: arm64
base: macos-14 # always arm64-darwin
- os: linux
cpu: x86_64
base: ubuntu-24.04 # always x86_64-linux-gnu
- os: linux
cpu: aarch64
base: arm-4core-linux-ubuntu24.04 # always aarch64-linux-gnu
nix:
- 24.05

name: Test Nix (${{ matrix.platform.cpu }}-${{ matrix.platform.os }}, ${{ matrix.nix }})
runs-on: ${{ matrix.platform.base }}

permissions:
contents: read
id-token: write

env:
SKIP_SIMPLECOV: 1
DD_INSTRUMENTATION_TELEMETRY_ENABLED: false
DD_REMOTE_CONFIGURATION_ENABLED: false

steps:
- name: Check CPU arch
run: |
test "$(uname -m)" = "${{ matrix.platform.cpu }}"
- uses: actions/checkout@v4
- uses: DeterminateSystems/nix-installer-action@main
- uses: DeterminateSystems/magic-nix-cache-action@main
- name: Print ruby version
run: |
nix develop --command which ruby
nix develop --command ruby --version
- name: Bundle install
run: nix develop --command bundle install
- name: Run spec:main
run: nix develop --command bundle exec rake spec:main
5 changes: 3 additions & 2 deletions .github/workflows/system-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ env:
REGISTRY: ghcr.io
REPO: ghcr.io/datadog/dd-trace-rb
ST_REF: main
FORCE_TESTS: -F tests/appsec/waf/test_addresses.py::Test_GraphQL -F tests/appsec/test_blocking_addresses.py::Test_BlockingGraphqlResolvers
FORCE_TESTS_SCENARIO: GRAPHQL_APPSEC
FORCE_TESTS: -F tests/appsec/test_asm_standalone.py
FORCE_TESTS_SCENARIO: APPSEC_STANDALONE

jobs:
build-harness:
Expand Down Expand Up @@ -199,6 +199,7 @@ jobs:
- APPSEC_DISABLED
- APPSEC_BLOCKING_FULL_DENYLIST
- APPSEC_REQUEST_BLOCKING
- APPSEC_STANDALONE
include:
- library: ruby
app: rack
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/test-yjit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ jobs:
- '--yjit'
- '--yjit --yjit-stats=quiet'
exclude:
# NOTE: `--yjit-stats=quiet` is only present in 3.3+.
# We've had `--yjit-stats` since 3.0, but
# 1) it dumps stat info to stdout, and
# 2) none of our <3.3 tests require the additional stats
- ruby: '3.2'
rubyopt: '--yjit --yjit-stats=quiet'
name: Test YJIT (${{ matrix.os }}, ${{ matrix.ruby }} ${{ matrix.rubyopt }})
Expand Down
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ ext/**/skipped_reason.txt
# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
.rvmrc
# Ignore local variables
.envrc
/.envrc
/.direnv

# lock files
Gemfile.lock
Expand Down
7 changes: 4 additions & 3 deletions .gitlab/benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,22 +95,23 @@ only-profiling-alloc:
DD_PROFILING_ALLOCATION_ENABLED: "true"
ADD_TO_GEMFILE: "gem 'datadog', github: 'datadog/dd-trace-rb', ref: '$CI_COMMIT_SHA'"

only-profiling-alloc-ddprof:
only-profiling-heap:
extends: .benchmarks
variables:
DD_BENCHMARKS_DDPROF: "true"
DD_BENCHMARKS_CONFIGURATION: only-profiling
DD_PROFILING_ENABLED: "true"
DD_PROFILING_ALLOCATION_ENABLED: "true"
DD_PROFILING_EXPERIMENTAL_HEAP_ENABLED: "true"
ADD_TO_GEMFILE: "gem 'datadog', github: 'datadog/dd-trace-rb', ref: '$CI_COMMIT_SHA'"

only-profiling-heap:
only-profiling-heap-clean-after-gc:
extends: .benchmarks
variables:
DD_BENCHMARKS_CONFIGURATION: only-profiling
DD_PROFILING_ENABLED: "true"
DD_PROFILING_ALLOCATION_ENABLED: "true"
DD_PROFILING_EXPERIMENTAL_HEAP_ENABLED: "true"
DD_PROFILING_HEAP_CLEAN_AFTER_GC_ENABLED: "true"
ADD_TO_GEMFILE: "gem 'datadog', github: 'datadog/dd-trace-rb', ref: '$CI_COMMIT_SHA'"

profiling-and-tracing:
Expand Down
2 changes: 1 addition & 1 deletion .gitlab/install_datadog_deps.rb
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@
}

[
'debase-ruby_core_source',
'datadog-ruby_core_source',
'ffi',
'libddwaf',
'msgpack',
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## [Unreleased]

### Added

* AppSec: Add Experimental Standalone AppSec Threats billing ([#3965][])

## [2.4.0] - 2024-10-11

### Added
Expand Down
9 changes: 1 addition & 8 deletions benchmarks/profiler_gc.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,7 @@

class ProfilerGcBenchmark
def create_profiler
@recorder = Datadog::Profiling::StackRecorder.new(
cpu_time_enabled: true,
alloc_samples_enabled: false,
heap_samples_enabled: false,
heap_size_enabled: false,
heap_sample_every: 1,
timeline_enabled: true,
)
@recorder = Datadog::Profiling::StackRecorder.for_testing(timeline_enabled: true)
@collector = Datadog::Profiling::Collectors::ThreadContext.for_testing(recorder: @recorder, timeline_enabled: true)

# We take a dummy sample so that the context for the main thread is created, as otherwise the GC profiling methods do
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_memory_sample_serialize.rb
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ def setup
@retain_every = (ENV['RETAIN_EVERY'] || '10').to_i
@skip_end_gc = ENV['SKIP_END_GC'] == 'true'
@recorder_factory = proc {
Datadog::Profiling::StackRecorder.new(
Datadog::Profiling::StackRecorder.for_testing(
cpu_time_enabled: false,
alloc_samples_enabled: true,
heap_samples_enabled: @heap_samples_enabled,
Expand Down
9 changes: 1 addition & 8 deletions benchmarks/profiler_sample_gvl.rb
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,7 @@ def initialize
end

def create_profiler
@recorder = Datadog::Profiling::StackRecorder.new(
cpu_time_enabled: true,
alloc_samples_enabled: false,
heap_samples_enabled: false,
heap_size_enabled: false,
heap_sample_every: 1,
timeline_enabled: true,
)
@recorder = Datadog::Profiling::StackRecorder.for_testing(timeline_enabled: true)
@collector = Datadog::Profiling::Collectors::ThreadContext.for_testing(
recorder: @recorder,
waiting_for_gvl_threshold_ns: 0,
Expand Down
9 changes: 1 addition & 8 deletions benchmarks/profiler_sample_loop_v2.rb
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,7 @@ class ProfilerSampleLoopBenchmark
PROFILER_OVERHEAD_STACK_THREAD = Thread.new { sleep }

def create_profiler
@recorder = Datadog::Profiling::StackRecorder.new(
cpu_time_enabled: true,
alloc_samples_enabled: false,
heap_samples_enabled: false,
heap_size_enabled: false,
heap_sample_every: 1,
timeline_enabled: false,
)
@recorder = Datadog::Profiling::StackRecorder.for_testing
@collector = Datadog::Profiling::Collectors::ThreadContext.for_testing(recorder: @recorder)
end

Expand Down
9 changes: 1 addition & 8 deletions benchmarks/profiler_sample_serialize.rb
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,7 @@ class ProfilerSampleSerializeBenchmark

def create_profiler
timeline_enabled = ENV['TIMELINE'] == 'true'
@recorder = Datadog::Profiling::StackRecorder.new(
cpu_time_enabled: true,
alloc_samples_enabled: false,
heap_samples_enabled: false,
heap_size_enabled: false,
heap_sample_every: 1,
timeline_enabled: timeline_enabled,
)
@recorder = Datadog::Profiling::StackRecorder.for_testing(timeline_enabled: timeline_enabled)
@collector = Datadog::Profiling::Collectors::ThreadContext.for_testing(recorder: @recorder, timeline_enabled: timeline_enabled)
end

Expand Down
2 changes: 1 addition & 1 deletion datadog.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Gem::Specification.new do |spec|
# Used by the profiler native extension to support Ruby < 2.6 and > 3.2
#
# We decided to pin it at the latest available version and will manually bump the dependency as needed.
spec.add_dependency 'debase-ruby_core_source', '= 3.3.1'
spec.add_dependency 'datadog-ruby_core_source', '= 3.3.6'

# Used by appsec
spec.add_dependency 'libddwaf', '~> 1.14.0.0.0'
Expand Down
11 changes: 11 additions & 0 deletions default.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# flake-compat shim for usage without flakes
(import
(
let lock = builtins.fromJSON (builtins.readFile ./flake.lock); in
fetchTarball {
url = lock.nodes.flake-compat.locked.url or "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz";
sha256 = lock.nodes.flake-compat.locked.narHash;
}
)
{ src = ./.; }
).defaultNix
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ internal types, structures and functions).
Because these private header files are not included in regular Ruby installations, we have two different workarounds:

1. for Ruby versions 2.6 to 3.2 we make use use the Ruby private MJIT header
2. for Ruby versions < 2.6 and > 3.2 we make use of the `debase-ruby_core_source` gem
2. for Ruby versions < 2.6 and > 3.2 we make use of the `datadog-ruby_core_source` gem

Functions which make use of these headers are defined in the <private_vm_api_acccess.c> file.

Expand All @@ -91,9 +91,9 @@ version. e.g. `rb_mjit_min_header-2.7.4.h`.

This header was removed in Ruby 3.3.

### Approach 2: Using the `debase-ruby_core_source` gem
### Approach 2: Using the `datadog-ruby_core_source` gem

The [`debase-ruby_core_source`](https://github.com/ruby-debug/debase-ruby_core_source) contains almost no code;
The [`datadog-ruby_core_source`](https://github.com/DataDog/datadog-ruby_core_source) contains almost no code;
instead, it just contains per-Ruby-version folders with the private VM headers (`.h`) files for that version.

Thus, even though a regular Ruby installation does not include these files, we can access the copy inside this gem.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -815,6 +815,9 @@ VALUE thread_context_collector_sample_after_gc(VALUE self_instance) {

state->stats.gc_samples++;

// Let recorder do any cleanup/updates it requires after a GC step.
recorder_after_gc_step(state->recorder_instance);

// Return a VALUE to make it easier to call this function from Ruby APIs that expect a return value (such as rb_rescue2)
return Qnil;
}
Expand Down Expand Up @@ -1441,7 +1444,8 @@ void thread_context_collector_sample_allocation(VALUE self_instance, unsigned in
class_name = ruby_value_type_to_class_name(type);
}
} else {
// Fallback for objects with no class
// Fallback for objects with no class. Objects with no class are a way for the Ruby VM to mark them
// as internal objects; see rb_objspace_internal_object_p for details.
class_name = ruby_value_type_to_class_name(type);
}
} else if (type == RUBY_T_IMEMO) {
Expand Down
16 changes: 8 additions & 8 deletions ext/datadog_profiling_native_extension/extconf.rb
Original file line number Diff line number Diff line change
Expand Up @@ -256,21 +256,21 @@ def skip_building_extension!(reason)
create_makefile EXTENSION_NAME
else
# The MJIT header was introduced on 2.6 and removed on 3.3; for other Rubies we rely on
# the debase-ruby_core_source gem to get access to private VM headers.
# the datadog-ruby_core_source gem to get access to private VM headers.
# This gem ships source code copies of these VM headers for the different Ruby VM versions;
# see https://github.com/ruby-debug/debase-ruby_core_source for details
# see https://github.com/DataDog/datadog-ruby_core_source for details

create_header

require "debase/ruby_core_source"
require "datadog/ruby_core_source"
dir_config("ruby") # allow user to pass in non-standard core include directory

# This is a workaround for a weird issue...
#
# The mkmf tool defines a `with_cppflags` helper that debase-ruby_core_source uses. This helper temporarily
# The mkmf tool defines a `with_cppflags` helper that datadog-ruby_core_source uses. This helper temporarily
# replaces `$CPPFLAGS` (aka the C pre-processor [not c++!] flags) with a different set when doing something.
#
# The debase-ruby_core_source gem uses `with_cppflags` during makefile generation to inject extra headers into the
# The datadog-ruby_core_source gem uses `with_cppflags` during makefile generation to inject extra headers into the
# path. But because `with_cppflags` replaces `$CPPFLAGS`, well, the default `$CPPFLAGS` are not included in the
# makefile.
#
Expand All @@ -281,12 +281,12 @@ def skip_building_extension!(reason)
# `VM_CHECK_MODE=1` when building Ruby will trigger this issue (because somethings in structures the profiler reads
# are ifdef'd out using this setting).
#
# To workaround this issue, we override `with_cppflags` for debase-ruby_core_source to still include `$CPPFLAGS`.
Debase::RubyCoreSource.define_singleton_method(:with_cppflags) do |newflags, &block|
# To workaround this issue, we override `with_cppflags` for datadog-ruby_core_source to still include `$CPPFLAGS`.
Datadog::RubyCoreSource.define_singleton_method(:with_cppflags) do |newflags, &block|
super("#{newflags} #{$CPPFLAGS}", &block)
end

Debase::RubyCoreSource
Datadog::RubyCoreSource
.create_makefile_with_core(
proc do
headers_available =
Expand Down
Loading

0 comments on commit 3cff768

Please sign in to comment.