Error loading libddwaf when using generic ruby platform

[luc-financeit]

Current behaviour

After upgrading from 2.4.0 to 2.5.0, I'm seeing an error at boot time loading libddwaf:

Error & Backtrace

$ bundle exec rails c
/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/ffi-1.17.0-arm64-darwin/lib/ffi/dynamic_library.rb:94:in `load_library': Could not open library '/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/../../../vendor/libddwaf/libddwaf-1.15.0-darwin-arm64/lib/libddwaf.dylib': dlopen(/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/../../../vendor/libddwaf/libddwaf-1.15.0-darwin-arm64/lib/libddwaf.dylib, 0x0005): tried: '/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/../../../vendor/libddwaf/libddwaf-1.15.0-darwin-arm64/lib/libddwaf.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/../../../vendor/libddwaf/libddwaf-1.15.0-darwin-arm64/lib/libddwaf.dylib' (no such file), '/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/../../../vendor/libddwaf/libddwaf-1.15.0-darwin-arm64/lib/libddwaf.dylib' (no such file). (LoadError)
Searched in <system library path>, /opt/homebrew/lib, /opt/local/lib, /usr/local/lib, /usr/lib
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/ffi-1.17.0-arm64-darwin/lib/ffi/library.rb:95:in `block in ffi_lib'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/ffi-1.17.0-arm64-darwin/lib/ffi/library.rb:94:in `map'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/ffi-1.17.0-arm64-darwin/lib/ffi/library.rb:94:in `ffi_lib'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/waf.rb:100:in `<module:LibDDWAF>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/waf.rb:9:in `<module:WAF>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/waf.rb:8:in `<module:AppSec>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/waf.rb:6:in `<module:Datadog>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/datadog/appsec/waf.rb:5:in `<main>'
  from <internal:/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
  from <internal:/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/libddwaf-1.15.0.0.0/lib/libddwaf.rb:1:in `<main>'
  from <internal:/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
  from <internal:/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/appsec/processor/context.rb:3:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/appsec/processor.rb:3:in `require_relative'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/appsec/processor.rb:3:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/appsec/component.rb:3:in `require_relative'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/appsec/component.rb:3:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/core/configuration/components.rb:15:in `require_relative'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/core/configuration/components.rb:15:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/core/configuration.rb:3:in `require_relative'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/core/configuration.rb:3:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/core/extensions.rb:3:in `require_relative'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/core/extensions.rb:3:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/core.rb:4:in `require_relative'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/core.rb:4:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/tracing.rb:3:in `require_relative'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/tracing.rb:3:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog.rb:4:in `require_relative'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog.rb:4:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/auto_instrument.rb:6:in `require_relative'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/datadog-2.5.0/lib/datadog/auto_instrument.rb:6:in `<main>'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/bundler/runtime.rb:60:in `require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/bundler/runtime.rb:60:in `block (2 levels) in require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/bundler/runtime.rb:55:in `each'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/bundler/runtime.rb:55:in `block in require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/bundler/runtime.rb:44:in `each'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/bundler/runtime.rb:44:in `require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/bundler.rb:212:in `require'
  from /Users/lthevenard/code/libddwaf-load-error/config/application.rb:7:in `<main>'
  from <internal:/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
  from <internal:/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/railties-7.0.8.6/lib/rails/command/actions.rb:22:in `require_application!'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/railties-7.0.8.6/lib/rails/command/actions.rb:14:in `require_application_and_environment!'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/railties-7.0.8.6/lib/rails/commands/runner/runner_command.rb:33:in `perform'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/thor-1.3.2/lib/thor/command.rb:28:in `run'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/thor-1.3.2/lib/thor/invocation.rb:127:in `invoke_command'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/thor-1.3.2/lib/thor.rb:538:in `dispatch'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/railties-7.0.8.6/lib/rails/command/base.rb:87:in `perform'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/railties-7.0.8.6/lib/rails/command.rb:48:in `invoke'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/railties-7.0.8.6/lib/rails/commands.rb:18:in `<main>'
  from <internal:/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
  from <internal:/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
  from /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
  from bin/rails:4:in `<main>'

Expected behaviour

No crash.

Steps to reproduce

For reasons unknown, I had BUNDLE_FORCE_RUBY_PLATFORM=true in my shell profile, so I ended up installing a version of the libddwaf gem with no pre-built binaries. From a quick skim of the datadog source code, it looks like this might have worked in 2.4 because libddwaf was not loaded unless AppSec was enabled and/or load errors were rescued, whereas in 2.5 libddwaf is being loaded unconditionally? I've put together a minimal Rails app that reproduces the issue; see instructions in the README.

In all honesty, I don't know why I had BUNDLE_FORCE_RUBY_PLATFORM=true in my profile, and I've resolved the issue for myself by just removing that env var. Just thought it was maybe worth a mention anyways; if nothing else, it seems odd for AppSec::Processor to use such precaution loading libddwaf when appsec/processor/context.rb has already loaded it 🤷‍♂

Environment

• datadog version: 2.15.0
• Operating system: MacOS

[Strech]

Hey @luc-financeit 👋🏼

Thanks for reporting, this is indeed an unintentional mistake which will be fixed ASAP (probably as v2.5.1)

[Strech]

@luc-financeit If you can, I would like to ask you to share a single top block of your bundle env output. Basically the whole first chuck which looks like this:

Bundler       2.4.10
  Platforms   ruby, arm64-darwin-23
... remove for example  ...

And also output of the bundle config if it exists.

I think it might be another issue with installation. It would be good to be able to reproduce it.

[Strech]

@luc-financeit Good news, we have released v2.6.0 with a fix. But I would still appreciate some details about your bundler.

[luc-financeit]

Can confirm 2.6.0 fixes the issue, thanks! Here's the additional bundler info:

bundle env

Environment

Bundler       2.5.23
  Platforms   ruby, arm64-darwin-22
Ruby          3.1.6p260 (2024-05-29 revision a777087be612b7675fe012560bf0d8fddff20f55) [arm64-darwin-22]
  Full Path   /Users/lthevenard/.asdf/installs/ruby/3.1.6/bin/ruby
  Config Dir  /Users/lthevenard/.asdf/installs/ruby/3.1.6/etc
RubyGems      3.5.23
  Gem Home    /Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0
  Gem Path    /Users/lthevenard/.gem/ruby/3.1.0:/Users/lthevenard/.asdf/installs/ruby/3.1.6/lib/ruby/gems/3.1.0
  User Home   /Users/lthevenard
  User Path   /Users/lthevenard/.gem/ruby/3.1.0
  Bin Dir     /Users/lthevenard/.asdf/installs/ruby/3.1.6/bin
Tools         
  Git         2.46.0
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Built At          2024-11-05
Git SHA           35f4611265e
Released Version  true

Bundler settings

force_ruby_platform
  Set via BUNDLE_FORCE_RUBY_PLATFORM: true

bundle config

Settings are listed in order of priority. The top value will be used.
force_ruby_platform
Set via BUNDLE_FORCE_RUBY_PLATFORM: true

[Strech]

Happy to hear that! We will take a look into installation process to see possible issues there. I'm going to close this issue with a follow up issue in DataDog/libddwaf-rb#60