Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V5.0.0 proposal #3917

Closed
wants to merge 147 commits into from
Closed

V5.0.0 proposal #3917

wants to merge 147 commits into from

Conversation

khanayan123
Copy link
Collaborator

@khanayan123 khanayan123 commented Jan 2, 2024

Breaking Changes

More information about the breaking changes from this release can be found in
the migration guide.

Features

Improvements

Bug Fixes

Copy link

github-actions bot commented Jan 2, 2024

Overall package size

Self size: 5.82 MB
Deduped: 61.24 MB
No deduping: 61.99 MB

Dependency sizes

name version self size total size
@datadog/native-iast-taint-tracking 1.6.4 16.43 MB 16.44 MB
@datadog/native-appsec 6.0.0 14.48 MB 14.48 MB
@datadog/pprof 4.1.0 9.36 MB 10.21 MB
protobufjs 7.2.5 2.77 MB 6.56 MB
@datadog/native-iast-rewriter 2.2.2 2.29 MB 2.37 MB
@opentelemetry/core 1.14.0 872.87 kB 1.47 MB
@datadog/native-metrics 2.0.0 898.77 kB 1.3 MB
@opentelemetry/api 1.4.1 780.32 kB 780.32 kB
import-in-the-middle 1.4.2 41.4 kB 704.79 kB
pprof-format 2.0.7 588.12 kB 588.12 kB
msgpack-lite 0.1.26 201.16 kB 281.59 kB
opentracing 0.14.7 194.81 kB 194.81 kB
semver 7.5.4 93.4 kB 123.8 kB
@datadog/sketches-js 2.1.0 109.9 kB 109.9 kB
lodash.sortby 4.7.0 75.76 kB 75.76 kB
lru-cache 7.14.0 74.95 kB 74.95 kB
ipaddr.js 2.1.0 60.23 kB 60.23 kB
ignore 5.2.4 51.22 kB 51.22 kB
int64-buffer 0.1.10 49.18 kB 49.18 kB
istanbul-lib-coverage 3.2.0 29.34 kB 29.34 kB
lodash.uniq 4.5.0 25.01 kB 25.01 kB
tlhunter-sorted-set 0.1.0 24.94 kB 24.94 kB
limiter 1.1.5 23.17 kB 23.17 kB
dc-polyfill 0.1.2 22.77 kB 22.77 kB
retry 0.13.1 18.85 kB 18.85 kB
lodash.kebabcase 4.1.1 17.75 kB 17.75 kB
node-abort-controller 3.1.1 16.89 kB 16.89 kB
lodash.pick 4.4.0 16.33 kB 16.33 kB
jest-docblock 29.7.0 8.99 kB 12.76 kB
crypto-randomuuid 1.0.0 11.18 kB 11.18 kB
path-to-regexp 0.1.7 6.78 kB 6.78 kB
koalas 1.0.2 6.47 kB 6.47 kB
methods 1.1.2 5.29 kB 5.29 kB
module-details-from-path 1.0.3 4.47 kB 4.47 kB

🤖 This report was automatically generated by heaviest-objects-in-the-universe

Copy link

codecov bot commented Jan 2, 2024

Codecov Report

Attention: 107 lines in your changes are missing coverage. Please review.

Comparison is base (30e7a40) 84.82% compared to head (777fbea) 84.69%.
Report is 1 commits behind head on v5.x.

Files Patch % Lines
packages/datadog-plugin-mocha/src/index.js 0.00% 23 Missing ⚠️
packages/datadog-instrumentations/src/mocha.js 0.00% 18 Missing ⚠️
packages/dd-trace/src/datastreams/processor.js 79.03% 13 Missing ⚠️
packages/dd-trace/src/plugins/util/test.js 65.00% 7 Missing ⚠️
...src/appsec/iast/vulnerabilities-formatter/utils.js 93.33% 6 Missing ⚠️
...y/intelligent-test-runner/get-itr-configuration.js 66.66% 6 Missing ⚠️
packages/dd-trace/src/plugins/index.js 0.00% 6 Missing ⚠️
packages/dd-trace/src/appsec/index.js 87.50% 4 Missing ⚠️
packages/dd-trace/src/profiling/profiler.js 71.42% 4 Missing ⚠️
...ion/sensitive-analyzers/ldap-sensitive-analyzer.js 86.66% 2 Missing ⚠️
... and 13 more
Additional details and impacted files
@@            Coverage Diff             @@
##             v5.x    #3917      +/-   ##
==========================================
- Coverage   84.82%   84.69%   -0.13%     
==========================================
  Files         220      237      +17     
  Lines        8967    10147    +1180     
  Branches       33       33              
==========================================
+ Hits         7606     8594     +988     
- Misses       1361     1553     +192     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

szegedi and others added 27 commits January 2, 2024 13:56
…erimental ones (#3659)

* Deprecate EXPERIMENTAL_ env vars for code hostpots; introduce non-experimental variants
* make sure code hotspots being enabled is consistent with endpoint profiling being enabled
* Instrumentate nextjs and call to waf

* Feature implementation, pending to fix the tests

* Add appsec nextjs pipeline

* execute less tests

* Execute tests again

* try other approach to instrumentate next app request

* Fix lints

* Extract body and query in diagnostics channel publish not in subscribe

* Fix tests

* Fix query monitoring nextjs

* small fixes

* Remove unnecessary files

* Restore next test

* fixes

* Fix from PR comments

* Small changes from PR comments

* Replace some exec by fs functions
* Fix SQLi location when using knex

* Remove onProgress wrapping from knex then

* Rename asyncResource

* Add spacing in code to improve readability

* Avoid using shimmer to wrap then from knex raw result

* Detect dialect for SQLi with knex

* Instrument knex catch and asCallback
* Prevent sending query address twice to the waf

* Small changes + test

* Fix invalid query type test and behaviour

* Comment in the PR
* Enable appsec telemetry before waf init

* test reportWafInit is called when enabling appsec
* Fix supported MongoDB versions in NoSQL injection test

* Simplify version range for mongodb
Add tracking of supported integrations and supported integration versions and output data as artifacts during testing
---------

Co-authored-by: Stephen Belanger <[email protected]>
sabrenner and others added 25 commits January 2, 2024 13:56
* default error tagging for next and pageLoad safeguarding
#3889)

* Configure parametric test to get the nodejs tracer in same way as system-tests
* Upload module skeleton.

* Blocking in apollo, very very first version

* Move graphql implementation to another module.

* Blocking for apollo-server-core, ugly but it works, lets find a better way

* Use real blocking data

* Set blocking to true.

* Throw before resolver execution in order to stop the operation's execution flow.

* Use HttpQueryError in apollo-server-core

* Blocking test in apollo-server-fastify

* Refactor graphql blocking.

* Remove previous implementation which only supported monitoring.
* Add new waf address in order to check the payload of every resolver.
* Use apm start resolver address instead of a new one.
* Remove mock and perform an actual call to the waf.

* Add non blocking graphql test

* Move abortController constructor to context creation.

This reduces the performance overhead due to just one instance is shared across the whole query exectution.

* Add pollo-server-express block tests

* Add unit tests.

* Add @apollo/server tests

* Update test rules for blocking by `graphql.server.resolver`

* Block with graphql templates data

* Add tests.

* Block with graphql data in graphql endpoint

* Fix tests.

* Execute @apollo/server and apollo-server-express tests

* Unify code in @apollo/server and apollo-server-core

* update comments

* Add appsec.blocked tag in blocked requests

* Add test with non graphql block response

* Tests for block with redirect

* Prevent creation of resolve span when it is blocked before the execution of the resolve code

* Refactor addResolver in order to get directives information.

* Add tests to block on directives.

* Add test for directives.

* Undo prevent creating resolve span

* Configurable graphql blocking json

* Refactor graphql

* Using resolver instead of resolvers.
* Change graphql channel name to be consistent with the others.

* Small changes in blocking

* Move resover information resolution to plugin.

* Revert "Move resover information resolution to plugin."

This reverts commit 7cc8561.

* Remove resolver information from context, pass it in a different field instead.

* Throw custom exception rather than send an empty array.

* Update packages/datadog-instrumentations/src/graphql.js

Co-authored-by: Ugaitz Urien <[email protected]>

* Change a bit apollo-server-core instrumentation

* Protect Header map, if in future version it is moved/removed, prevent breaks

* Remove some duplicated code

* Update packages/datadog-instrumentations/src/apollo-server.js

Co-authored-by: Carles Capell <[email protected]>

* Fix comments in the PR

* Fix PR comments.

* Fix some comments in the PR

* Move resolver information formatting to the plugin.

* Fix PR comments.

* Fix proper use of Promise.race.

---------

Co-authored-by: Ugaitz Urien <[email protected]>
Co-authored-by: Carles Capell <[email protected]>
* Add thread id to heap and wall profiles

* Allow generateLabels to be called without arguments

In wall profile, if generateLabels is called without arguments, return
thread labels.
* New RC ASM_API_SECURITY_SAMPLE_RATE capability

* Do not report ASM_API_SECURITY_SAMPLE_RATE capability if apiSecurity is not enabled

* Change Activation constants case

* Update ASM_API_SECURITY_SAMPLE_RATE capability when enabling RemoteConfig

* Apply same coertion logic as in config.js

* Clean up

* Do not turn off ASM_API_SECURITY_SAMPLE_RATE when disabling waf update

* Include api_security_sampler test case

* Fix test comment

* remove NOTE
* Ensure recording start time of next profile is the same as the recording end time of the current profile.
* Pass the start-end range to profilers so they can use it
* Use passed-in start-end instead of internal duration computations
* Use end date to cut off late events for the next profile
…t are Recorded as Errors When Using Node 20 (#3853)" (#3896)

This reverts commit 59c8ea4.
* github: suggest new issues use helpdesk instead

* remove deprecated bug report / feature request options
* Update waf bindings to 6.0.0.

* Pass graphql.server.resolver as ephemeral address type.

* Add test.
* Update test agent configuration

* Update test sgent config

* Update test agent configuration

---------

Co-authored-by: William Conti <[email protected]>
* add support for node specifiers
* simplify ci build for aerospike
Node 14 versions prior to 14.18 do not support require statements with
`node:` prefix.
* Bump github actions versions

* Stop using deprecated set-output command

* Fix incompatible GLIBC version in GHA
Add a `process_id` that contains process pid to profiles.
@pr-commenter
Copy link

pr-commenter bot commented Jan 2, 2024

Benchmarks

Benchmark execution time: 2024-01-02 19:08:29

Comparing candidate commit 777fbea in PR branch v5.0.0-proposal with baseline commit 30e7a40 in branch v5.x.

Found 2 performance improvements and 67 performance regressions! Performance is the same for 313 metrics, 10 unstable metrics.

scenario:appsec-appsec-enabled-16

  • 🟥 cpu_user_time [+31.729ms; +39.177ms] or [+7.919%; +9.778%]
  • 🟥 execution_time [+53.968ms; +70.685ms] or [+11.933%; +15.629%]
  • 🟥 instructions [+103.4M instructions; +104.5M instructions] or [+10.022%; +10.136%]

scenario:appsec-appsec-enabled-18

  • 🟥 execution_time [+34.228ms; +51.590ms] or [+7.500%; +11.305%]
  • 🟥 instructions [+96.9M instructions; +97.5M instructions] or [+9.405%; +9.469%]

scenario:appsec-appsec-enabled-with-attacks-16

  • 🟥 cpu_user_time [+40.361ms; +48.461ms] or [+9.501%; +11.408%]
  • 🟥 execution_time [+56.231ms; +72.557ms] or [+11.720%; +15.123%]
  • 🟥 instructions [+103.0M instructions; +104.3M instructions] or [+9.153%; +9.266%]

scenario:appsec-appsec-enabled-with-attacks-18

  • 🟥 execution_time [+34.517ms; +54.904ms] or [+7.037%; +11.193%]
  • 🟥 instructions [+94.1M instructions; +94.9M instructions] or [+8.381%; +8.451%]

scenario:appsec-control-16

  • 🟥 cpu_user_time [+20.212ms; +27.217ms] or [+5.847%; +7.874%]
  • 🟥 execution_time [+40.617ms; +56.761ms] or [+10.209%; +14.267%]

scenario:appsec-control-18

  • 🟥 execution_time [+29.178ms; +47.998ms] or [+7.365%; +12.116%]

scenario:appsec-control-with-attacks-16

  • 🟥 execution_time [+37.545ms; +53.247ms] or [+9.815%; +13.919%]

scenario:appsec-control-with-attacks-18

  • 🟥 execution_time [+31.605ms; +48.948ms] or [+8.361%; +12.949%]

scenario:appsec-iast-no-vulnerability-control-16

  • 🟥 execution_time [+94.357ms; +119.541ms] or [+15.375%; +19.479%]
  • 🟩 cpu_usage_percentage [-12.044%; -9.402%]

scenario:appsec-iast-no-vulnerability-control-18

  • 🟥 execution_time [+102.311ms; +124.369ms] or [+18.103%; +22.006%]
  • 🟩 cpu_usage_percentage [-13.533%; -10.767%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-18

  • 🟥 cpu_user_time [+107.912ms; +136.448ms] or [+11.006%; +13.916%]
  • 🟥 execution_time [+166.498ms; +212.121ms] or [+15.223%; +19.395%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-16

  • 🟥 max_rss_usage [+6.100MB; +9.463MB] or [+5.638%; +8.745%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-18

  • 🟥 cpu_user_time [+95.520ms; +120.800ms] or [+9.926%; +12.553%]
  • 🟥 execution_time [+152.860ms; +189.886ms] or [+14.360%; +17.838%]
  • 🟥 max_rss_usage [+5.728MB; +9.048MB] or [+5.236%; +8.270%]

scenario:appsec-iast-with-vulnerability-control-16

  • 🟥 execution_time [+76.832ms; +125.039ms] or [+6.839%; +11.130%]

scenario:appsec-iast-with-vulnerability-control-18

  • 🟥 execution_time [+108.769ms; +153.356ms] or [+9.869%; +13.914%]

scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-18

  • 🟥 cpu_user_time [+108.944ms; +158.097ms] or [+7.470%; +10.841%]
  • 🟥 execution_time [+259.319ms; +360.042ms] or [+13.177%; +18.295%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-18

  • 🟥 execution_time [+175.838ms; +293.257ms] or [+9.874%; +16.468%]

scenario:log-skip-log-16

  • 🟥 cpu_user_time [+28.689ms; +36.036ms] or [+10.138%; +12.734%]
  • 🟥 execution_time [+31.401ms; +34.522ms] or [+10.069%; +11.070%]
  • 🟥 instructions [+254.8M instructions; +255.2M instructions] or [+24.805%; +24.842%]

scenario:log-skip-log-18

  • 🟥 cpu_user_time [+30.585ms; +39.218ms] or [+10.170%; +13.041%]
  • 🟥 execution_time [+35.188ms; +35.992ms] or [+10.495%; +10.734%]
  • 🟥 instructions [+252.1M instructions; +252.6M instructions] or [+23.998%; +24.044%]

scenario:log-with-debug-16

  • 🟥 cpu_user_time [+31.876ms; +39.844ms] or [+11.278%; +14.097%]
  • 🟥 execution_time [+33.719ms; +34.557ms] or [+10.851%; +11.120%]
  • 🟥 instructions [+254.7M instructions; +255.4M instructions] or [+24.796%; +24.871%]

scenario:log-with-debug-18

  • 🟥 cpu_user_time [+35.783ms; +44.985ms] or [+11.932%; +15.001%]
  • 🟥 execution_time [+35.094ms; +35.840ms] or [+10.463%; +10.686%]
  • 🟥 instructions [+251.8M instructions; +252.7M instructions] or [+23.973%; +24.054%]

scenario:log-with-error-16

  • 🟥 cpu_user_time [+32.772ms; +42.124ms] or [+11.668%; +14.998%]
  • 🟥 execution_time [+33.653ms; +34.302ms] or [+10.827%; +11.036%]
  • 🟥 instructions [+254.6M instructions; +255.5M instructions] or [+24.791%; +24.876%]

scenario:log-with-error-18

  • 🟥 cpu_user_time [+35.280ms; +43.864ms] or [+11.744%; +14.602%]
  • 🟥 execution_time [+35.166ms; +35.946ms] or [+10.491%; +10.723%]
  • 🟥 instructions [+252.1M instructions; +252.6M instructions] or [+24.001%; +24.050%]

scenario:log-without-log-16

  • 🟥 cpu_user_time [+30.317ms; +39.173ms] or [+11.220%; +14.497%]
  • 🟥 execution_time [+36.224ms; +37.253ms] or [+12.159%; +12.504%]
  • 🟥 instructions [+234.6M instructions; +234.9M instructions] or [+26.443%; +26.486%]

scenario:log-without-log-18

  • 🟥 cpu_user_time [+38.312ms; +46.506ms] or [+13.688%; +16.616%]
  • 🟥 execution_time [+40.081ms; +41.022ms] or [+12.785%; +13.085%]
  • 🟥 instructions [+232.0M instructions; +232.3M instructions] or [+25.485%; +25.527%]

scenario:net-with-tracer-16

  • 🟥 cpu_user_time [+17.375ms; +24.610ms] or [+5.370%; +7.606%]

scenario:plugin-bluebird-with-tracer-16

  • 🟥 execution_time [+16.993ms; +19.130ms] or [+5.252%; +5.913%]

scenario:spans-finish-immediately-16

  • 🟥 cpu_user_time [+34.241ms; +41.093ms] or [+7.345%; +8.815%]
  • 🟥 execution_time [+36.730ms; +39.161ms] or [+7.367%; +7.855%]
  • 🟥 instructions [+133.6M instructions; +137.4M instructions] or [+7.500%; +7.709%]

scenario:spans-finish-immediately-18

  • 🟥 cpu_user_time [+32.268ms; +39.475ms] or [+6.884%; +8.422%]
  • 🟥 execution_time [+37.268ms; +39.856ms] or [+7.387%; +7.900%]
  • 🟥 instructions [+129.6M instructions; +133.1M instructions] or [+7.491%; +7.695%]

scenario:spans-finish-later-16

  • 🟥 cpu_user_time [+76.251ms; +91.394ms] or [+9.211%; +11.040%]
  • 🟥 execution_time [+87.715ms; +96.257ms] or [+9.335%; +10.244%]
  • 🟥 max_rss_usage [+17.392MB; +17.634MB] or [+8.970%; +9.095%]

scenario:spans-finish-later-18

  • 🟥 cpu_user_time [+74.828ms; +87.840ms] or [+9.070%; +10.647%]
  • 🟥 execution_time [+84.288ms; +91.622ms] or [+8.958%; +9.737%]
  • 🟥 instructions [+123.5M instructions; +144.1M instructions] or [+6.002%; +7.002%]
  • 🟥 max_rss_usage [+14.896MB; +15.137MB] or [+7.205%; +7.322%]

@tlhunter
Copy link
Member

tlhunter commented Jan 2, 2024

This looks huge! Are these commits not already in v4?

@khanayan123 khanayan123 deleted the branch v5.x January 2, 2024 21:59
@khanayan123 khanayan123 closed this Jan 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.