Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add smoke test for Exception Replay #7504

Merged
merged 2 commits into from
Sep 5, 2024
Merged

Add smoke test for Exception Replay #7504

merged 2 commits into from
Sep 5, 2024

Conversation

jpbempel
Copy link
Member

@jpbempel jpbempel commented Aug 26, 2024 •
edited by jira bot
Loading

What Does This Do

Verifies that basic Exception Replay is working as expected using a
@trace annotation
No new capture after the first one for an hour and only 3 frames are
captured

Motivation

Additional Notes

Contributor Checklist

Jira ticket: DEBUG-2735

@pr-commenter
Copy link

pr-commenter bot commented Aug 26, 2024 •
edited
Loading

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1725548008 1725548388
end_time 2024-09-05T14:54:42 2024-09-05T15:01:02
git_branch master jpbempel/exception-debugger-smoke-tests
git_commit_sha 123a2c8 9d797a3
start_time 2024-09-05T14:53:29 2024-09-05T14:59:49
See matching parameters
Baseline Candidate
ci_job_id 628876773 628876773
ci_pipeline_id 43625036 43625036
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1725547412 1725547412

Summary

Found 0 performance improvements and 4 performance regressions! Performance is the same for 5 metrics, 6 unstable metrics.

scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:loop worse
[+318.626µs; +413.838µs] or [+3.074%; +3.992%]		 worse
[+316.874µs; +445.902µs] or [+3.003%; +4.226%]		 worse
[+298.106µs; +455.268µs] or [+2.803%; +4.281%]		 same worse
[-3.873op/s; -2.102op/s] or [-4.125%; -2.238%]
See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-35.069µs; +22.238µs] or [-13.272%; +8.416%]		 unstable
[-49.321µs; +35.153µs] or [-16.359%; +11.659%]		 unstable
[-64.280µs; +45.311µs] or [-20.318%; +14.322%]		 unstable
[-118.627µs; +89.876µs] or [-17.989%; +13.629%]		 same
scenario:basic same same same unstable
[-66.688µs; +25.717µs] or [-11.034%; +4.255%]		 unstable
[-180.517op/s; +180.517op/s] or [-7.040%; +7.040%]
Request duration reports for reports 
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (301.5 µs) : 252, 351
.   : milestone, 302,
basic (301.91 µs) : 293, 310
.   : milestone, 302,
loop (10.551 ms) : 10502, 10599
.   : milestone, 10551,
section candidate
noprobe (294.416 µs) : 267, 322
.   : milestone, 294,
basic (297.956 µs) : 289, 307
.   : milestone, 298,
loop (10.932 ms) : 10862, 11002
.   : milestone, 10932,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 301.5 µs [251.781 µs, 351.22 µs]
basic 301.91 µs [293.395 µs, 310.424 µs]
loop 10.551 ms [10.502 ms, 10.599 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 294.416 µs [266.549 µs, 322.284 µs]
basic 297.956 µs [289.132 µs, 306.78 µs]
loop 10.932 ms [10.862 ms, 11.002 ms]

@pr-commenter
Copy link

pr-commenter bot commented Aug 26, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/exception-debugger-smoke-tests
git_commit_date 1725545464 1725547412
git_commit_sha 123a2c8 9d797a3
release_version 1.40.0-SNAPSHOT~123a2c8614 1.39.0-SNAPSHOT~9d797a349c
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1725549837 1725549837
ci_job_id 628876759 628876759
ci_pipeline_id 43625036 43625036
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 16 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.39.0-SNAPSHOT~9d797a349c, baseline=1.40.0-SNAPSHOT~123a2c8614

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049317
Total [baseline] (10.379 s) : 0, 10379014
Agent [candidate] (1.051 s) : 0, 1051322
Total [candidate] (10.411 s) : 0, 10411160
section appsec
Agent [baseline] (1.181 s) : 0, 1180829
Total [baseline] (10.657 s) : 0, 10657094
Agent [candidate] (1.168 s) : 0, 1168333
Total [candidate] (10.48 s) : 0, 10480182
section iast
Agent [baseline] (1.203 s) : 0, 1202744
Total [baseline] (10.912 s) : 0, 10911588
Agent [candidate] (1.18 s) : 0, 1179900
Total [candidate] (10.84 s) : 0, 10840326
section profiling
Agent [baseline] (1.259 s) : 0, 1259442
Total [baseline] (10.652 s) : 0, 10651883
Agent [candidate] (1.243 s) : 0, 1242565
Total [candidate] (10.574 s) : 0, 10573959
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.049 s -
Agent appsec 1.181 s 131.512 ms (12.5%)
Agent iast 1.203 s 153.426 ms (14.6%)
Agent profiling 1.259 s 210.125 ms (20.0%)
Total tracing 10.379 s -
Total appsec 10.657 s 278.079 ms (2.7%)
Total iast 10.912 s 532.574 ms (5.1%)
Total profiling 10.652 s 272.869 ms (2.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.051 s -
Agent appsec 1.168 s 117.011 ms (11.1%)
Agent iast 1.18 s 128.578 ms (12.2%)
Agent profiling 1.243 s 191.243 ms (18.2%)
Total tracing 10.411 s -
Total appsec 10.48 s 69.022 ms (0.7%)
Total iast 10.84 s 429.166 ms (4.1%)
Total profiling 10.574 s 162.799 ms (1.6%) 
gantt
    title petclinic - break down per module: candidate=1.39.0-SNAPSHOT~9d797a349c, baseline=1.40.0-SNAPSHOT~123a2c8614

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (669.368 ms) : 0, 669368
BytebuddyAgent [candidate] (670.028 ms) : 0, 670028
GlobalTracer [baseline] (306.749 ms) : 0, 306749
GlobalTracer [candidate] (308.076 ms) : 0, 308076
AppSec [baseline] (51.473 ms) : 0, 51473
AppSec [candidate] (51.577 ms) : 0, 51577
Remote Config [baseline] (674.869 µs) : 0, 675
Remote Config [candidate] (677.135 µs) : 0, 677
Telemetry [baseline] (7.477 ms) : 0, 7477
Telemetry [candidate] (7.505 ms) : 0, 7505
section appsec
BytebuddyAgent [baseline] (689.082 ms) : 0, 689082
BytebuddyAgent [candidate] (678.346 ms) : 0, 678346
GlobalTracer [baseline] (300.128 ms) : 0, 300128
GlobalTracer [candidate] (299.558 ms) : 0, 299558
AppSec [baseline] (160.409 ms) : 0, 160409
AppSec [candidate] (157.28 ms) : 0, 157280
IAST [baseline] (18.334 ms) : 0, 18334
IAST [candidate] (20.722 ms) : 0, 20722
Remote Config [baseline] (619.413 µs) : 0, 619
Remote Config [candidate] (611.767 µs) : 0, 612
Telemetry [baseline] (7.878 ms) : 0, 7878
Telemetry [candidate] (8.7 ms) : 0, 8700
section iast
BytebuddyAgent [baseline] (800.943 ms) : 0, 800943
BytebuddyAgent [candidate] (784.503 ms) : 0, 784503
GlobalTracer [baseline] (302.601 ms) : 0, 302601
GlobalTracer [candidate] (297.802 ms) : 0, 297802
AppSec [baseline] (53.973 ms) : 0, 53973
AppSec [candidate] (53.843 ms) : 0, 53843
IAST [baseline] (22.518 ms) : 0, 22518
IAST [candidate] (21.685 ms) : 0, 21685
Remote Config [baseline] (605.533 µs) : 0, 606
Remote Config [candidate] (587.367 µs) : 0, 587
Telemetry [baseline] (8.228 ms) : 0, 8228
Telemetry [candidate] (7.919 ms) : 0, 7919
section profiling
BytebuddyAgent [baseline] (671.031 ms) : 0, 671031
BytebuddyAgent [candidate] (662.808 ms) : 0, 662808
GlobalTracer [baseline] (392.075 ms) : 0, 392075
GlobalTracer [candidate] (388.131 ms) : 0, 388131
AppSec [baseline] (52.366 ms) : 0, 52366
AppSec [candidate] (52.151 ms) : 0, 52151
Remote Config [baseline] (693.942 µs) : 0, 694
Remote Config [candidate] (683.51 µs) : 0, 684
Telemetry [baseline] (7.48 ms) : 0, 7480
Telemetry [candidate] (7.348 ms) : 0, 7348
ProfilingAgent [baseline] (97.618 ms) : 0, 97618
ProfilingAgent [candidate] (94.365 ms) : 0, 94365
Profiling [baseline] (97.642 ms) : 0, 97642
Profiling [candidate] (94.39 ms) : 0, 94390
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.39.0-SNAPSHOT~9d797a349c, baseline=1.40.0-SNAPSHOT~123a2c8614

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1056997
Total [baseline] (8.487 s) : 0, 8487033
Agent [candidate] (1.046 s) : 0, 1046425
Total [candidate] (8.497 s) : 0, 8497315
section iast
Agent [baseline] (1.171 s) : 0, 1170833
Total [baseline] (9.009 s) : 0, 9008956
Agent [candidate] (1.173 s) : 0, 1173482
Total [candidate] (8.953 s) : 0, 8952982
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.173 s) : 0, 1172597
Total [baseline] (8.932 s) : 0, 8932100
Agent [candidate] (1.172 s) : 0, 1171838
Total [candidate] (8.934 s) : 0, 8933562
section iast_TELEMETRY_OFF
Agent [baseline] (1.17 s) : 0, 1170364
Total [baseline] (8.995 s) : 0, 8994918
Agent [candidate] (1.172 s) : 0, 1171523
Total [candidate] (8.984 s) : 0, 8984326
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.057 s -
Agent iast 1.171 s 113.836 ms (10.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.173 s 115.6 ms (10.9%)
Agent iast_TELEMETRY_OFF 1.17 s 113.367 ms (10.7%)
Total tracing 8.487 s -
Total iast 9.009 s 521.923 ms (6.1%)
Total iast_HARDCODED_SECRET_DISABLED 8.932 s 445.067 ms (5.2%)
Total iast_TELEMETRY_OFF 8.995 s 507.885 ms (6.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.046 s -
Agent iast 1.173 s 127.058 ms (12.1%)
Agent iast_HARDCODED_SECRET_DISABLED 1.172 s 125.414 ms (12.0%)
Agent iast_TELEMETRY_OFF 1.172 s 125.099 ms (12.0%)
Total tracing 8.497 s -
Total iast 8.953 s 455.667 ms (5.4%)
Total iast_HARDCODED_SECRET_DISABLED 8.934 s 436.247 ms (5.1%)
Total iast_TELEMETRY_OFF 8.984 s 487.012 ms (5.7%) 
gantt
    title insecure-bank - break down per module: candidate=1.39.0-SNAPSHOT~9d797a349c, baseline=1.40.0-SNAPSHOT~123a2c8614

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (674.687 ms) : 0, 674687
BytebuddyAgent [candidate] (667.621 ms) : 0, 667621
GlobalTracer [baseline] (308.718 ms) : 0, 308718
GlobalTracer [candidate] (305.698 ms) : 0, 305698
AppSec [baseline] (51.673 ms) : 0, 51673
AppSec [candidate] (51.435 ms) : 0, 51435
Remote Config [baseline] (671.599 µs) : 0, 672
Remote Config [candidate] (674.497 µs) : 0, 674
Telemetry [baseline] (7.537 ms) : 0, 7537
Telemetry [candidate] (7.535 ms) : 0, 7535
section iast
BytebuddyAgent [baseline] (778.85 ms) : 0, 778850
BytebuddyAgent [candidate] (778.911 ms) : 0, 778911
GlobalTracer [baseline] (295.628 ms) : 0, 295628
GlobalTracer [candidate] (295.965 ms) : 0, 295965
AppSec [baseline] (52.069 ms) : 0, 52069
AppSec [candidate] (51.328 ms) : 0, 51328
IAST [baseline] (22.836 ms) : 0, 22836
IAST [candidate] (24.455 ms) : 0, 24455
Remote Config [baseline] (587.082 µs) : 0, 587
Remote Config [candidate] (593.997 µs) : 0, 594
Telemetry [baseline] (7.287 ms) : 0, 7287
Telemetry [candidate] (8.766 ms) : 0, 8766
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (779.649 ms) : 0, 779649
BytebuddyAgent [candidate] (777.513 ms) : 0, 777513
GlobalTracer [baseline] (295.704 ms) : 0, 295704
GlobalTracer [candidate] (295.98 ms) : 0, 295980
AppSec [baseline] (52.255 ms) : 0, 52255
AppSec [candidate] (50.545 ms) : 0, 50545
IAST [baseline] (22.691 ms) : 0, 22691
IAST [candidate] (23.473 ms) : 0, 23473
Remote Config [baseline] (588.76 µs) : 0, 589
Remote Config [candidate] (609.147 µs) : 0, 609
Telemetry [baseline] (8.116 ms) : 0, 8116
Telemetry [candidate] (10.238 ms) : 0, 10238
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (777.835 ms) : 0, 777835
BytebuddyAgent [candidate] (776.4 ms) : 0, 776400
GlobalTracer [baseline] (296.694 ms) : 0, 296694
GlobalTracer [candidate] (298.052 ms) : 0, 298052
AppSec [baseline] (52.081 ms) : 0, 52081
AppSec [candidate] (53.679 ms) : 0, 53679
IAST [baseline] (22.354 ms) : 0, 22354
IAST [candidate] (21.386 ms) : 0, 21386
Remote Config [baseline] (592.364 µs) : 0, 592
Remote Config [candidate] (584.994 µs) : 0, 585
Telemetry [baseline] (7.177 ms) : 0, 7177
Telemetry [candidate] (7.902 ms) : 0, 7902
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-09-05T14:54:39 2024-09-05T15:01:30
git_branch master jpbempel/exception-debugger-smoke-tests
git_commit_date 1725545464 1725547412
git_commit_sha 123a2c8 9d797a3
release_version 1.40.0-SNAPSHOT~123a2c8614 1.39.0-SNAPSHOT~9d797a349c
start_time 2024-09-05T14:54:25 2024-09-05T15:01:16
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1725548837 1725548837
ci_job_id 628876760 628876760
ci_pipeline_id 43625036 43625036
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 18 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~9d797a349c, baseline=1.40.0-SNAPSHOT~123a2c8614
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.362 ms) : 1343, 1381
.   : milestone, 1362,
appsec (1.722 ms) : 1697, 1746
.   : milestone, 1722,
appsec_no_iast (1.743 ms) : 1719, 1767
.   : milestone, 1743,
iast (1.496 ms) : 1474, 1519
.   : milestone, 1496,
profiling (1.485 ms) : 1462, 1508
.   : milestone, 1485,
tracing (1.473 ms) : 1448, 1498
.   : milestone, 1473,
section candidate
no_agent (1.363 ms) : 1343, 1383
.   : milestone, 1363,
appsec (1.738 ms) : 1714, 1761
.   : milestone, 1738,
appsec_no_iast (1.73 ms) : 1705, 1755
.   : milestone, 1730,
iast (1.489 ms) : 1466, 1512
.   : milestone, 1489,
profiling (1.499 ms) : 1473, 1524
.   : milestone, 1499,
tracing (1.466 ms) : 1441, 1490
.   : milestone, 1466,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.362 ms [1.343 ms, 1.381 ms] -
appsec 1.722 ms [1.697 ms, 1.746 ms] 359.659 µs (26.4%)
appsec_no_iast 1.743 ms [1.719 ms, 1.767 ms] 381.017 µs (28.0%)
iast 1.496 ms [1.474 ms, 1.519 ms] 134.044 µs (9.8%)
profiling 1.485 ms [1.462 ms, 1.508 ms] 122.925 µs (9.0%)
tracing 1.473 ms [1.448 ms, 1.498 ms] 111.253 µs (8.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.363 ms [1.343 ms, 1.383 ms] -
appsec 1.738 ms [1.714 ms, 1.761 ms] 374.757 µs (27.5%)
appsec_no_iast 1.73 ms [1.705 ms, 1.755 ms] 366.915 µs (26.9%)
iast 1.489 ms [1.466 ms, 1.512 ms] 126.375 µs (9.3%)
profiling 1.499 ms [1.473 ms, 1.524 ms] 135.594 µs (9.9%)
tracing 1.466 ms [1.441 ms, 1.49 ms] 102.823 µs (7.5%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~9d797a349c, baseline=1.40.0-SNAPSHOT~123a2c8614
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.894 µs) : 348, 388
.   : milestone, 368,
iast (494.237 µs) : 472, 517
.   : milestone, 494,
iast_FULL (562.045 µs) : 540, 584
.   : milestone, 562,
iast_GLOBAL (519.695 µs) : 497, 543
.   : milestone, 520,
iast_HARDCODED_SECRET_DISABLED (494.354 µs) : 472, 517
.   : milestone, 494,
iast_INACTIVE (451.651 µs) : 430, 473
.   : milestone, 452,
iast_TELEMETRY_OFF (482.742 µs) : 460, 506
.   : milestone, 483,
tracing (446.762 µs) : 426, 467
.   : milestone, 447,
section candidate
no_agent (367.312 µs) : 348, 387
.   : milestone, 367,
iast (487.485 µs) : 465, 510
.   : milestone, 487,
iast_FULL (561.453 µs) : 540, 583
.   : milestone, 561,
iast_GLOBAL (524.414 µs) : 500, 548
.   : milestone, 524,
iast_HARDCODED_SECRET_DISABLED (488.478 µs) : 466, 511
.   : milestone, 488,
iast_INACTIVE (452.525 µs) : 432, 473
.   : milestone, 453,
iast_TELEMETRY_OFF (478.43 µs) : 457, 500
.   : milestone, 478,
tracing (448.641 µs) : 427, 470
.   : milestone, 449,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.894 µs [348.263 µs, 387.524 µs] -
iast 494.237 µs [471.837 µs, 516.638 µs] 126.343 µs (34.3%)
iast_FULL 562.045 µs [540.13 µs, 583.96 µs] 194.152 µs (52.8%)
iast_GLOBAL 519.695 µs [496.524 µs, 542.865 µs] 151.801 µs (41.3%)
iast_HARDCODED_SECRET_DISABLED 494.354 µs [472.055 µs, 516.654 µs] 126.461 µs (34.4%)
iast_INACTIVE 451.651 µs [430.442 µs, 472.859 µs] 83.757 µs (22.8%)
iast_TELEMETRY_OFF 482.742 µs [459.71 µs, 505.774 µs] 114.849 µs (31.2%)
tracing 446.762 µs [426.225 µs, 467.3 µs] 78.869 µs (21.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.312 µs [347.671 µs, 386.952 µs] -
iast 487.485 µs [464.827 µs, 510.143 µs] 120.173 µs (32.7%)
iast_FULL 561.453 µs [539.617 µs, 583.288 µs] 194.141 µs (52.9%)
iast_GLOBAL 524.414 µs [500.426 µs, 548.401 µs] 157.102 µs (42.8%)
iast_HARDCODED_SECRET_DISABLED 488.478 µs [465.938 µs, 511.018 µs] 121.167 µs (33.0%)
iast_INACTIVE 452.525 µs [431.63 µs, 473.421 µs] 85.214 µs (23.2%)
iast_TELEMETRY_OFF 478.43 µs [456.581 µs, 500.28 µs] 111.119 µs (30.3%)
tracing 448.641 µs [427.141 µs, 470.142 µs] 81.329 µs (22.1%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/exception-debugger-smoke-tests
git_commit_date 1725545464 1725547412
git_commit_sha 123a2c8 9d797a3
release_version 1.40.0-SNAPSHOT~123a2c8614 1.39.0-SNAPSHOT~9d797a349c
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1725549346 1725549346
ci_job_id 628876763 628876763
ci_pipeline_id 43625036 43625036
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~9d797a349c, baseline=1.40.0-SNAPSHOT~123a2c8614
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.352 s) : 15352000, 15352000
.   : milestone, 15352000,
appsec (15.361 s) : 15361000, 15361000
.   : milestone, 15361000,
iast (19.015 s) : 19015000, 19015000
.   : milestone, 19015000,
iast_GLOBAL (18.166 s) : 18166000, 18166000
.   : milestone, 18166000,
profiling (15.303 s) : 15303000, 15303000
.   : milestone, 15303000,
tracing (14.953 s) : 14953000, 14953000
.   : milestone, 14953000,
section candidate
no_agent (15.189 s) : 15189000, 15189000
.   : milestone, 15189000,
appsec (15.041 s) : 15041000, 15041000
.   : milestone, 15041000,
iast (18.768 s) : 18768000, 18768000
.   : milestone, 18768000,
iast_GLOBAL (18.11 s) : 18110000, 18110000
.   : milestone, 18110000,
profiling (14.862 s) : 14862000, 14862000
.   : milestone, 14862000,
tracing (15.023 s) : 15023000, 15023000
.   : milestone, 15023000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.352 s [15.352 s, 15.352 s] -
appsec 15.361 s [15.361 s, 15.361 s] 9.0 ms (0.1%)
iast 19.015 s [19.015 s, 19.015 s] 3.663 s (23.9%)
iast_GLOBAL 18.166 s [18.166 s, 18.166 s] 2.814 s (18.3%)
profiling 15.303 s [15.303 s, 15.303 s] -49.0 ms (-0.3%)
tracing 14.953 s [14.953 s, 14.953 s] -399.0 ms (-2.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.189 s [15.189 s, 15.189 s] -
appsec 15.041 s [15.041 s, 15.041 s] -148.0 ms (-1.0%)
iast 18.768 s [18.768 s, 18.768 s] 3.579 s (23.6%)
iast_GLOBAL 18.11 s [18.11 s, 18.11 s] 2.921 s (19.2%)
profiling 14.862 s [14.862 s, 14.862 s] -327.0 ms (-2.2%)
tracing 15.023 s [15.023 s, 15.023 s] -166.0 ms (-1.1%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~9d797a349c, baseline=1.40.0-SNAPSHOT~123a2c8614
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.457 ms) : 1446, 1469
.   : milestone, 1457,
appsec (2.218 ms) : 2183, 2253
.   : milestone, 2218,
iast (1.968 ms) : 1925, 2010
.   : milestone, 1968,
iast_GLOBAL (2.0 ms) : 1957, 2044
.   : milestone, 2000,
profiling (1.853 ms) : 1819, 1887
.   : milestone, 1853,
tracing (1.825 ms) : 1793, 1858
.   : milestone, 1825,
section candidate
no_agent (1.455 ms) : 1444, 1466
.   : milestone, 1455,
appsec (2.213 ms) : 2178, 2248
.   : milestone, 2213,
iast (1.981 ms) : 1938, 2024
.   : milestone, 1981,
iast_GLOBAL (2.011 ms) : 1967, 2054
.   : milestone, 2011,
profiling (1.848 ms) : 1814, 1881
.   : milestone, 1848,
tracing (1.834 ms) : 1801, 1867
.   : milestone, 1834,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.457 ms [1.446 ms, 1.469 ms] -
appsec 2.218 ms [2.183 ms, 2.253 ms] 760.561 µs (52.2%)
iast 1.968 ms [1.925 ms, 2.01 ms] 510.463 µs (35.0%)
iast_GLOBAL 2.0 ms [1.957 ms, 2.044 ms] 543.005 µs (37.3%)
profiling 1.853 ms [1.819 ms, 1.887 ms] 395.975 µs (27.2%)
tracing 1.825 ms [1.793 ms, 1.858 ms] 368.048 µs (25.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.455 ms [1.444 ms, 1.466 ms] -
appsec 2.213 ms [2.178 ms, 2.248 ms] 757.621 µs (52.1%)
iast 1.981 ms [1.938 ms, 2.024 ms] 525.866 µs (36.1%)
iast_GLOBAL 2.011 ms [1.967 ms, 2.054 ms] 555.826 µs (38.2%)
profiling 1.848 ms [1.814 ms, 1.881 ms] 392.413 µs (27.0%)
tracing 1.834 ms [1.801 ms, 1.867 ms] 379.211 µs (26.1%)

@jpbempel jpbempel force-pushed the jpbempel/exception-debugger-smoke-tests branch 2 times, most recently from f1b03cb to 729e5d5 Compare August 27, 2024 16:04
@jpbempel jpbempel changed the title DRAFT Add smoke test for Exception Replay Aug 27, 2024
@jpbempel jpbempel force-pushed the jpbempel/exception-debugger-smoke-tests branch from 729e5d5 to c2329c2 Compare August 27, 2024 16:37
@jpbempel
Add smoke test for Exception Replay
d89cb71 
Verifies that basic Exception Replay is working as expected using a
@trace annotation
No new capture after the first one for an hour and only 3 frames are
captured
@jpbempel jpbempel force-pushed the jpbempel/exception-debugger-smoke-tests branch from c2329c2 to d89cb71 Compare August 28, 2024 06:12
@jpbempel jpbempel added comp: debugger Dynamic Instrumentation comp: testing Testing labels Aug 28, 2024
@jpbempel jpbempel marked this pull request as ready for review August 28, 2024 08:10
@jpbempel jpbempel requested a review from a team as a code owner August 28, 2024 08:10
@jpbempel jpbempel requested review from shatzi and removed request for a team August 28, 2024 08:10
shatzi
shatzi approved these changes Sep 5, 2024
View reviewed changes
...ntegration-tests/src/main/java/datadog/smoketest/debugger/ServerDebuggerTestApplication.java
tracedMethod(42, "foobar", 3.42, map, "var1", "var2", "var3");
if ("oops".equals(arg)) {
tracedMethodWithException(42, "foobar", 3.42, map, "var1", "var2", "var3");
} else if ("deepOops".equals(arg)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deepOops :D

...ke-tests/debugger-integration-tests/src/test/java/datadog/smoketest/BaseIntegrationTest.java Outdated
@@ -145,8 +145,8 @@ protected List<String> getDebuggerCommandParams() {
Arrays.asList(
"-Ddd.service.name=" + getAppId(),
"-Ddd.profiling.enabled=false",
"-Ddatadog.slf4j.simpleLogger.defaultLogLevel=debug",
"-Dorg.slf4j.simpleLogger.defaultLogLevel=info",
// "-Ddatadog.slf4j.simpleLogger.defaultLogLevel=debug",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove comment?

@jpbempel jpbempel merged commit a6d3fd2 into master Sep 5, 2024
99 checks passed
@jpbempel jpbempel deleted the jpbempel/exception-debugger-smoke-tests branch September 5, 2024 15:36
@github-actions github-actions bot added this to the 1.40.0 milestone Sep 5, 2024
@jpbempel jpbempel added the tag: no release notes Changes to exclude from release notes label Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shatzi shatzi shatzi approved these changes

Assignees
No one assigned
Labels
comp: debugger Dynamic Instrumentation comp: testing Testing tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
1.40.0
Development

Successfully merging this pull request may close these issues.
2 participants
@jpbempel @shatzi