Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix evaluation errors sampling #7656

Merged
merged 1 commit into from
Sep 20, 2024
Merged

Fix evaluation errors sampling #7656

merged 1 commit into from
Sep 20, 2024

Conversation

jpbempel
Copy link
Member

@jpbempel jpbempel commented Sep 20, 2024 •
edited by jira bot
Loading

What Does This Do

evaluation error snapshots were sent even if rate limited now we are checking sampled state before sending

Motivation

Additional Notes

Contributor Checklist

Jira ticket: DEBUG-2892

@jpbempel
Fix evaluation errors sampling
ac44f21 
evaluation error snapshots were sent even if rate limited
now we are checking sampled state before sending
@jpbempel jpbempel added type: bug comp: debugger Dynamic Instrumentation labels Sep 20, 2024
@jpbempel jpbempel requested a review from a team as a code owner September 20, 2024 12:48
@jpbempel jpbempel requested review from evanchooly and removed request for a team September 20, 2024 12:48
@pr-commenter
Copy link

pr-commenter bot commented Sep 20, 2024

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1726837217 1726837593
end_time 2024-09-20T13:01:32 2024-09-20T13:07:48
git_branch master jpbempel/fix-eval-errors-sampling
git_commit_sha 5867971 ac44f21
start_time 2024-09-20T13:00:18 2024-09-20T13:06:33
See matching parameters
Baseline Candidate
ci_job_id 646380732 646380732
ci_pipeline_id 44786335 44786335
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1726836409 1726836409

Summary

Found 0 performance improvements and 5 performance regressions! Performance is the same for 2 metrics, 8 unstable metrics.

scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:loop worse
[+670.825µs; +700.305µs] or [+6.617%; +6.908%]		 worse
[+666.425µs; +722.055µs] or [+6.474%; +7.015%]		 worse
[+595.043µs; +706.833µs] or [+5.716%; +6.790%]		 worse
[+422.099µs; +855.145µs] or [+3.909%; +7.919%]		 worse
[-7.390op/s; -4.625op/s] or [-7.722%; -4.834%]
See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-51.766µs; +62.290µs] or [-18.881%; +22.719%]		 unstable
[-61.710µs; +78.989µs] or [-19.585%; +25.069%]		 unstable
[-70.852µs; +91.457µs] or [-21.464%; +27.706%]		 unstable
[-796.903µs; +883.967µs] or [-93.514%; +103.730%]		 same
scenario:basic same unstable
[-14.890µs; +19.922µs] or [-4.957%; +6.632%]		 unstable
[-19.023µs; +27.190µs] or [-6.124%; +8.753%]		 unstable
[-104.475µs; +152.020µs] or [-18.976%; +27.611%]		 unstable
[-236.857op/s; +236.857op/s] or [-9.237%; +9.237%]
Request duration reports for reports 
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (315.087 µs) : 252, 378
.   : milestone, 315,
basic (300.378 µs) : 283, 317
.   : milestone, 300,
loop (10.294 ms) : 10265, 10322
.   : milestone, 10294,
section candidate
noprobe (323.727 µs) : 253, 394
.   : milestone, 324,
basic (302.894 µs) : 286, 320
.   : milestone, 303,
loop (10.988 ms) : 10965, 11011
.   : milestone, 10988,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 315.087 µs [251.795 µs, 378.38 µs]
basic 300.378 µs [283.462 µs, 317.293 µs]
loop 10.294 ms [10.265 ms, 10.322 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 323.727 µs [253.042 µs, 394.412 µs]
basic 302.894 µs [285.755 µs, 320.033 µs]
loop 10.988 ms [10.965 ms, 11.011 ms]

@pr-commenter
Copy link

pr-commenter bot commented Sep 20, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/fix-eval-errors-sampling
git_commit_date 1726823241 1726836409
git_commit_sha 5867971 ac44f21
release_version 1.40.0-SNAPSHOT~58679718b3 1.40.0-SNAPSHOT~ac44f215cf
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1726838978 1726838978
ci_job_id 646380726 646380726
ci_pipeline_id 44786335 44786335
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 15 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.40.0-SNAPSHOT~ac44f215cf, baseline=1.40.0-SNAPSHOT~58679718b3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1065876
Total [baseline] (10.411 s) : 0, 10410874
Agent [candidate] (1.067 s) : 0, 1066929
Total [candidate] (10.422 s) : 0, 10421933
section appsec
Agent [baseline] (1.197 s) : 0, 1196586
Total [baseline] (10.653 s) : 0, 10652517
Agent [candidate] (1.199 s) : 0, 1198824
Total [candidate] (10.602 s) : 0, 10601576
section iast
Agent [baseline] (1.196 s) : 0, 1196131
Total [baseline] (10.812 s) : 0, 10811791
Agent [candidate] (1.19 s) : 0, 1189992
Total [candidate] (10.878 s) : 0, 10878424
section profiling
Agent [baseline] (1.269 s) : 0, 1269161
Total [baseline] (10.542 s) : 0, 10541858
Agent [candidate] (1.271 s) : 0, 1271297
Total [candidate] (10.566 s) : 0, 10566462
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.066 s -
Agent appsec 1.197 s 130.711 ms (12.3%)
Agent iast 1.196 s 130.255 ms (12.2%)
Agent profiling 1.269 s 203.286 ms (19.1%)
Total tracing 10.411 s -
Total appsec 10.653 s 241.643 ms (2.3%)
Total iast 10.812 s 400.917 ms (3.9%)
Total profiling 10.542 s 130.984 ms (1.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.067 s -
Agent appsec 1.199 s 131.895 ms (12.4%)
Agent iast 1.19 s 123.062 ms (11.5%)
Agent profiling 1.271 s 204.368 ms (19.2%)
Total tracing 10.422 s -
Total appsec 10.602 s 179.644 ms (1.7%)
Total iast 10.878 s 456.491 ms (4.4%)
Total profiling 10.566 s 144.529 ms (1.4%) 
gantt
    title petclinic - break down per module: candidate=1.40.0-SNAPSHOT~ac44f215cf, baseline=1.40.0-SNAPSHOT~58679718b3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (678.37 ms) : 0, 678370
BytebuddyAgent [candidate] (679.043 ms) : 0, 679043
GlobalTracer [baseline] (311.731 ms) : 0, 311731
GlobalTracer [candidate] (311.658 ms) : 0, 311658
AppSec [baseline] (53.881 ms) : 0, 53881
AppSec [candidate] (54.223 ms) : 0, 54223
Remote Config [baseline] (665.375 µs) : 0, 665
Remote Config [candidate] (682.183 µs) : 0, 682
Telemetry [baseline] (7.645 ms) : 0, 7645
Telemetry [candidate] (7.691 ms) : 0, 7691
section appsec
BytebuddyAgent [baseline] (700.202 ms) : 0, 700202
BytebuddyAgent [candidate] (700.26 ms) : 0, 700260
GlobalTracer [baseline] (302.387 ms) : 0, 302387
GlobalTracer [candidate] (302.385 ms) : 0, 302385
AppSec [baseline] (161.451 ms) : 0, 161451
AppSec [candidate] (162.608 ms) : 0, 162608
Remote Config [baseline] (642.041 µs) : 0, 642
Remote Config [candidate] (636.969 µs) : 0, 637
Telemetry [baseline] (8.06 ms) : 0, 8060
Telemetry [candidate] (8.815 ms) : 0, 8815
IAST [baseline] (20.587 ms) : 0, 20587
IAST [candidate] (21.118 ms) : 0, 21118
section iast
BytebuddyAgent [baseline] (794.737 ms) : 0, 794737
BytebuddyAgent [candidate] (791.651 ms) : 0, 791651
GlobalTracer [baseline] (300.607 ms) : 0, 300607
GlobalTracer [candidate] (298.679 ms) : 0, 298679
AppSec [baseline] (53.582 ms) : 0, 53582
AppSec [candidate] (53.337 ms) : 0, 53337
Remote Config [baseline] (625.449 µs) : 0, 625
Remote Config [candidate] (658.767 µs) : 0, 659
Telemetry [baseline] (7.4 ms) : 0, 7400
Telemetry [candidate] (7.388 ms) : 0, 7388
IAST [baseline] (25.534 ms) : 0, 25534
IAST [candidate] (24.653 ms) : 0, 24653
section profiling
ProfilingAgent [baseline] (95.773 ms) : 0, 95773
ProfilingAgent [candidate] (96.856 ms) : 0, 96856
BytebuddyAgent [baseline] (677.117 ms) : 0, 677117
BytebuddyAgent [candidate] (676.482 ms) : 0, 676482
GlobalTracer [baseline] (395.271 ms) : 0, 395271
GlobalTracer [candidate] (396.835 ms) : 0, 396835
AppSec [baseline] (54.587 ms) : 0, 54587
AppSec [candidate] (54.78 ms) : 0, 54780
Remote Config [baseline] (657.024 µs) : 0, 657
Remote Config [candidate] (664.5 µs) : 0, 665
Telemetry [baseline] (7.464 ms) : 0, 7464
Telemetry [candidate] (7.475 ms) : 0, 7475
Profiling [baseline] (95.796 ms) : 0, 95796
Profiling [candidate] (96.881 ms) : 0, 96881
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.40.0-SNAPSHOT~ac44f215cf, baseline=1.40.0-SNAPSHOT~58679718b3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1062941
Total [baseline] (8.508 s) : 0, 8507547
Agent [candidate] (1.064 s) : 0, 1064285
Total [candidate] (8.526 s) : 0, 8525592
section iast
Agent [baseline] (1.188 s) : 0, 1187923
Total [baseline] (8.993 s) : 0, 8992690
Agent [candidate] (1.206 s) : 0, 1206359
Total [candidate] (9.003 s) : 0, 9003166
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.188 s) : 0, 1188346
Total [baseline] (8.949 s) : 0, 8949283
Agent [candidate] (1.188 s) : 0, 1188168
Total [candidate] (9.001 s) : 0, 9000652
section iast_TELEMETRY_OFF
Agent [baseline] (1.194 s) : 0, 1194038
Total [baseline] (9.045 s) : 0, 9044946
Agent [candidate] (1.186 s) : 0, 1186198
Total [candidate] (8.989 s) : 0, 8989087
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.063 s -
Agent iast 1.188 s 124.982 ms (11.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.188 s 125.405 ms (11.8%)
Agent iast_TELEMETRY_OFF 1.194 s 131.098 ms (12.3%)
Total tracing 8.508 s -
Total iast 8.993 s 485.143 ms (5.7%)
Total iast_HARDCODED_SECRET_DISABLED 8.949 s 441.735 ms (5.2%)
Total iast_TELEMETRY_OFF 9.045 s 537.399 ms (6.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.064 s -
Agent iast 1.206 s 142.074 ms (13.3%)
Agent iast_HARDCODED_SECRET_DISABLED 1.188 s 123.883 ms (11.6%)
Agent iast_TELEMETRY_OFF 1.186 s 121.913 ms (11.5%)
Total tracing 8.526 s -
Total iast 9.003 s 477.574 ms (5.6%)
Total iast_HARDCODED_SECRET_DISABLED 9.001 s 475.06 ms (5.6%)
Total iast_TELEMETRY_OFF 8.989 s 463.496 ms (5.4%) 
gantt
    title insecure-bank - break down per module: candidate=1.40.0-SNAPSHOT~ac44f215cf, baseline=1.40.0-SNAPSHOT~58679718b3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (677.964 ms) : 0, 677964
BytebuddyAgent [candidate] (678.783 ms) : 0, 678783
GlobalTracer [baseline] (309.635 ms) : 0, 309635
GlobalTracer [candidate] (310.087 ms) : 0, 310087
AppSec [baseline] (53.507 ms) : 0, 53507
AppSec [candidate] (53.565 ms) : 0, 53565
Remote Config [baseline] (671.854 µs) : 0, 672
Remote Config [candidate] (665.269 µs) : 0, 665
Telemetry [baseline] (7.58 ms) : 0, 7580
Telemetry [candidate] (7.557 ms) : 0, 7557
section iast
BytebuddyAgent [baseline] (790.971 ms) : 0, 790971
BytebuddyAgent [candidate] (803.019 ms) : 0, 803019
GlobalTracer [baseline] (297.501 ms) : 0, 297501
GlobalTracer [candidate] (302.636 ms) : 0, 302636
AppSec [baseline] (53.77 ms) : 0, 53770
AppSec [candidate] (57.072 ms) : 0, 57072
Remote Config [baseline] (629.793 µs) : 0, 630
Remote Config [candidate] (604.215 µs) : 0, 604
Telemetry [baseline] (7.279 ms) : 0, 7279
Telemetry [candidate] (7.286 ms) : 0, 7286
IAST [baseline] (24.162 ms) : 0, 24162
IAST [candidate] (21.883 ms) : 0, 21883
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (791.153 ms) : 0, 791153
BytebuddyAgent [candidate] (790.025 ms) : 0, 790025
GlobalTracer [baseline] (298.069 ms) : 0, 298069
GlobalTracer [candidate] (298.112 ms) : 0, 298112
AppSec [baseline] (53.955 ms) : 0, 53955
AppSec [candidate] (54.97 ms) : 0, 54970
Remote Config [baseline] (618.742 µs) : 0, 619
Remote Config [candidate] (612.185 µs) : 0, 612
Telemetry [baseline] (7.333 ms) : 0, 7333
Telemetry [candidate] (7.322 ms) : 0, 7322
IAST [baseline] (23.561 ms) : 0, 23561
IAST [candidate] (23.49 ms) : 0, 23490
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (793.503 ms) : 0, 793503
BytebuddyAgent [candidate] (787.753 ms) : 0, 787753
GlobalTracer [baseline] (300.282 ms) : 0, 300282
GlobalTracer [candidate] (298.694 ms) : 0, 298694
AppSec [baseline] (52.713 ms) : 0, 52713
AppSec [candidate] (53.612 ms) : 0, 53612
Remote Config [baseline] (1.369 ms) : 0, 1369
Remote Config [candidate] (595.963 µs) : 0, 596
Telemetry [baseline] (7.294 ms) : 0, 7294
Telemetry [candidate] (7.132 ms) : 0, 7132
IAST [baseline] (25.152 ms) : 0, 25152
IAST [candidate] (24.786 ms) : 0, 24786
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-09-20T13:00:15 2024-09-20T13:07:04
git_branch master jpbempel/fix-eval-errors-sampling
git_commit_date 1726823241 1726836409
git_commit_sha 5867971 ac44f21
release_version 1.40.0-SNAPSHOT~58679718b3 1.40.0-SNAPSHOT~ac44f215cf
start_time 2024-09-20T13:00:02 2024-09-20T13:06:51
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1726837970 1726837970
ci_job_id 646380727 646380727
ci_pipeline_id 44786335 44786335
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 18 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~ac44f215cf, baseline=1.40.0-SNAPSHOT~58679718b3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.348 ms) : 1328, 1368
.   : milestone, 1348,
appsec (1.724 ms) : 1701, 1748
.   : milestone, 1724,
appsec_no_iast (1.697 ms) : 1672, 1722
.   : milestone, 1697,
iast (1.47 ms) : 1447, 1493
.   : milestone, 1470,
profiling (1.483 ms) : 1458, 1508
.   : milestone, 1483,
tracing (1.46 ms) : 1436, 1484
.   : milestone, 1460,
section candidate
no_agent (1.319 ms) : 1300, 1339
.   : milestone, 1319,
appsec (1.721 ms) : 1697, 1744
.   : milestone, 1721,
appsec_no_iast (1.725 ms) : 1700, 1749
.   : milestone, 1725,
iast (1.478 ms) : 1455, 1501
.   : milestone, 1478,
profiling (1.5 ms) : 1474, 1526
.   : milestone, 1500,
tracing (1.482 ms) : 1458, 1507
.   : milestone, 1482,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.348 ms [1.328 ms, 1.368 ms] -
appsec 1.724 ms [1.701 ms, 1.748 ms] 376.202 µs (27.9%)
appsec_no_iast 1.697 ms [1.672 ms, 1.722 ms] 348.931 µs (25.9%)
iast 1.47 ms [1.447 ms, 1.493 ms] 122.008 µs (9.1%)
profiling 1.483 ms [1.458 ms, 1.508 ms] 135.292 µs (10.0%)
tracing 1.46 ms [1.436 ms, 1.484 ms] 112.265 µs (8.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.319 ms [1.3 ms, 1.339 ms] -
appsec 1.721 ms [1.697 ms, 1.744 ms] 401.812 µs (30.5%)
appsec_no_iast 1.725 ms [1.7 ms, 1.749 ms] 405.75 µs (30.8%)
iast 1.478 ms [1.455 ms, 1.501 ms] 158.647 µs (12.0%)
profiling 1.5 ms [1.474 ms, 1.526 ms] 180.608 µs (13.7%)
tracing 1.482 ms [1.458 ms, 1.507 ms] 163.297 µs (12.4%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~ac44f215cf, baseline=1.40.0-SNAPSHOT~58679718b3
    dateFormat X
    axisFormat %s
section baseline
no_agent (365.293 µs) : 345, 385
.   : milestone, 365,
iast (478.803 µs) : 458, 500
.   : milestone, 479,
iast_FULL (549.392 µs) : 528, 571
.   : milestone, 549,
iast_GLOBAL (504.685 µs) : 483, 526
.   : milestone, 505,
iast_HARDCODED_SECRET_DISABLED (478.902 µs) : 458, 500
.   : milestone, 479,
iast_INACTIVE (447.77 µs) : 426, 469
.   : milestone, 448,
iast_TELEMETRY_OFF (466.417 µs) : 444, 489
.   : milestone, 466,
tracing (436.105 µs) : 416, 457
.   : milestone, 436,
section candidate
no_agent (373.579 µs) : 354, 393
.   : milestone, 374,
iast (481.022 µs) : 459, 503
.   : milestone, 481,
iast_FULL (543.825 µs) : 522, 565
.   : milestone, 544,
iast_GLOBAL (500.644 µs) : 479, 523
.   : milestone, 501,
iast_HARDCODED_SECRET_DISABLED (475.967 µs) : 455, 497
.   : milestone, 476,
iast_INACTIVE (441.815 µs) : 421, 463
.   : milestone, 442,
iast_TELEMETRY_OFF (473.07 µs) : 451, 496
.   : milestone, 473,
tracing (441.167 µs) : 421, 462
.   : milestone, 441,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 365.293 µs [345.126 µs, 385.459 µs] -
iast 478.803 µs [457.727 µs, 499.88 µs] 113.511 µs (31.1%)
iast_FULL 549.392 µs [528.114 µs, 570.67 µs] 184.099 µs (50.4%)
iast_GLOBAL 504.685 µs [482.952 µs, 526.419 µs] 139.393 µs (38.2%)
iast_HARDCODED_SECRET_DISABLED 478.902 µs [457.675 µs, 500.13 µs] 113.61 µs (31.1%)
iast_INACTIVE 447.77 µs [426.229 µs, 469.31 µs] 82.477 µs (22.6%)
iast_TELEMETRY_OFF 466.417 µs [443.865 µs, 488.968 µs] 101.124 µs (27.7%)
tracing 436.105 µs [415.585 µs, 456.625 µs] 70.812 µs (19.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 373.579 µs [353.993 µs, 393.166 µs] -
iast 481.022 µs [459.374 µs, 502.67 µs] 107.443 µs (28.8%)
iast_FULL 543.825 µs [522.42 µs, 565.23 µs] 170.245 µs (45.6%)
iast_GLOBAL 500.644 µs [478.716 µs, 522.573 µs] 127.065 µs (34.0%)
iast_HARDCODED_SECRET_DISABLED 475.967 µs [454.983 µs, 496.95 µs] 102.387 µs (27.4%)
iast_INACTIVE 441.815 µs [420.751 µs, 462.878 µs] 68.235 µs (18.3%)
iast_TELEMETRY_OFF 473.07 µs [450.562 µs, 495.577 µs] 99.49 µs (26.6%)
tracing 441.167 µs [420.652 µs, 461.683 µs] 67.588 µs (18.1%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/fix-eval-errors-sampling
git_commit_date 1726823241 1726836409
git_commit_sha 5867971 ac44f21
release_version 1.40.0-SNAPSHOT~58679718b3 1.40.0-SNAPSHOT~ac44f215cf
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1726838531 1726838531
ci_job_id 646380728 646380728
ci_pipeline_id 44786335 44786335
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~ac44f215cf, baseline=1.40.0-SNAPSHOT~58679718b3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.47 ms) : 1459, 1482
.   : milestone, 1470,
appsec (2.329 ms) : 2289, 2370
.   : milestone, 2329,
iast (2.053 ms) : 2004, 2102
.   : milestone, 2053,
iast_GLOBAL (2.097 ms) : 2046, 2147
.   : milestone, 2097,
profiling (1.928 ms) : 1889, 1968
.   : milestone, 1928,
tracing (1.899 ms) : 1861, 1936
.   : milestone, 1899,
section candidate
no_agent (1.47 ms) : 1459, 1482
.   : milestone, 1470,
appsec (2.317 ms) : 2276, 2357
.   : milestone, 2317,
iast (2.063 ms) : 2013, 2113
.   : milestone, 2063,
iast_GLOBAL (2.11 ms) : 2060, 2160
.   : milestone, 2110,
profiling (1.932 ms) : 1891, 1972
.   : milestone, 1932,
tracing (1.898 ms) : 1859, 1936
.   : milestone, 1898,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.47 ms [1.459 ms, 1.482 ms] -
appsec 2.329 ms [2.289 ms, 2.37 ms] 859.283 µs (58.4%)
iast 2.053 ms [2.004 ms, 2.102 ms] 582.729 µs (39.6%)
iast_GLOBAL 2.097 ms [2.046 ms, 2.147 ms] 626.539 µs (42.6%)
profiling 1.928 ms [1.889 ms, 1.968 ms] 458.201 µs (31.2%)
tracing 1.899 ms [1.861 ms, 1.936 ms] 428.565 µs (29.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.47 ms [1.459 ms, 1.482 ms] -
appsec 2.317 ms [2.276 ms, 2.357 ms] 846.378 µs (57.6%)
iast 2.063 ms [2.013 ms, 2.113 ms] 592.857 µs (40.3%)
iast_GLOBAL 2.11 ms [2.06 ms, 2.16 ms] 639.416 µs (43.5%)
profiling 1.932 ms [1.891 ms, 1.972 ms] 461.532 µs (31.4%)
tracing 1.898 ms [1.859 ms, 1.936 ms] 427.213 µs (29.1%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~ac44f215cf, baseline=1.40.0-SNAPSHOT~58679718b3
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.928 s) : 14928000, 14928000
.   : milestone, 14928000,
appsec (15.185 s) : 15185000, 15185000
.   : milestone, 15185000,
iast (18.996 s) : 18996000, 18996000
.   : milestone, 18996000,
iast_GLOBAL (17.811 s) : 17811000, 17811000
.   : milestone, 17811000,
profiling (15.036 s) : 15036000, 15036000
.   : milestone, 15036000,
tracing (15.016 s) : 15016000, 15016000
.   : milestone, 15016000,
section candidate
no_agent (14.897 s) : 14897000, 14897000
.   : milestone, 14897000,
appsec (15.231 s) : 15231000, 15231000
.   : milestone, 15231000,
iast (18.959 s) : 18959000, 18959000
.   : milestone, 18959000,
iast_GLOBAL (17.908 s) : 17908000, 17908000
.   : milestone, 17908000,
profiling (15.139 s) : 15139000, 15139000
.   : milestone, 15139000,
tracing (15.136 s) : 15136000, 15136000
.   : milestone, 15136000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.928 s [14.928 s, 14.928 s] -
appsec 15.185 s [15.185 s, 15.185 s] 257.0 ms (1.7%)
iast 18.996 s [18.996 s, 18.996 s] 4.068 s (27.3%)
iast_GLOBAL 17.811 s [17.811 s, 17.811 s] 2.883 s (19.3%)
profiling 15.036 s [15.036 s, 15.036 s] 108.0 ms (0.7%)
tracing 15.016 s [15.016 s, 15.016 s] 88.0 ms (0.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.897 s [14.897 s, 14.897 s] -
appsec 15.231 s [15.231 s, 15.231 s] 334.0 ms (2.2%)
iast 18.959 s [18.959 s, 18.959 s] 4.062 s (27.3%)
iast_GLOBAL 17.908 s [17.908 s, 17.908 s] 3.011 s (20.2%)
profiling 15.139 s [15.139 s, 15.139 s] 242.0 ms (1.6%)
tracing 15.136 s [15.136 s, 15.136 s] 239.0 ms (1.6%)

@jpbempel jpbempel merged commit aa45861 into master Sep 20, 2024
105 checks passed
@jpbempel jpbempel deleted the jpbempel/fix-eval-errors-sampling branch September 20, 2024 17:45
@github-actions github-actions bot added this to the 1.40.0 milestone Sep 20, 2024
jordan-wong pushed a commit that referenced this pull request Sep 23, 2024
@jpbempel @jordan-wong
Fix evaluation errors sampling (#7656)
4e22333 
evaluation error snapshots were sent even if rate limited
now we are checking sampled state before sending
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shatzi shatzi shatzi approved these changes

@evanchooly evanchooly Awaiting requested review from evanchooly evanchooly is a code owner automatically assigned from DataDog/debugger-java

Assignees
No one assigned
Labels
comp: debugger Dynamic Instrumentation type: bug
Projects
None yet
Milestone
1.40.0
Development

Successfully merging this pull request may close these issues.
2 participants
@jpbempel @shatzi