Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enabled kafka-clients 3.8+ by default #7818

Merged
merged 1 commit into from
Oct 22, 2024
Merged

Enabled kafka-clients 3.8+ by default #7818

merged 1 commit into from
Oct 22, 2024

Conversation

nayeem-kamal
Copy link
Contributor

@nayeem-kamal nayeem-kamal commented Oct 21, 2024 •
edited
Loading

What Does This Do

This PR enables the kafka-clients-3.8 instrumentation which was previously disabled due to issues with testing in CI.

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@nayeem-kamal nayeem-kamal added the inst: kafka Kafka instrumentation label Oct 21, 2024
@pr-commenter
Copy link

pr-commenter bot commented Oct 21, 2024

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nayeem-kamal/kafka-clients-enabled
git_commit_date 1729520735 1729521392
git_commit_sha a11c206 8c7495a
See matching parameters
Baseline Candidate
ci_job_date 1729522477 1729522477
ci_job_id 678516954 678516954
ci_pipeline_id 47046451 47046451
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.21 11.0.21
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.21+9-post-Ubuntu-0ubuntu122.04 11.0.21+9-post-Ubuntu-0ubuntu122.04

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 1 metrics, 0 unstable metrics.

scenario Δ mean throughput
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce worse
[-54306.326op/s; -48654.577op/s] or [-31.514%; -28.234%]
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce worse
[-48740.165op/s; -44147.039op/s] or [-35.156%; -31.843%]
See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce same

@pr-commenter
Copy link

pr-commenter bot commented Oct 21, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nayeem-kamal/kafka-clients-enabled
git_commit_date 1729520735 1729521392
git_commit_sha a11c206 8c7495a
release_version 1.42.0-SNAPSHOT~a11c2065b4 1.42.0-SNAPSHOT~8c7495a542
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1729523737 1729523737
ci_job_id 678516950 678516950
ci_pipeline_id 47046451 47046451
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.42.0-SNAPSHOT~8c7495a542, baseline=1.42.0-SNAPSHOT~a11c2065b4

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.087 s) : 0, 1086609
Total [baseline] (8.593 s) : 0, 8592908
Agent [candidate] (1.085 s) : 0, 1085266
Total [candidate] (8.609 s) : 0, 8608880
section iast
Agent [baseline] (1.215 s) : 0, 1215455
Total [baseline] (9.135 s) : 0, 9134692
Agent [candidate] (1.213 s) : 0, 1213114
Total [candidate] (9.12 s) : 0, 9119905
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.213 s) : 0, 1212820
Total [baseline] (9.136 s) : 0, 9136420
Agent [candidate] (1.212 s) : 0, 1212044
Total [candidate] (9.107 s) : 0, 9107365
section iast_TELEMETRY_OFF
Agent [baseline] (1.203 s) : 0, 1202687
Total [baseline] (9.111 s) : 0, 9110603
Agent [candidate] (1.201 s) : 0, 1201498
Total [candidate] (9.087 s) : 0, 9087442
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.087 s -
Agent iast 1.215 s 128.846 ms (11.9%)
Agent iast_HARDCODED_SECRET_DISABLED 1.213 s 126.211 ms (11.6%)
Agent iast_TELEMETRY_OFF 1.203 s 116.077 ms (10.7%)
Total tracing 8.593 s -
Total iast 9.135 s 541.785 ms (6.3%)
Total iast_HARDCODED_SECRET_DISABLED 9.136 s 543.512 ms (6.3%)
Total iast_TELEMETRY_OFF 9.111 s 517.696 ms (6.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.085 s -
Agent iast 1.213 s 127.848 ms (11.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.212 s 126.778 ms (11.7%)
Agent iast_TELEMETRY_OFF 1.201 s 116.232 ms (10.7%)
Total tracing 8.609 s -
Total iast 9.12 s 511.025 ms (5.9%)
Total iast_HARDCODED_SECRET_DISABLED 9.107 s 498.485 ms (5.8%)
Total iast_TELEMETRY_OFF 9.087 s 478.562 ms (5.6%) 
gantt
    title insecure-bank - break down per module: candidate=1.42.0-SNAPSHOT~8c7495a542, baseline=1.42.0-SNAPSHOT~a11c2065b4

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (692.711 ms) : 0, 692711
BytebuddyAgent [candidate] (691.846 ms) : 0, 691846
GlobalTracer [baseline] (314.873 ms) : 0, 314873
GlobalTracer [candidate] (315.466 ms) : 0, 315466
AppSec [baseline] (54.259 ms) : 0, 54259
AppSec [candidate] (54.453 ms) : 0, 54453
Remote Config [baseline] (673.372 µs) : 0, 673
Remote Config [candidate] (669.512 µs) : 0, 670
Telemetry [baseline] (10.261 ms) : 0, 10261
Telemetry [candidate] (8.951 ms) : 0, 8951
section iast
BytebuddyAgent [baseline] (810.321 ms) : 0, 810321
BytebuddyAgent [candidate] (809.093 ms) : 0, 809093
GlobalTracer [baseline] (304.572 ms) : 0, 304572
GlobalTracer [candidate] (303.824 ms) : 0, 303824
AppSec [baseline] (57.332 ms) : 0, 57332
AppSec [candidate] (56.481 ms) : 0, 56481
IAST [baseline] (21.557 ms) : 0, 21557
IAST [candidate] (22.131 ms) : 0, 22131
Remote Config [baseline] (626.595 µs) : 0, 627
Remote Config [candidate] (608.026 µs) : 0, 608
Telemetry [baseline] (7.152 ms) : 0, 7152
Telemetry [candidate] (7.075 ms) : 0, 7075
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (808.259 ms) : 0, 808259
BytebuddyAgent [candidate] (807.515 ms) : 0, 807515
GlobalTracer [baseline] (303.335 ms) : 0, 303335
GlobalTracer [candidate] (304.215 ms) : 0, 304215
AppSec [baseline] (57.775 ms) : 0, 57775
AppSec [candidate] (57.117 ms) : 0, 57117
IAST [baseline] (20.87 ms) : 0, 20870
IAST [candidate] (21.573 ms) : 0, 21573
Remote Config [baseline] (629.85 µs) : 0, 630
Remote Config [candidate] (626.094 µs) : 0, 626
Telemetry [baseline] (8.076 ms) : 0, 8076
Telemetry [candidate] (7.11 ms) : 0, 7110
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (800.335 ms) : 0, 800335
BytebuddyAgent [candidate] (799.825 ms) : 0, 799825
GlobalTracer [baseline] (302.75 ms) : 0, 302750
GlobalTracer [candidate] (302.589 ms) : 0, 302589
AppSec [baseline] (57.96 ms) : 0, 57960
AppSec [candidate] (57.451 ms) : 0, 57451
IAST [baseline] (20.219 ms) : 0, 20219
IAST [candidate] (20.227 ms) : 0, 20227
Remote Config [baseline] (615.543 µs) : 0, 616
Remote Config [candidate] (599.808 µs) : 0, 600
Telemetry [baseline] (7.036 ms) : 0, 7036
Telemetry [candidate] (7.033 ms) : 0, 7033
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.42.0-SNAPSHOT~8c7495a542, baseline=1.42.0-SNAPSHOT~a11c2065b4

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.088 s) : 0, 1087615
Total [baseline] (10.401 s) : 0, 10400524
Agent [candidate] (1.087 s) : 0, 1086761
Total [candidate] (10.426 s) : 0, 10426361
section appsec
Agent [baseline] (1.219 s) : 0, 1218877
Total [baseline] (10.645 s) : 0, 10645433
Agent [candidate] (1.215 s) : 0, 1214645
Total [candidate] (10.64 s) : 0, 10639943
section iast
Agent [baseline] (1.203 s) : 0, 1203288
Total [baseline] (10.845 s) : 0, 10845223
Agent [candidate] (1.22 s) : 0, 1219613
Total [candidate] (10.883 s) : 0, 10883056
section profiling
Agent [baseline] (1.272 s) : 0, 1272442
Total [baseline] (10.63 s) : 0, 10629520
Agent [candidate] (1.275 s) : 0, 1275337
Total [candidate] (10.737 s) : 0, 10736583
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.088 s -
Agent appsec 1.219 s 131.262 ms (12.1%)
Agent iast 1.203 s 115.673 ms (10.6%)
Agent profiling 1.272 s 184.827 ms (17.0%)
Total tracing 10.401 s -
Total appsec 10.645 s 244.909 ms (2.4%)
Total iast 10.845 s 444.699 ms (4.3%)
Total profiling 10.63 s 228.997 ms (2.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.087 s -
Agent appsec 1.215 s 127.884 ms (11.8%)
Agent iast 1.22 s 132.852 ms (12.2%)
Agent profiling 1.275 s 188.576 ms (17.4%)
Total tracing 10.426 s -
Total appsec 10.64 s 213.582 ms (2.0%)
Total iast 10.883 s 456.696 ms (4.4%)
Total profiling 10.737 s 310.222 ms (3.0%) 
gantt
    title petclinic - break down per module: candidate=1.42.0-SNAPSHOT~8c7495a542, baseline=1.42.0-SNAPSHOT~a11c2065b4

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (692.799 ms) : 0, 692799
BytebuddyAgent [candidate] (691.955 ms) : 0, 691955
GlobalTracer [baseline] (315.896 ms) : 0, 315896
GlobalTracer [candidate] (314.924 ms) : 0, 314924
AppSec [baseline] (54.696 ms) : 0, 54696
AppSec [candidate] (54.459 ms) : 0, 54459
Remote Config [baseline] (687.72 µs) : 0, 688
Remote Config [candidate] (663.516 µs) : 0, 664
Telemetry [baseline] (9.679 ms) : 0, 9679
Telemetry [candidate] (10.923 ms) : 0, 10923
section appsec
BytebuddyAgent [baseline] (708.527 ms) : 0, 708527
BytebuddyAgent [candidate] (705.402 ms) : 0, 705402
GlobalTracer [baseline] (311.747 ms) : 0, 311747
GlobalTracer [candidate] (311.144 ms) : 0, 311144
AppSec [baseline] (165.232 ms) : 0, 165232
AppSec [candidate] (165.418 ms) : 0, 165418
Remote Config [baseline] (636.046 µs) : 0, 636
Remote Config [candidate] (641.973 µs) : 0, 642
Telemetry [baseline] (8.119 ms) : 0, 8119
Telemetry [candidate] (8.142 ms) : 0, 8142
IAST [baseline] (21.606 ms) : 0, 21606
IAST [candidate] (20.27 ms) : 0, 20270
section iast
BytebuddyAgent [baseline] (801.514 ms) : 0, 801514
BytebuddyAgent [candidate] (815.296 ms) : 0, 815296
GlobalTracer [baseline] (302.218 ms) : 0, 302218
GlobalTracer [candidate] (303.732 ms) : 0, 303732
AppSec [baseline] (56.838 ms) : 0, 56838
AppSec [candidate] (57.466 ms) : 0, 57466
Remote Config [baseline] (607.687 µs) : 0, 608
Remote Config [candidate] (614.795 µs) : 0, 615
Telemetry [baseline] (7.081 ms) : 0, 7081
Telemetry [candidate] (7.17 ms) : 0, 7170
IAST [baseline] (21.284 ms) : 0, 21284
IAST [candidate] (21.396 ms) : 0, 21396
section profiling
BytebuddyAgent [baseline] (679.086 ms) : 0, 679086
BytebuddyAgent [candidate] (681.565 ms) : 0, 681565
GlobalTracer [baseline] (396.066 ms) : 0, 396066
GlobalTracer [candidate] (397.097 ms) : 0, 397097
AppSec [baseline] (54.455 ms) : 0, 54455
AppSec [candidate] (54.505 ms) : 0, 54505
Remote Config [baseline] (654.335 µs) : 0, 654
Remote Config [candidate] (647.529 µs) : 0, 648
Telemetry [baseline] (11.605 ms) : 0, 11605
Telemetry [candidate] (13.481 ms) : 0, 13481
ProfilingAgent [baseline] (91.746 ms) : 0, 91746
ProfilingAgent [candidate] (89.092 ms) : 0, 89092
Profiling [baseline] (91.769 ms) : 0, 91769
Profiling [candidate] (89.115 ms) : 0, 89115
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-10-21T14:48:08 2024-10-21T14:54:57
git_branch master nayeem-kamal/kafka-clients-enabled
git_commit_date 1729520735 1729521392
git_commit_sha a11c206 8c7495a
release_version 1.42.0-SNAPSHOT~a11c2065b4 1.42.0-SNAPSHOT~8c7495a542
start_time 2024-10-21T14:47:55 2024-10-21T14:54:43
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1729522844 1729522844
ci_job_id 678516951 678516951
ci_pipeline_id 47046451 47046451
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.42.0-SNAPSHOT~8c7495a542, baseline=1.42.0-SNAPSHOT~a11c2065b4
    dateFormat X
    axisFormat %s
section baseline
no_agent (368.336 µs) : 348, 389
.   : milestone, 368,
iast (478.397 µs) : 457, 499
.   : milestone, 478,
iast_FULL (548.862 µs) : 528, 570
.   : milestone, 549,
iast_GLOBAL (507.521 µs) : 486, 529
.   : milestone, 508,
iast_HARDCODED_SECRET_DISABLED (474.837 µs) : 454, 496
.   : milestone, 475,
iast_INACTIVE (444.152 µs) : 423, 465
.   : milestone, 444,
iast_TELEMETRY_OFF (466.781 µs) : 445, 488
.   : milestone, 467,
tracing (436.062 µs) : 416, 457
.   : milestone, 436,
section candidate
no_agent (368.777 µs) : 346, 391
.   : milestone, 369,
iast (480.379 µs) : 459, 502
.   : milestone, 480,
iast_FULL (550.179 µs) : 529, 572
.   : milestone, 550,
iast_GLOBAL (504.747 µs) : 483, 526
.   : milestone, 505,
iast_HARDCODED_SECRET_DISABLED (481.362 µs) : 460, 503
.   : milestone, 481,
iast_INACTIVE (443.326 µs) : 422, 464
.   : milestone, 443,
iast_TELEMETRY_OFF (468.953 µs) : 447, 491
.   : milestone, 469,
tracing (442.137 µs) : 421, 463
.   : milestone, 442,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.336 µs [348.105 µs, 388.567 µs] -
iast 478.397 µs [457.44 µs, 499.354 µs] 110.061 µs (29.9%)
iast_FULL 548.862 µs [527.612 µs, 570.112 µs] 180.526 µs (49.0%)
iast_GLOBAL 507.521 µs [485.609 µs, 529.433 µs] 139.185 µs (37.8%)
iast_HARDCODED_SECRET_DISABLED 474.837 µs [453.999 µs, 495.675 µs] 106.501 µs (28.9%)
iast_INACTIVE 444.152 µs [423.346 µs, 464.958 µs] 75.816 µs (20.6%)
iast_TELEMETRY_OFF 466.781 µs [445.236 µs, 488.327 µs] 98.445 µs (26.7%)
tracing 436.062 µs [415.604 µs, 456.519 µs] 67.726 µs (18.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.777 µs [346.465 µs, 391.089 µs] -
iast 480.379 µs [458.795 µs, 501.963 µs] 111.602 µs (30.3%)
iast_FULL 550.179 µs [528.771 µs, 571.588 µs] 181.403 µs (49.2%)
iast_GLOBAL 504.747 µs [483.455 µs, 526.038 µs] 135.97 µs (36.9%)
iast_HARDCODED_SECRET_DISABLED 481.362 µs [460.011 µs, 502.713 µs] 112.586 µs (30.5%)
iast_INACTIVE 443.326 µs [422.352 µs, 464.3 µs] 74.549 µs (20.2%)
iast_TELEMETRY_OFF 468.953 µs [447.348 µs, 490.558 µs] 100.176 µs (27.2%)
tracing 442.137 µs [421.279 µs, 462.996 µs] 73.361 µs (19.9%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.42.0-SNAPSHOT~8c7495a542, baseline=1.42.0-SNAPSHOT~a11c2065b4
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.338 ms) : 1319, 1357
.   : milestone, 1338,
appsec (1.714 ms) : 1691, 1738
.   : milestone, 1714,
appsec_no_iast (1.698 ms) : 1674, 1722
.   : milestone, 1698,
iast (1.473 ms) : 1450, 1496
.   : milestone, 1473,
profiling (1.481 ms) : 1459, 1504
.   : milestone, 1481,
tracing (1.465 ms) : 1440, 1489
.   : milestone, 1465,
section candidate
no_agent (1.332 ms) : 1312, 1352
.   : milestone, 1332,
appsec (1.7 ms) : 1677, 1723
.   : milestone, 1700,
appsec_no_iast (1.717 ms) : 1692, 1742
.   : milestone, 1717,
iast (1.482 ms) : 1460, 1505
.   : milestone, 1482,
profiling (1.526 ms) : 1503, 1549
.   : milestone, 1526,
tracing (1.47 ms) : 1446, 1494
.   : milestone, 1470,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.338 ms [1.319 ms, 1.357 ms] -
appsec 1.714 ms [1.691 ms, 1.738 ms] 376.475 µs (28.1%)
appsec_no_iast 1.698 ms [1.674 ms, 1.722 ms] 360.207 µs (26.9%)
iast 1.473 ms [1.45 ms, 1.496 ms] 135.175 µs (10.1%)
profiling 1.481 ms [1.459 ms, 1.504 ms] 143.54 µs (10.7%)
tracing 1.465 ms [1.44 ms, 1.489 ms] 126.608 µs (9.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.332 ms [1.312 ms, 1.352 ms] -
appsec 1.7 ms [1.677 ms, 1.723 ms] 367.804 µs (27.6%)
appsec_no_iast 1.717 ms [1.692 ms, 1.742 ms] 384.844 µs (28.9%)
iast 1.482 ms [1.46 ms, 1.505 ms] 150.053 µs (11.3%)
profiling 1.526 ms [1.503 ms, 1.549 ms] 193.937 µs (14.6%)
tracing 1.47 ms [1.446 ms, 1.494 ms] 138.248 µs (10.4%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nayeem-kamal/kafka-clients-enabled
git_commit_date 1729520735 1729521392
git_commit_sha a11c206 8c7495a
release_version 1.42.0-SNAPSHOT~a11c2065b4 1.42.0-SNAPSHOT~8c7495a542
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1729523257 1729523257
ci_job_id 678516952 678516952
ci_pipeline_id 47046451 47046451
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.42.0-SNAPSHOT~8c7495a542, baseline=1.42.0-SNAPSHOT~a11c2065b4
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.85 s) : 15850000, 15850000
.   : milestone, 15850000,
appsec (15.326 s) : 15326000, 15326000
.   : milestone, 15326000,
iast (18.814 s) : 18814000, 18814000
.   : milestone, 18814000,
iast_GLOBAL (18.039 s) : 18039000, 18039000
.   : milestone, 18039000,
profiling (15.257 s) : 15257000, 15257000
.   : milestone, 15257000,
tracing (15.079 s) : 15079000, 15079000
.   : milestone, 15079000,
section candidate
no_agent (15.403 s) : 15403000, 15403000
.   : milestone, 15403000,
appsec (15.252 s) : 15252000, 15252000
.   : milestone, 15252000,
iast (18.794 s) : 18794000, 18794000
.   : milestone, 18794000,
iast_GLOBAL (18.095 s) : 18095000, 18095000
.   : milestone, 18095000,
profiling (14.987 s) : 14987000, 14987000
.   : milestone, 14987000,
tracing (15.459 s) : 15459000, 15459000
.   : milestone, 15459000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.85 s [15.85 s, 15.85 s] -
appsec 15.326 s [15.326 s, 15.326 s] -524.0 ms (-3.3%)
iast 18.814 s [18.814 s, 18.814 s] 2.964 s (18.7%)
iast_GLOBAL 18.039 s [18.039 s, 18.039 s] 2.189 s (13.8%)
profiling 15.257 s [15.257 s, 15.257 s] -593.0 ms (-3.7%)
tracing 15.079 s [15.079 s, 15.079 s] -771.0 ms (-4.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.403 s [15.403 s, 15.403 s] -
appsec 15.252 s [15.252 s, 15.252 s] -151.0 ms (-1.0%)
iast 18.794 s [18.794 s, 18.794 s] 3.391 s (22.0%)
iast_GLOBAL 18.095 s [18.095 s, 18.095 s] 2.692 s (17.5%)
profiling 14.987 s [14.987 s, 14.987 s] -416.0 ms (-2.7%)
tracing 15.459 s [15.459 s, 15.459 s] 56.0 ms (0.4%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.42.0-SNAPSHOT~8c7495a542, baseline=1.42.0-SNAPSHOT~a11c2065b4
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.46 ms) : 1448, 1471
.   : milestone, 1460,
appsec (2.306 ms) : 2266, 2347
.   : milestone, 2306,
iast (2.071 ms) : 2019, 2123
.   : milestone, 2071,
iast_GLOBAL (2.119 ms) : 2066, 2172
.   : milestone, 2119,
profiling (1.939 ms) : 1897, 1982
.   : milestone, 1939,
tracing (1.908 ms) : 1868, 1947
.   : milestone, 1908,
section candidate
no_agent (1.463 ms) : 1452, 1475
.   : milestone, 1463,
appsec (2.323 ms) : 2282, 2364
.   : milestone, 2323,
iast (2.067 ms) : 2015, 2119
.   : milestone, 2067,
iast_GLOBAL (2.12 ms) : 2067, 2173
.   : milestone, 2120,
profiling (1.945 ms) : 1902, 1988
.   : milestone, 1945,
tracing (1.919 ms) : 1879, 1959
.   : milestone, 1919,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.46 ms [1.448 ms, 1.471 ms] -
appsec 2.306 ms [2.266 ms, 2.347 ms] 846.632 µs (58.0%)
iast 2.071 ms [2.019 ms, 2.123 ms] 610.938 µs (41.9%)
iast_GLOBAL 2.119 ms [2.066 ms, 2.172 ms] 659.38 µs (45.2%)
profiling 1.939 ms [1.897 ms, 1.982 ms] 479.457 µs (32.8%)
tracing 1.908 ms [1.868 ms, 1.947 ms] 447.918 µs (30.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.463 ms [1.452 ms, 1.475 ms] -
appsec 2.323 ms [2.282 ms, 2.364 ms] 859.286 µs (58.7%)
iast 2.067 ms [2.015 ms, 2.119 ms] 603.297 µs (41.2%)
iast_GLOBAL 2.12 ms [2.067 ms, 2.173 ms] 656.741 µs (44.9%)
profiling 1.945 ms [1.902 ms, 1.988 ms] 481.345 µs (32.9%)
tracing 1.919 ms [1.879 ms, 1.959 ms] 455.759 µs (31.1%)

@pr-commenter
Copy link

pr-commenter bot commented Oct 21, 2024

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nayeem-kamal/kafka-clients-enabled
git_commit_date 1729520735 1729521392
git_commit_sha a11c206 8c7495a
See matching parameters
Baseline Candidate
ci_job_date 1729522530 1729522530
ci_job_id 678516955 678516955
ci_pipeline_id 47046451 47046451
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.21 11.0.21
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.21+9-post-Ubuntu-0ubuntu122.04 11.0.21+9-post-Ubuntu-0ubuntu122.04

Summary

Found 1 performance improvements and 1 performance regressions! Performance is the same for 1 metrics, 0 unstable metrics.

scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume worse
[-14755.043op/s; -6030.994op/s] or [-4.907%; -2.006%]
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume better
[+4580.010op/s; +8876.188op/s] or [+2.663%; +5.161%]
See unchanged results
scenario Δ mean throughput
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

@nayeem-kamal nayeem-kamal changed the title Enable kafka-clients by default Enable kafka-clients 3.8+ by default Oct 22, 2024
@nayeem-kamal nayeem-kamal marked this pull request as ready for review October 22, 2024 15:09
@nayeem-kamal nayeem-kamal requested review from a team as code owners October 22, 2024 15:09
@nayeem-kamal nayeem-kamal changed the title Enable kafka-clients 3.8+ by default Enabled kafka-clients 3.8+ by default Oct 22, 2024
@nayeem-kamal nayeem-kamal merged commit 179c62a into master Oct 22, 2024
108 checks passed
@nayeem-kamal nayeem-kamal deleted the nayeem-kamal/kafka-clients-enabled branch October 22, 2024 15:10
@github-actions github-actions bot added this to the 1.42.0 milestone Oct 22, 2024
@nayeem-kamal nayeem-kamal mentioned this pull request Nov 7, 2024
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dougqh dougqh dougqh approved these changes

@manuel-alvarez-alvarez manuel-alvarez-alvarez Awaiting requested review from manuel-alvarez-alvarez manuel-alvarez-alvarez is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
inst: kafka Kafka instrumentation
Projects
None yet
Milestone
1.42.0
Development

Successfully merging this pull request may close these issues.
2 participants
@nayeem-kamal @dougqh