Support for kafka lag metrics in spark streaming applications #7474

kr-igor · 2024-08-19T19:51:46Z

Spark Structured Streaming doesn't commit Kafka offsets, instead they are being tracked in checkpoints. This PR adds offset tracking via DSM, which allows tracking Kafka lag per topic / partition.

pr-commenter · 2024-08-19T20:34:52Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	kr-igor/kafka-lag-spark-streaming
git_commit_date	1729693448	1729711068
git_commit_sha	`2153fe5`	`5e607a0`
release_version	1.42.0-SNAPSHOT~2153fe529d	1.42.0-SNAPSHOT~5e607a0494

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1729713500	1729713500
ci_job_id	682091562	682091562
ci_pipeline_id	47276917	47276917
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 metrics, 9 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.42.0-SNAPSHOT~5e607a0494, baseline=1.42.0-SNAPSHOT~2153fe529d

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.076 s) : 0, 1075777
Total [baseline] (10.413 s) : 0, 10412648
Agent [candidate] (1.083 s) : 0, 1083497
Total [candidate] (10.401 s) : 0, 10401480
section appsec
Agent [baseline] (1.209 s) : 0, 1208628
Total [baseline] (10.622 s) : 0, 10622491
Agent [candidate] (1.21 s) : 0, 1209819
Total [candidate] (10.614 s) : 0, 10613822
section iast
Agent [baseline] (1.205 s) : 0, 1205407
Total [baseline] (10.869 s) : 0, 10868730
Agent [candidate] (1.203 s) : 0, 1202917
Total [candidate] (10.881 s) : 0, 10881143
section profiling
Agent [baseline] (1.282 s) : 0, 1281569
Total [baseline] (10.76 s) : 0, 10759909
Agent [candidate] (1.276 s) : 0, 1276297
Total [candidate] (10.723 s) : 0, 10723201

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.076 s	-
Agent	appsec	1.209 s	132.852 ms (12.3%)
Agent	iast	1.205 s	129.63 ms (12.0%)
Agent	profiling	1.282 s	205.792 ms (19.1%)
Total	tracing	10.413 s	-
Total	appsec	10.622 s	209.844 ms (2.0%)
Total	iast	10.869 s	456.082 ms (4.4%)
Total	profiling	10.76 s	347.262 ms (3.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.083 s	-
Agent	appsec	1.21 s	126.323 ms (11.7%)
Agent	iast	1.203 s	119.42 ms (11.0%)
Agent	profiling	1.276 s	192.8 ms (17.8%)
Total	tracing	10.401 s	-
Total	appsec	10.614 s	212.342 ms (2.0%)
Total	iast	10.881 s	479.663 ms (4.6%)
Total	profiling	10.723 s	321.721 ms (3.1%)

gantt
    title petclinic - break down per module: candidate=1.42.0-SNAPSHOT~5e607a0494, baseline=1.42.0-SNAPSHOT~2153fe529d

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (685.356 ms) : 0, 685356
BytebuddyAgent [candidate] (690.495 ms) : 0, 690495
GlobalTracer [baseline] (313.175 ms) : 0, 313175
GlobalTracer [candidate] (314.75 ms) : 0, 314750
AppSec [baseline] (54.108 ms) : 0, 54108
AppSec [candidate] (54.224 ms) : 0, 54224
Remote Config [baseline] (664.159 µs) : 0, 664
Remote Config [candidate] (656.181 µs) : 0, 656
Telemetry [baseline] (8.844 ms) : 0, 8844
Telemetry [candidate] (9.642 ms) : 0, 9642
section appsec
BytebuddyAgent [baseline] (701.387 ms) : 0, 701387
BytebuddyAgent [candidate] (702.126 ms) : 0, 702126
GlobalTracer [baseline] (309.81 ms) : 0, 309810
GlobalTracer [candidate] (310.066 ms) : 0, 310066
AppSec [baseline] (165.868 ms) : 0, 165868
AppSec [candidate] (166.336 ms) : 0, 166336
IAST [baseline] (20.632 ms) : 0, 20632
IAST [candidate] (19.324 ms) : 0, 19324
Remote Config [baseline] (632.614 µs) : 0, 633
Remote Config [candidate] (632.45 µs) : 0, 632
Telemetry [baseline] (6.995 ms) : 0, 6995
Telemetry [candidate] (7.328 ms) : 0, 7328
section iast
BytebuddyAgent [baseline] (804.139 ms) : 0, 804139
BytebuddyAgent [candidate] (802.144 ms) : 0, 802144
GlobalTracer [baseline] (302.368 ms) : 0, 302368
GlobalTracer [candidate] (301.814 ms) : 0, 301814
AppSec [baseline] (57.127 ms) : 0, 57127
AppSec [candidate] (56.58 ms) : 0, 56580
IAST [baseline] (20.437 ms) : 0, 20437
IAST [candidate] (20.325 ms) : 0, 20325
Remote Config [baseline] (603.957 µs) : 0, 604
Remote Config [candidate] (591.646 µs) : 0, 592
Telemetry [baseline] (7.018 ms) : 0, 7018
Telemetry [candidate] (7.802 ms) : 0, 7802
section profiling
BytebuddyAgent [baseline] (685.225 ms) : 0, 685225
BytebuddyAgent [candidate] (681.178 ms) : 0, 681178
GlobalTracer [baseline] (398.904 ms) : 0, 398904
GlobalTracer [candidate] (398.117 ms) : 0, 398117
AppSec [baseline] (54.815 ms) : 0, 54815
AppSec [candidate] (54.447 ms) : 0, 54447
Remote Config [baseline] (655.954 µs) : 0, 656
Remote Config [candidate] (652.063 µs) : 0, 652
Telemetry [baseline] (13.673 ms) : 0, 13673
Telemetry [candidate] (12.811 ms) : 0, 12811
ProfilingAgent [baseline] (89.321 ms) : 0, 89321
ProfilingAgent [candidate] (90.359 ms) : 0, 90359
Profiling [baseline] (89.344 ms) : 0, 89344
Profiling [candidate] (90.382 ms) : 0, 90382

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.42.0-SNAPSHOT~5e607a0494, baseline=1.42.0-SNAPSHOT~2153fe529d

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.077 s) : 0, 1077235
Total [baseline] (8.582 s) : 0, 8581779
Agent [candidate] (1.081 s) : 0, 1080790
Total [candidate] (8.58 s) : 0, 8579992
section iast
Agent [baseline] (1.201 s) : 0, 1201205
Total [baseline] (9.105 s) : 0, 9104707
Agent [candidate] (1.201 s) : 0, 1200599
Total [candidate] (9.105 s) : 0, 9104915
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.216 s) : 0, 1215745
Total [baseline] (9.085 s) : 0, 9084931
Agent [candidate] (1.206 s) : 0, 1206077
Total [candidate] (9.084 s) : 0, 9083818
section iast_TELEMETRY_OFF
Agent [baseline] (1.196 s) : 0, 1196274
Total [baseline] (9.067 s) : 0, 9067131
Agent [candidate] (1.206 s) : 0, 1205605
Total [candidate] (9.132 s) : 0, 9131680

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.077 s	-
Agent	iast	1.201 s	123.97 ms (11.5%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.216 s	138.509 ms (12.9%)
Agent	iast_TELEMETRY_OFF	1.196 s	119.039 ms (11.1%)
Total	tracing	8.582 s	-
Total	iast	9.105 s	522.928 ms (6.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.085 s	503.152 ms (5.9%)
Total	iast_TELEMETRY_OFF	9.067 s	485.352 ms (5.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.081 s	-
Agent	iast	1.201 s	119.809 ms (11.1%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.206 s	125.287 ms (11.6%)
Agent	iast_TELEMETRY_OFF	1.206 s	124.814 ms (11.5%)
Total	tracing	8.58 s	-
Total	iast	9.105 s	524.923 ms (6.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.084 s	503.826 ms (5.9%)
Total	iast_TELEMETRY_OFF	9.132 s	551.687 ms (6.4%)

gantt
    title insecure-bank - break down per module: candidate=1.42.0-SNAPSHOT~5e607a0494, baseline=1.42.0-SNAPSHOT~2153fe529d

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.829 ms) : 0, 686829
BytebuddyAgent [candidate] (687.833 ms) : 0, 687833
GlobalTracer [baseline] (313.518 ms) : 0, 313518
GlobalTracer [candidate] (315.596 ms) : 0, 315596
AppSec [baseline] (53.743 ms) : 0, 53743
AppSec [candidate] (54.04 ms) : 0, 54040
Remote Config [baseline] (661.591 µs) : 0, 662
Remote Config [candidate] (670.824 µs) : 0, 671
Telemetry [baseline] (8.753 ms) : 0, 8753
Telemetry [candidate] (8.91 ms) : 0, 8910
section iast
BytebuddyAgent [baseline] (800.911 ms) : 0, 800911
BytebuddyAgent [candidate] (800.101 ms) : 0, 800101
GlobalTracer [baseline] (301.456 ms) : 0, 301456
GlobalTracer [candidate] (301.893 ms) : 0, 301893
AppSec [baseline] (56.545 ms) : 0, 56545
AppSec [candidate] (57.186 ms) : 0, 57186
IAST [baseline] (21.037 ms) : 0, 21037
IAST [candidate] (20.147 ms) : 0, 20147
Remote Config [baseline] (585.559 µs) : 0, 586
Remote Config [candidate] (574.073 µs) : 0, 574
Telemetry [baseline] (6.985 ms) : 0, 6985
Telemetry [candidate] (7.025 ms) : 0, 7025
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (811.772 ms) : 0, 811772
BytebuddyAgent [candidate] (804.299 ms) : 0, 804299
GlobalTracer [baseline] (305.041 ms) : 0, 305041
GlobalTracer [candidate] (302.974 ms) : 0, 302974
AppSec [baseline] (55.535 ms) : 0, 55535
AppSec [candidate] (57.062 ms) : 0, 57062
IAST [baseline] (21.991 ms) : 0, 21991
IAST [candidate] (20.266 ms) : 0, 20266
Remote Config [baseline] (594.559 µs) : 0, 595
Remote Config [candidate] (591.941 µs) : 0, 592
Telemetry [baseline] (6.994 ms) : 0, 6994
Telemetry [candidate] (7.112 ms) : 0, 7112
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (796.678 ms) : 0, 796678
BytebuddyAgent [candidate] (802.595 ms) : 0, 802595
GlobalTracer [baseline] (300.951 ms) : 0, 300951
GlobalTracer [candidate] (304.093 ms) : 0, 304093
AppSec [baseline] (57.532 ms) : 0, 57532
AppSec [candidate] (56.786 ms) : 0, 56786
IAST [baseline] (19.937 ms) : 0, 19937
IAST [candidate] (20.854 ms) : 0, 20854
Remote Config [baseline] (604.07 µs) : 0, 604
Remote Config [candidate] (598.403 µs) : 0, 598
Telemetry [baseline] (6.96 ms) : 0, 6960
Telemetry [candidate] (6.949 ms) : 0, 6949

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-10-23T19:28:47	2024-10-23T19:35:38
git_branch	master	kr-igor/kafka-lag-spark-streaming
git_commit_date	1729693448	1729711068
git_commit_sha	`2153fe5`	`5e607a0`
release_version	1.42.0-SNAPSHOT~2153fe529d	1.42.0-SNAPSHOT~5e607a0494
start_time	2024-10-23T19:28:34	2024-10-23T19:35:25

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1729712485	1729712485
ci_job_id	682091563	682091563
ci_pipeline_id	47276917	47276917
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.42.0-SNAPSHOT~5e607a0494, baseline=1.42.0-SNAPSHOT~2153fe529d
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.339 ms) : 1319, 1360
.   : milestone, 1339,
appsec (1.706 ms) : 1681, 1730
.   : milestone, 1706,
appsec_no_iast (1.712 ms) : 1686, 1738
.   : milestone, 1712,
iast (1.484 ms) : 1462, 1507
.   : milestone, 1484,
profiling (1.468 ms) : 1446, 1491
.   : milestone, 1468,
tracing (1.471 ms) : 1447, 1494
.   : milestone, 1471,
section candidate
no_agent (1.339 ms) : 1319, 1358
.   : milestone, 1339,
appsec (1.707 ms) : 1683, 1731
.   : milestone, 1707,
appsec_no_iast (1.714 ms) : 1689, 1739
.   : milestone, 1714,
iast (1.482 ms) : 1459, 1505
.   : milestone, 1482,
profiling (1.512 ms) : 1487, 1536
.   : milestone, 1512,
tracing (1.457 ms) : 1432, 1482
.   : milestone, 1457,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.339 ms [1.319 ms, 1.36 ms]	-
appsec	1.706 ms [1.681 ms, 1.73 ms]	366.443 µs (27.4%)
appsec_no_iast	1.712 ms [1.686 ms, 1.738 ms]	372.835 µs (27.8%)
iast	1.484 ms [1.462 ms, 1.507 ms]	144.971 µs (10.8%)
profiling	1.468 ms [1.446 ms, 1.491 ms]	128.84 µs (9.6%)
tracing	1.471 ms [1.447 ms, 1.494 ms]	131.337 µs (9.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.339 ms [1.319 ms, 1.358 ms]	-
appsec	1.707 ms [1.683 ms, 1.731 ms]	368.32 µs (27.5%)
appsec_no_iast	1.714 ms [1.689 ms, 1.739 ms]	375.258 µs (28.0%)
iast	1.482 ms [1.459 ms, 1.505 ms]	143.566 µs (10.7%)
profiling	1.512 ms [1.487 ms, 1.536 ms]	172.825 µs (12.9%)
tracing	1.457 ms [1.432 ms, 1.482 ms]	118.063 µs (8.8%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.42.0-SNAPSHOT~5e607a0494, baseline=1.42.0-SNAPSHOT~2153fe529d
    dateFormat X
    axisFormat %s
section baseline
no_agent (370.568 µs) : 350, 391
.   : milestone, 371,
iast (483.565 µs) : 462, 505
.   : milestone, 484,
iast_FULL (560.695 µs) : 540, 582
.   : milestone, 561,
iast_GLOBAL (517.237 µs) : 496, 539
.   : milestone, 517,
iast_HARDCODED_SECRET_DISABLED (492.96 µs) : 472, 514
.   : milestone, 493,
iast_INACTIVE (449.652 µs) : 429, 470
.   : milestone, 450,
iast_TELEMETRY_OFF (467.755 µs) : 447, 489
.   : milestone, 468,
tracing (446.827 µs) : 426, 467
.   : milestone, 447,
section candidate
no_agent (371.649 µs) : 352, 391
.   : milestone, 372,
iast (480.531 µs) : 459, 502
.   : milestone, 481,
iast_FULL (559.13 µs) : 538, 581
.   : milestone, 559,
iast_GLOBAL (518.325 µs) : 495, 541
.   : milestone, 518,
iast_HARDCODED_SECRET_DISABLED (485.107 µs) : 464, 507
.   : milestone, 485,
iast_INACTIVE (444.781 µs) : 424, 465
.   : milestone, 445,
iast_TELEMETRY_OFF (469.789 µs) : 449, 491
.   : milestone, 470,
tracing (446.462 µs) : 424, 469
.   : milestone, 446,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	370.568 µs [350.424 µs, 390.711 µs]	-
iast	483.565 µs [462.204 µs, 504.927 µs]	112.998 µs (30.5%)
iast_FULL	560.695 µs [539.543 µs, 581.848 µs]	190.127 µs (51.3%)
iast_GLOBAL	517.237 µs [495.694 µs, 538.779 µs]	146.669 µs (39.6%)
iast_HARDCODED_SECRET_DISABLED	492.96 µs [471.63 µs, 514.29 µs]	122.393 µs (33.0%)
iast_INACTIVE	449.652 µs [428.93 µs, 470.374 µs]	79.085 µs (21.3%)
iast_TELEMETRY_OFF	467.755 µs [446.825 µs, 488.686 µs]	97.187 µs (26.2%)
tracing	446.827 µs [426.321 µs, 467.334 µs]	76.26 µs (20.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	371.649 µs [352.185 µs, 391.113 µs]	-
iast	480.531 µs [459.246 µs, 501.817 µs]	108.882 µs (29.3%)
iast_FULL	559.13 µs [537.754 µs, 580.505 µs]	187.481 µs (50.4%)
iast_GLOBAL	518.325 µs [495.476 µs, 541.175 µs]	146.676 µs (39.5%)
iast_HARDCODED_SECRET_DISABLED	485.107 µs [463.654 µs, 506.56 µs]	113.458 µs (30.5%)
iast_INACTIVE	444.781 µs [424.284 µs, 465.278 µs]	73.132 µs (19.7%)
iast_TELEMETRY_OFF	469.789 µs [448.704 µs, 490.875 µs]	98.14 µs (26.4%)
tracing	446.462 µs [424.342 µs, 468.582 µs]	74.813 µs (20.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	kr-igor/kafka-lag-spark-streaming
git_commit_date	1729693448	1729711068
git_commit_sha	`2153fe5`	`5e607a0`
release_version	1.42.0-SNAPSHOT~2153fe529d	1.42.0-SNAPSHOT~5e607a0494

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1729713015	1729713015
ci_job_id	682091564	682091564
ci_pipeline_id	47276917	47276917
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.42.0-SNAPSHOT~5e607a0494, baseline=1.42.0-SNAPSHOT~2153fe529d
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.86 s) : 14860000, 14860000
.   : milestone, 14860000,
appsec (15.203 s) : 15203000, 15203000
.   : milestone, 15203000,
iast (19.024 s) : 19024000, 19024000
.   : milestone, 19024000,
iast_GLOBAL (17.987 s) : 17987000, 17987000
.   : milestone, 17987000,
profiling (14.899 s) : 14899000, 14899000
.   : milestone, 14899000,
tracing (15.23 s) : 15230000, 15230000
.   : milestone, 15230000,
section candidate
no_agent (14.955 s) : 14955000, 14955000
.   : milestone, 14955000,
appsec (15.045 s) : 15045000, 15045000
.   : milestone, 15045000,
iast (18.833 s) : 18833000, 18833000
.   : milestone, 18833000,
iast_GLOBAL (17.998 s) : 17998000, 17998000
.   : milestone, 17998000,
profiling (15.028 s) : 15028000, 15028000
.   : milestone, 15028000,
tracing (15.062 s) : 15062000, 15062000
.   : milestone, 15062000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.86 s [14.86 s, 14.86 s]	-
appsec	15.203 s [15.203 s, 15.203 s]	343.0 ms (2.3%)
iast	19.024 s [19.024 s, 19.024 s]	4.164 s (28.0%)
iast_GLOBAL	17.987 s [17.987 s, 17.987 s]	3.127 s (21.0%)
profiling	14.899 s [14.899 s, 14.899 s]	39.0 ms (0.3%)
tracing	15.23 s [15.23 s, 15.23 s]	370.0 ms (2.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.955 s [14.955 s, 14.955 s]	-
appsec	15.045 s [15.045 s, 15.045 s]	90.0 ms (0.6%)
iast	18.833 s [18.833 s, 18.833 s]	3.878 s (25.9%)
iast_GLOBAL	17.998 s [17.998 s, 17.998 s]	3.043 s (20.3%)
profiling	15.028 s [15.028 s, 15.028 s]	73.0 ms (0.5%)
tracing	15.062 s [15.062 s, 15.062 s]	107.0 ms (0.7%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.42.0-SNAPSHOT~5e607a0494, baseline=1.42.0-SNAPSHOT~2153fe529d
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.463 ms) : 1452, 1475
.   : milestone, 1463,
appsec (2.316 ms) : 2276, 2357
.   : milestone, 2316,
iast (2.072 ms) : 2019, 2124
.   : milestone, 2072,
iast_GLOBAL (2.124 ms) : 2071, 2177
.   : milestone, 2124,
profiling (1.923 ms) : 1882, 1964
.   : milestone, 1923,
tracing (1.905 ms) : 1866, 1944
.   : milestone, 1905,
section candidate
no_agent (1.465 ms) : 1453, 1476
.   : milestone, 1465,
appsec (2.323 ms) : 2282, 2365
.   : milestone, 2323,
iast (2.067 ms) : 2016, 2119
.   : milestone, 2067,
iast_GLOBAL (2.123 ms) : 2071, 2176
.   : milestone, 2123,
profiling (2.424 ms) : 2236, 2612
.   : milestone, 2424,
tracing (1.91 ms) : 1870, 1949
.   : milestone, 1910,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.463 ms [1.452 ms, 1.475 ms]	-
appsec	2.316 ms [2.276 ms, 2.357 ms]	852.912 µs (58.3%)
iast	2.072 ms [2.019 ms, 2.124 ms]	608.302 µs (41.6%)
iast_GLOBAL	2.124 ms [2.071 ms, 2.177 ms]	660.713 µs (45.2%)
profiling	1.923 ms [1.882 ms, 1.964 ms]	460.085 µs (31.4%)
tracing	1.905 ms [1.866 ms, 1.944 ms]	441.613 µs (30.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.465 ms [1.453 ms, 1.476 ms]	-
appsec	2.323 ms [2.282 ms, 2.365 ms]	858.481 µs (58.6%)
iast	2.067 ms [2.016 ms, 2.119 ms]	602.661 µs (41.1%)
iast_GLOBAL	2.123 ms [2.071 ms, 2.176 ms]	658.746 µs (45.0%)
profiling	2.424 ms [2.236 ms, 2.612 ms]	959.262 µs (65.5%)
tracing	1.91 ms [1.87 ms, 1.949 ms]	445.098 µs (30.4%)

PerfectSlayer · 2024-08-21T08:56:43Z

Hey @kr-igor

Can you check the https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#pull-request-guidelines for title format and labels? 🙏

amarziali · 2024-08-28T12:58:11Z

...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java

+    // check if this is a kafka source
+    if (progress.description().toLowerCase().startsWith("kafka")) {
+      try {
+        ObjectMapper objectMapper = new ObjectMapper();


That object can be safely cached.

amarziali · 2024-08-28T12:58:34Z

...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java

+                .trackBacklog(sortedTags, topicNode.get(partition).asLong());
+          }
+        }
+      } catch (Exception e) {


you might want to widen to Throwable

kr-igor · 2024-09-25T19:50:45Z

@amarziali can you please re-review? I've made the changes you requested. Thanks!

amarziali · 2024-10-21T15:44:04Z

...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java

+    }
+
+    // check if this is a kafka source
+    if (progress.description().toLowerCase().startsWith("kafka")) {


nit: a null check here is recommended just in case description one day can return null things (it's outside the try catch)

amarziali · 2024-10-21T15:44:41Z

...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java

+      try {
+        // parse offsets from endOffsets json, reported in a format:
+        // "topic" -> ["partition":value]
+        JsonNode jsonNode = objectMapper.readTree(progress.endOffset());


Is this library always available? I saw it was already imported.

Spark uses jackson for passing metadata, so It should be always imported
https://github.com/apache/spark/blob/3e597e2ec7f61d6944b154afbb5259f6ff7885d5/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala#L22

amarziali

LGTM. I left some comments for improvements

kr-igor added 2 commits August 19, 2024 14:33

Spotless apply

eff2fcf

Added a debug tag

d73762f

kr-igor added 4 commits August 19, 2024 15:35

Added debug logs

38d1812

Added a bit more logs

6346d74

Fixed parsing error

bf7ae0c

Added more logs

50612d7

kr-igor marked this pull request as ready for review August 20, 2024 17:26

kr-igor requested a review from a team as a code owner August 20, 2024 17:26

Final cleanup

e01394f

kr-igor changed the title ~~[DSM] - Kafka lag spark streaming~~ Support for data_streams.kafka.* metrics in spark streaming applications Aug 23, 2024

kr-igor changed the title ~~Support for data_streams.kafka.* metrics in spark streaming applications~~ Support for kafka lag metrics in spark streaming applications Aug 23, 2024

kr-igor added 2 commits August 23, 2024 13:15

Merge branch 'master' into kr-igor/kafka-lag-spark-streaming

4a06b50

Merge branch 'master' into kr-igor/kafka-lag-spark-streaming

4b6dc92

amarziali reviewed Aug 28, 2024

View reviewed changes

kr-igor added 4 commits August 30, 2024 09:43

Merge branch 'master' into kr-igor/kafka-lag-spark-streaming

320e6cd

Use static object mapper

4d58aa0

Merge branch 'master' into kr-igor/kafka-lag-spark-streaming

d01114e

Merge branch 'master' into kr-igor/kafka-lag-spark-streaming

bc53b4f

amarziali reviewed Oct 21, 2024

View reviewed changes

amarziali approved these changes Oct 21, 2024

View reviewed changes

kr-igor added 2 commits October 23, 2024 14:17

Merge branch 'master' into kr-igor/kafka-lag-spark-streaming

5e607a0

Added null check

6761cc3

kr-igor enabled auto-merge October 23, 2024 19:27

kr-igor merged commit 27861e4 into master Oct 23, 2024
99 of 102 checks passed

kr-igor deleted the kr-igor/kafka-lag-spark-streaming branch October 23, 2024 19:59

github-actions bot added this to the 1.42.0 milestone Oct 23, 2024

jordan-wong added the inst: apache spark Apache Spark instrumentation label Nov 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for kafka lag metrics in spark streaming applications #7474

Support for kafka lag metrics in spark streaming applications #7474

kr-igor commented Aug 19, 2024 •

edited

Loading

pr-commenter bot commented Aug 19, 2024 •

edited

Loading

PerfectSlayer commented Aug 21, 2024

amarziali Aug 28, 2024

amarziali Aug 28, 2024

kr-igor commented Sep 25, 2024

amarziali Oct 21, 2024

amarziali Oct 21, 2024

kr-igor Oct 23, 2024

amarziali left a comment

Support for kafka lag metrics in spark streaming applications #7474

Support for kafka lag metrics in spark streaming applications #7474

Conversation

kr-igor commented Aug 19, 2024 • edited Loading

pr-commenter bot commented Aug 19, 2024 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

PerfectSlayer commented Aug 21, 2024

amarziali Aug 28, 2024

Choose a reason for hiding this comment

amarziali Aug 28, 2024

Choose a reason for hiding this comment

kr-igor commented Sep 25, 2024

amarziali Oct 21, 2024

Choose a reason for hiding this comment

amarziali Oct 21, 2024

Choose a reason for hiding this comment

kr-igor Oct 23, 2024

Choose a reason for hiding this comment

amarziali left a comment

Choose a reason for hiding this comment

kr-igor commented Aug 19, 2024 •

edited

Loading

pr-commenter bot commented Aug 19, 2024 •

edited

Loading