[WIP] Add file attribute to JUnit reports #8216
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 5 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~2d66a235f6, baseline=1.46.0-SNAPSHOT~0da31f6ae9
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1054449
Total [baseline] (8.627 s) : 0, 8627400
Agent [candidate] (1.052 s) : 0, 1052322
Total [candidate] (8.612 s) : 0, 8611809
section iast
Agent [baseline] (1.199 s) : 0, 1198803
Total [baseline] (9.269 s) : 0, 9268667
Agent [candidate] (1.179 s) : 0, 1179425
Total [candidate] (9.206 s) : 0, 9205973
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.188 s) : 0, 1187790
Total [baseline] (9.2 s) : 0, 9199665
Agent [candidate] (1.185 s) : 0, 1184925
Total [candidate] (9.237 s) : 0, 9236842
section iast_TELEMETRY_OFF
Agent [baseline] (1.191 s) : 0, 1191176
Total [baseline] (9.2 s) : 0, 9199985
Agent [candidate] (1.192 s) : 0, 1191570
Total [candidate] (9.219 s) : 0, 9218589
gantt
title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~2d66a235f6, baseline=1.46.0-SNAPSHOT~0da31f6ae9
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (713.6 ms) : 0, 713600
BytebuddyAgent [candidate] (713.229 ms) : 0, 713229
GlobalTracer [baseline] (255.427 ms) : 0, 255427
GlobalTracer [candidate] (255.086 ms) : 0, 255086
AppSec [baseline] (55.949 ms) : 0, 55949
AppSec [candidate] (56.043 ms) : 0, 56043
Remote Config [baseline] (714.343 µs) : 0, 714
Remote Config [candidate] (722.739 µs) : 0, 723
Telemetry [baseline] (13.764 ms) : 0, 13764
Telemetry [candidate] (12.256 ms) : 0, 12256
section iast
BytebuddyAgent [baseline] (843.442 ms) : 0, 843442
BytebuddyAgent [candidate] (829.594 ms) : 0, 829594
GlobalTracer [baseline] (249.301 ms) : 0, 249301
GlobalTracer [candidate] (246.236 ms) : 0, 246236
AppSec [baseline] (58.789 ms) : 0, 58789
AppSec [candidate] (57.679 ms) : 0, 57679
Remote Config [baseline] (697.935 µs) : 0, 698
Remote Config [candidate] (656.998 µs) : 0, 657
Telemetry [baseline] (9.117 ms) : 0, 9117
Telemetry [candidate] (8.785 ms) : 0, 8785
IAST [baseline] (22.287 ms) : 0, 22287
IAST [candidate] (21.45 ms) : 0, 21450
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (835.811 ms) : 0, 835811
BytebuddyAgent [candidate] (832.427 ms) : 0, 832427
GlobalTracer [baseline] (247.492 ms) : 0, 247492
GlobalTracer [candidate] (248.198 ms) : 0, 248198
AppSec [baseline] (58.231 ms) : 0, 58231
AppSec [candidate] (58.102 ms) : 0, 58102
Remote Config [baseline] (672.088 µs) : 0, 672
Remote Config [candidate] (661.807 µs) : 0, 662
Telemetry [baseline] (8.781 ms) : 0, 8781
Telemetry [candidate] (8.84 ms) : 0, 8840
IAST [baseline] (21.715 ms) : 0, 21715
IAST [candidate] (21.658 ms) : 0, 21658
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (839.41 ms) : 0, 839410
BytebuddyAgent [candidate] (839.912 ms) : 0, 839912
GlobalTracer [baseline] (247.709 ms) : 0, 247709
GlobalTracer [candidate] (247.687 ms) : 0, 247687
AppSec [baseline] (58.075 ms) : 0, 58075
AppSec [candidate] (58.214 ms) : 0, 58214
Remote Config [baseline] (684.673 µs) : 0, 685
Remote Config [candidate] (674.441 µs) : 0, 674
Telemetry [baseline] (8.726 ms) : 0, 8726
Telemetry [candidate] (8.732 ms) : 0, 8732
IAST [baseline] (21.296 ms) : 0, 21296
IAST [candidate] (21.167 ms) : 0, 21167
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~2d66a235f6, baseline=1.46.0-SNAPSHOT~0da31f6ae9
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1054885
Total [baseline] (10.409 s) : 0, 10408614
Agent [candidate] (1.055 s) : 0, 1055284
Total [candidate] (10.48 s) : 0, 10479513
section appsec
Agent [baseline] (1.192 s) : 0, 1192018
Total [baseline] (10.692 s) : 0, 10692291
Agent [candidate] (1.19 s) : 0, 1190496
Total [candidate] (10.697 s) : 0, 10696777
section iast
Agent [baseline] (1.19 s) : 0, 1189514
Total [baseline] (10.987 s) : 0, 10986920
Agent [candidate] (1.182 s) : 0, 1181759
Total [candidate] (10.971 s) : 0, 10971176
section profiling
Agent [baseline] (1.252 s) : 0, 1251661
Total [baseline] (10.75 s) : 0, 10749687
Agent [candidate] (1.262 s) : 0, 1261636
Total [candidate] (10.81 s) : 0, 10809716
gantt
title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~2d66a235f6, baseline=1.46.0-SNAPSHOT~0da31f6ae9
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (713.29 ms) : 0, 713290
BytebuddyAgent [candidate] (713.193 ms) : 0, 713193
GlobalTracer [baseline] (255.405 ms) : 0, 255405
GlobalTracer [candidate] (255.409 ms) : 0, 255409
AppSec [baseline] (56.074 ms) : 0, 56074
AppSec [candidate] (55.909 ms) : 0, 55909
Remote Config [baseline] (716.168 µs) : 0, 716
Remote Config [candidate] (720.049 µs) : 0, 720
Telemetry [baseline] (14.477 ms) : 0, 14477
Telemetry [candidate] (14.994 ms) : 0, 14994
section appsec
BytebuddyAgent [baseline] (733.227 ms) : 0, 733227
BytebuddyAgent [candidate] (732.356 ms) : 0, 732356
GlobalTracer [baseline] (253.48 ms) : 0, 253480
GlobalTracer [candidate] (253.349 ms) : 0, 253349
AppSec [baseline] (171.325 ms) : 0, 171325
AppSec [candidate] (171.11 ms) : 0, 171110
Remote Config [baseline] (671.797 µs) : 0, 672
Remote Config [candidate] (667.761 µs) : 0, 668
Telemetry [baseline] (8.193 ms) : 0, 8193
Telemetry [candidate] (8.229 ms) : 0, 8229
IAST [baseline] (20.279 ms) : 0, 20279
IAST [candidate] (19.581 ms) : 0, 19581
section iast
BytebuddyAgent [baseline] (836.952 ms) : 0, 836952
BytebuddyAgent [candidate] (831.457 ms) : 0, 831457
GlobalTracer [baseline] (248.201 ms) : 0, 248201
GlobalTracer [candidate] (246.408 ms) : 0, 246408
AppSec [baseline] (58.286 ms) : 0, 58286
AppSec [candidate] (57.979 ms) : 0, 57979
Remote Config [baseline] (669.402 µs) : 0, 669
Remote Config [candidate] (654.535 µs) : 0, 655
Telemetry [baseline] (8.825 ms) : 0, 8825
Telemetry [candidate] (8.745 ms) : 0, 8745
IAST [baseline] (21.573 ms) : 0, 21573
IAST [candidate] (21.521 ms) : 0, 21521
section profiling
ProfilingAgent [baseline] (95.444 ms) : 0, 95444
ProfilingAgent [candidate] (95.401 ms) : 0, 95401
BytebuddyAgent [baseline] (701.342 ms) : 0, 701342
BytebuddyAgent [candidate] (708.397 ms) : 0, 708397
GlobalTracer [baseline] (349.332 ms) : 0, 349332
GlobalTracer [candidate] (352.218 ms) : 0, 352218
AppSec [baseline] (54.313 ms) : 0, 54313
AppSec [candidate] (53.794 ms) : 0, 53794
Remote Config [baseline] (655.828 µs) : 0, 656
Remote Config [candidate] (661.335 µs) : 0, 661
Telemetry [baseline] (8.8 ms) : 0, 8800
Telemetry [candidate] (8.925 ms) : 0, 8925
Profiling [baseline] (95.467 ms) : 0, 95467
Profiling [candidate] (95.424 ms) : 0, 95424
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~2d66a235f6, baseline=1.46.0-SNAPSHOT~0da31f6ae9
dateFormat X
axisFormat %s
section baseline
no_agent (375.988 µs) : 356, 396
. : milestone, 376,
iast (503.73 µs) : 482, 525
. : milestone, 504,
iast_FULL (745.639 µs) : 724, 768
. : milestone, 746,
iast_GLOBAL (556.919 µs) : 534, 580
. : milestone, 557,
iast_HARDCODED_SECRET_DISABLED (511.798 µs) : 490, 534
. : milestone, 512,
iast_INACTIVE (459.87 µs) : 438, 481
. : milestone, 460,
iast_TELEMETRY_OFF (488.811 µs) : 467, 510
. : milestone, 489,
tracing (448.207 µs) : 427, 469
. : milestone, 448,
section candidate
no_agent (377.927 µs) : 358, 397
. : milestone, 378,
iast (502.362 µs) : 481, 524
. : milestone, 502,
iast_FULL (744.474 µs) : 723, 766
. : milestone, 744,
iast_GLOBAL (548.182 µs) : 527, 570
. : milestone, 548,
iast_HARDCODED_SECRET_DISABLED (509.854 µs) : 488, 532
. : milestone, 510,
iast_INACTIVE (461.041 µs) : 440, 482
. : milestone, 461,
iast_TELEMETRY_OFF (495.993 µs) : 474, 518
. : milestone, 496,
tracing (445.386 µs) : 425, 466
. : milestone, 445,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~2d66a235f6, baseline=1.46.0-SNAPSHOT~0da31f6ae9
dateFormat X
axisFormat %s
section baseline
no_agent (1.364 ms) : 1345, 1383
. : milestone, 1364,
appsec (1.74 ms) : 1716, 1764
. : milestone, 1740,
appsec_no_iast (1.763 ms) : 1740, 1787
. : milestone, 1763,
iast (1.5 ms) : 1476, 1524
. : milestone, 1500,
profiling (1.51 ms) : 1486, 1535
. : milestone, 1510,
tracing (1.48 ms) : 1455, 1506
. : milestone, 1480,
section candidate
no_agent (1.346 ms) : 1327, 1366
. : milestone, 1346,
appsec (1.756 ms) : 1730, 1781
. : milestone, 1756,
appsec_no_iast (1.749 ms) : 1726, 1772
. : milestone, 1749,
iast (1.504 ms) : 1480, 1528
. : milestone, 1504,
profiling (1.522 ms) : 1499, 1546
. : milestone, 1522,
tracing (1.475 ms) : 1450, 1499
. : milestone, 1475,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~2d66a235f6, baseline=1.46.0-SNAPSHOT~0da31f6ae9
dateFormat X
axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
. : milestone, 1474,
appsec (2.37 ms) : 2327, 2413
. : milestone, 2370,
iast (2.12 ms) : 2066, 2175
. : milestone, 2120,
iast_GLOBAL (2.168 ms) : 2112, 2223
. : milestone, 2168,
profiling (1.977 ms) : 1934, 2021
. : milestone, 1977,
tracing (1.959 ms) : 1917, 2000
. : milestone, 1959,
section candidate
no_agent (1.478 ms) : 1466, 1490
. : milestone, 1478,
appsec (2.375 ms) : 2332, 2418
. : milestone, 2375,
iast (2.117 ms) : 2063, 2172
. : milestone, 2117,
iast_GLOBAL (2.16 ms) : 2106, 2215
. : milestone, 2160,
profiling (1.976 ms) : 1932, 2019
. : milestone, 1976,
tracing (1.966 ms) : 1924, 2008
. : milestone, 1966,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~2d66a235f6, baseline=1.46.0-SNAPSHOT~0da31f6ae9
dateFormat X
axisFormat %s
section baseline
no_agent (14.982 s) : 14982000, 14982000
. : milestone, 14982000,
appsec (14.915 s) : 14915000, 14915000
. : milestone, 14915000,
iast (19.077 s) : 19077000, 19077000
. : milestone, 19077000,
iast_GLOBAL (17.686 s) : 17686000, 17686000
. : milestone, 17686000,
profiling (15.203 s) : 15203000, 15203000
. : milestone, 15203000,
tracing (14.983 s) : 14983000, 14983000
. : milestone, 14983000,
section candidate
no_agent (14.784 s) : 14784000, 14784000
. : milestone, 14784000,
appsec (15.226 s) : 15226000, 15226000
. : milestone, 15226000,
iast (18.949 s) : 18949000, 18949000
. : milestone, 18949000,
iast_GLOBAL (17.956 s) : 17956000, 17956000
. : milestone, 17956000,
profiling (14.963 s) : 14963000, 14963000
. : milestone, 14963000,
tracing (14.722 s) : 14722000, 14722000
. : milestone, 14722000,
|
smola
reviewed
Jan 16, 2025
|
|
||
| // print to sourceFiles.xml only if source file has not already been added | ||
| if (!sourceFiles.containsKey(testClassName)) { | ||
| File sourceFile = new File(root + "/build/test-results/sourceFiles.xml"); |
There was a problem hiding this comment.
It seems we're not outputting here XML, but some custom format? We should use something like .dat or .txt here rather than .xml.
smola
reviewed
Jan 16, 2025
| @@ -2,3 +2,20 @@ apply plugin: 'datadog.dependency-locking' | |||
|
|
|||
| apply from: "$rootDir/gradle/java_deps.gradle" | |||
| apply from: "$rootDir/gradle/java_no_deps.gradle" | |||
|
|
|||
| tasks.named("test") { | |||
There was a problem hiding this comment.
There's more test tasks than these two. You may use something like:
tasks.withType(Test).configureEach {
finalizedBy("myCustomTask")
}
to make sure this runs after all of them.
smola
reviewed
Jan 16, 2025
| @@ -0,0 +1,48 @@ | |||
| package datadog.context; | |||
There was a problem hiding this comment.
The extension should probably live at ./utils/test-utils.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Continues experimentation from #8143. Instead of using two custom extensions, it uses one extension to output sourcefile information into
sourceFiles.xml, then uses this information incollect_results.shto reformat the XML files. A solution to keep all logic incollect_results.shwithout the use of any extension is in the works....Motivation
The eventual goal of this PR is to add the source file attribute to the elements in the JUnit XML report that is produced after tests are run. This will eventually allow us to assign codeowner data. However, this PR is experimental.
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: https://datadoghq.atlassian.net/browse/LANGPLAT-147