ITR code coverage support #7367
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikita-tkachenko-datadog
added
the
comp: ci visibility
nikita-tkachenko-datadog
force-pushed
the
nikita-tkachenko/code-coverage-with-itr
branch
from
28a956d
to
812e6a2
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058470
Total [baseline] (10.405 s) : 0, 10404701
Agent [candidate] (1.053 s) : 0, 1053180
Total [candidate] (10.35 s) : 0, 10349854
section appsec
Agent [baseline] (1.188 s) : 0, 1188350
Total [baseline] (10.578 s) : 0, 10577993
Agent [candidate] (1.183 s) : 0, 1182948
Total [candidate] (10.599 s) : 0, 10599115
section iast
Agent [baseline] (1.186 s) : 0, 1185643
Total [baseline] (10.818 s) : 0, 10818357
Agent [candidate] (1.183 s) : 0, 1183263
Total [candidate] (10.816 s) : 0, 10815824
section profiling
Agent [baseline] (1.252 s) : 0, 1252496
Total [baseline] (10.7 s) : 0, 10700059
Agent [candidate] (1.251 s) : 0, 1250650
Total [candidate] (10.667 s) : 0, 10666993
gantt
title petclinic - break down per module: candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (675.53 ms) : 0, 675530
BytebuddyAgent [candidate] (671.873 ms) : 0, 671873
GlobalTracer [baseline] (309.435 ms) : 0, 309435
GlobalTracer [candidate] (308.097 ms) : 0, 308097
AppSec [baseline] (51.601 ms) : 0, 51601
AppSec [candidate] (51.358 ms) : 0, 51358
Remote Config [baseline] (679.661 µs) : 0, 680
Remote Config [candidate] (688.945 µs) : 0, 689
Telemetry [baseline] (7.551 ms) : 0, 7551
Telemetry [candidate] (7.58 ms) : 0, 7580
section appsec
BytebuddyAgent [baseline] (693.267 ms) : 0, 693267
BytebuddyAgent [candidate] (689.262 ms) : 0, 689262
GlobalTracer [baseline] (302.296 ms) : 0, 302296
GlobalTracer [candidate] (300.584 ms) : 0, 300584
AppSec [baseline] (159.4 ms) : 0, 159400
AppSec [candidate] (159.573 ms) : 0, 159573
Remote Config [baseline] (631.38 µs) : 0, 631
Remote Config [candidate] (627.009 µs) : 0, 627
Telemetry [baseline] (8.192 ms) : 0, 8192
Telemetry [candidate] (9.547 ms) : 0, 9547
IAST [baseline] (21.884 ms) : 0, 21884
IAST [candidate] (19.613 ms) : 0, 19613
section iast
BytebuddyAgent [baseline] (788.776 ms) : 0, 788776
BytebuddyAgent [candidate] (786.85 ms) : 0, 786850
GlobalTracer [baseline] (298.397 ms) : 0, 298397
GlobalTracer [candidate] (298.778 ms) : 0, 298778
AppSec [baseline] (54.825 ms) : 0, 54825
AppSec [candidate] (51.594 ms) : 0, 51594
Remote Config [baseline] (596.707 µs) : 0, 597
Remote Config [candidate] (592.817 µs) : 0, 593
Telemetry [baseline] (7.425 ms) : 0, 7425
Telemetry [candidate] (7.304 ms) : 0, 7304
IAST [baseline] (21.916 ms) : 0, 21916
IAST [candidate] (24.443 ms) : 0, 24443
section profiling
BytebuddyAgent [baseline] (666.036 ms) : 0, 666036
BytebuddyAgent [candidate] (666.071 ms) : 0, 666071
GlobalTracer [baseline] (391.52 ms) : 0, 391520
GlobalTracer [candidate] (389.704 ms) : 0, 389704
AppSec [baseline] (52.541 ms) : 0, 52541
AppSec [candidate] (52.431 ms) : 0, 52431
Remote Config [baseline] (707.877 µs) : 0, 708
Remote Config [candidate] (681.191 µs) : 0, 681
Telemetry [baseline] (7.473 ms) : 0, 7473
Telemetry [candidate] (7.364 ms) : 0, 7364
ProfilingAgent [baseline] (96.367 ms) : 0, 96367
ProfilingAgent [candidate] (96.522 ms) : 0, 96522
Profiling [baseline] (96.391 ms) : 0, 96391
Profiling [candidate] (96.546 ms) : 0, 96546
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1057428
Total [baseline] (8.505 s) : 0, 8504932
Agent [candidate] (1.052 s) : 0, 1052101
Total [candidate] (8.502 s) : 0, 8501804
section iast
Agent [baseline] (1.175 s) : 0, 1174804
Total [baseline] (8.961 s) : 0, 8961435
Agent [candidate] (1.178 s) : 0, 1177604
Total [candidate] (8.97 s) : 0, 8969965
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.174 s) : 0, 1173569
Total [baseline] (8.945 s) : 0, 8944599
Agent [candidate] (1.174 s) : 0, 1174165
Total [candidate] (8.953 s) : 0, 8952973
section iast_TELEMETRY_OFF
Agent [baseline] (1.173 s) : 0, 1173083
Total [baseline] (8.953 s) : 0, 8952532
Agent [candidate] (1.173 s) : 0, 1172822
Total [candidate] (9.004 s) : 0, 9003701
gantt
title insecure-bank - break down per module: candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (674.995 ms) : 0, 674995
BytebuddyAgent [candidate] (670.941 ms) : 0, 670941
GlobalTracer [baseline] (309.231 ms) : 0, 309231
GlobalTracer [candidate] (308.235 ms) : 0, 308235
AppSec [baseline] (51.401 ms) : 0, 51401
AppSec [candidate] (51.277 ms) : 0, 51277
Remote Config [baseline] (680.799 µs) : 0, 681
Remote Config [candidate] (690.27 µs) : 0, 690
Telemetry [baseline] (7.446 ms) : 0, 7446
Telemetry [candidate] (7.375 ms) : 0, 7375
section iast
BytebuddyAgent [baseline] (781.848 ms) : 0, 781848
BytebuddyAgent [candidate] (782.924 ms) : 0, 782924
GlobalTracer [baseline] (296.27 ms) : 0, 296270
GlobalTracer [candidate] (296.972 ms) : 0, 296972
AppSec [baseline] (54.207 ms) : 0, 54207
AppSec [candidate] (52.618 ms) : 0, 52618
Remote Config [baseline] (600.702 µs) : 0, 601
Remote Config [candidate] (600.476 µs) : 0, 600
Telemetry [baseline] (7.395 ms) : 0, 7395
Telemetry [candidate] (7.285 ms) : 0, 7285
IAST [baseline] (20.86 ms) : 0, 20860
IAST [candidate] (23.543 ms) : 0, 23543
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (779.235 ms) : 0, 779235
BytebuddyAgent [candidate] (780.088 ms) : 0, 780088
GlobalTracer [baseline] (296.361 ms) : 0, 296361
GlobalTracer [candidate] (296.021 ms) : 0, 296021
AppSec [baseline] (50.613 ms) : 0, 50613
AppSec [candidate] (53.009 ms) : 0, 53009
Remote Config [baseline] (597.541 µs) : 0, 598
Remote Config [candidate] (593.603 µs) : 0, 594
Telemetry [baseline] (7.264 ms) : 0, 7264
Telemetry [candidate] (8.108 ms) : 0, 8108
IAST [baseline] (25.915 ms) : 0, 25915
IAST [candidate] (22.74 ms) : 0, 22740
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (778.981 ms) : 0, 778981
BytebuddyAgent [candidate] (778.98 ms) : 0, 778980
GlobalTracer [baseline] (296.272 ms) : 0, 296272
GlobalTracer [candidate] (296.397 ms) : 0, 296397
AppSec [baseline] (54.194 ms) : 0, 54194
AppSec [candidate] (53.756 ms) : 0, 53756
Remote Config [baseline] (591.14 µs) : 0, 591
Remote Config [candidate] (599.96 µs) : 0, 600
Telemetry [baseline] (7.244 ms) : 0, 7244
Telemetry [candidate] (8.017 ms) : 0, 8017
IAST [baseline] (22.178 ms) : 0, 22178
IAST [candidate] (21.44 ms) : 0, 21440
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 19 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
dateFormat X
axisFormat %s
section baseline
no_agent (1.332 ms) : 1314, 1351
. : milestone, 1332,
appsec (1.738 ms) : 1715, 1762
. : milestone, 1738,
appsec_no_iast (1.741 ms) : 1717, 1766
. : milestone, 1741,
iast (1.472 ms) : 1450, 1494
. : milestone, 1472,
profiling (1.468 ms) : 1444, 1491
. : milestone, 1468,
tracing (1.479 ms) : 1455, 1502
. : milestone, 1479,
section candidate
no_agent (1.348 ms) : 1329, 1366
. : milestone, 1348,
appsec (1.732 ms) : 1708, 1756
. : milestone, 1732,
appsec_no_iast (1.73 ms) : 1705, 1755
. : milestone, 1730,
iast (1.49 ms) : 1466, 1513
. : milestone, 1490,
profiling (1.468 ms) : 1445, 1491
. : milestone, 1468,
tracing (1.466 ms) : 1442, 1491
. : milestone, 1466,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
dateFormat X
axisFormat %s
section baseline
no_agent (368.205 µs) : 349, 388
. : milestone, 368,
iast (474.487 µs) : 453, 496
. : milestone, 474,
iast_FULL (541.34 µs) : 520, 563
. : milestone, 541,
iast_GLOBAL (495.847 µs) : 474, 517
. : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (476.845 µs) : 454, 500
. : milestone, 477,
iast_INACTIVE (436.534 µs) : 416, 457
. : milestone, 437,
iast_TELEMETRY_OFF (473.319 µs) : 451, 496
. : milestone, 473,
tracing (432.274 µs) : 412, 452
. : milestone, 432,
section candidate
no_agent (363.587 µs) : 344, 383
. : milestone, 364,
iast (477.028 µs) : 455, 499
. : milestone, 477,
iast_FULL (548.945 µs) : 528, 570
. : milestone, 549,
iast_GLOBAL (506.534 µs) : 485, 528
. : milestone, 507,
iast_HARDCODED_SECRET_DISABLED (478.238 µs) : 456, 500
. : milestone, 478,
iast_INACTIVE (439.448 µs) : 418, 461
. : milestone, 439,
iast_TELEMETRY_OFF (466.839 µs) : 444, 490
. : milestone, 467,
tracing (431.968 µs) : 412, 452
. : milestone, 432,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
dateFormat X
axisFormat %s
section baseline
no_agent (1.453 ms) : 1442, 1464
. : milestone, 1453,
appsec (2.217 ms) : 2182, 2252
. : milestone, 2217,
iast (1.978 ms) : 1935, 2021
. : milestone, 1978,
iast_GLOBAL (2.036 ms) : 1991, 2081
. : milestone, 2036,
profiling (1.857 ms) : 1823, 1892
. : milestone, 1857,
tracing (1.852 ms) : 1819, 1886
. : milestone, 1852,
section candidate
no_agent (1.455 ms) : 1444, 1466
. : milestone, 1455,
appsec (2.248 ms) : 2212, 2283
. : milestone, 2248,
iast (1.98 ms) : 1937, 2023
. : milestone, 1980,
iast_GLOBAL (2.034 ms) : 1990, 2079
. : milestone, 2034,
profiling (1.863 ms) : 1829, 1897
. : milestone, 1863,
tracing (1.836 ms) : 1803, 1869
. : milestone, 1836,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
dateFormat X
axisFormat %s
section baseline
no_agent (15.602 s) : 15602000, 15602000
. : milestone, 15602000,
appsec (15.182 s) : 15182000, 15182000
. : milestone, 15182000,
iast (18.811 s) : 18811000, 18811000
. : milestone, 18811000,
iast_GLOBAL (17.809 s) : 17809000, 17809000
. : milestone, 17809000,
profiling (15.924 s) : 15924000, 15924000
. : milestone, 15924000,
tracing (14.855 s) : 14855000, 14855000
. : milestone, 14855000,
section candidate
no_agent (15.38 s) : 15380000, 15380000
. : milestone, 15380000,
appsec (15.16 s) : 15160000, 15160000
. : milestone, 15160000,
iast (18.957 s) : 18957000, 18957000
. : milestone, 18957000,
iast_GLOBAL (18.121 s) : 18121000, 18121000
. : milestone, 18121000,
profiling (15.776 s) : 15776000, 15776000
. : milestone, 15776000,
tracing (14.829 s) : 14829000, 14829000
. : milestone, 14829000,
|
nikita-tkachenko-datadog
force-pushed
the
nikita-tkachenko/code-coverage-with-itr
branch
from
7852e93
to
4a40e16
Compare
|
Hi! 👋 Looks like you updated a Git Submodule.
|
1 similar comment
|
Hi! 👋 Looks like you updated a Git Submodule.
|
nikita-tkachenko-datadog
force-pushed
the
nikita-tkachenko/code-coverage-with-itr
branch
4 times, most recently
from
fdfc064
to
4530a48
Compare
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
|
|
||
| @Nonnull | ||
| public byte[] getCoverageData() { | ||
| return coverageData; |
There was a problem hiding this comment.
🟠 Code Quality Violation
Do not return an array that is a class member (...read more)
Avoid returning an array that was defined as a class member. If you want to return an array or the values of the array that is a class member, consider returning a copy of the array.
|
|
||
| @Override | ||
| public boolean equals(Object o) { | ||
| if (this == o) return true; |
There was a problem hiding this comment.
⚪ Code Quality Violation
Suggested change
| if (this == o) return true; | |
| if (this == o) {return true}; |
single if statement should be wrapped in a brace (...read more)
Omitting braces {} is valid in multiple statements, such as, for loops, if statements, and while loops. However, enforcing the use of control braces throughout your codebase will make the code more consistent and can make it easier to add statements in the future.
| @Override | ||
| public boolean equals(Object o) { | ||
| if (this == o) return true; | ||
| if (o == null || getClass() != o.getClass()) return false; |
There was a problem hiding this comment.
⚪ Code Quality Violation
Suggested change
| if (o == null || getClass() != o.getClass()) return false; | |
| if (o == null || getClass() != o.getClass()) {return false}; |
single if statement should be wrapped in a brace (...read more)
Omitting braces {} is valid in multiple statements, such as, for loops, if statements, and while loops. However, enforcing the use of control braces throughout your codebase will make the code more consistent and can make it easier to add statements in the future.
nikita-tkachenko-datadog
changed the title
nikita-tkachenko-datadog
force-pushed
the
nikita-tkachenko/code-coverage-with-itr
branch
from
db565fd
to
7fa7ab4
Compare
nikita-tkachenko-datadog
force-pushed
the
nikita-tkachenko/code-coverage-with-itr
branch
from
7fa7ab4
to
56bc06d
Compare
nikita-tkachenko-datadog
force-pushed
the
nikita-tkachenko/code-coverage-with-itr
branch
from
56bc06d
to
c0c0f53
Compare
anmarchenko
approved these changes
Sep 10, 2024
nikita-tkachenko-datadog
deleted the
nikita-tkachenko/code-coverage-with-itr
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Updates the logic that calculates code coverage percentage (
test.code_coverage.lines_pcttag in session and module events) to take into account tests skipped by ITR.Motivation
Showing "correct" coverage percentage for sessions where ITR tests skipping is enabled.
Here "correct" means "as if no tests were skipped".
Additional Notes
When requesting the list of skippable tests from the backend, the tracer also receives coverage data for these tests: a bitmap of covered lines for every file covered.
The tracer uses this data to "backfill" the coverage obtained at runtime.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: SDTEST-112