Common temporary location manager for profiling product #7971
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
...profiling-controller/src/main/java/com/datadog/profiling/controller/TempLocationManager.java
Outdated
Show resolved
Hide resolved
…va/com/datadog/profiling/controller/TempLocationManager.java Co-authored-by: datadog-datadog-prod-us1[bot] <88084959+datadog-datadog-prod-us1[bot]@users.noreply.github.com>
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 1 performance regressions! Performance is the same for 51 metrics, 11 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.43.0-SNAPSHOT~84d0501188, baseline=1.43.0-SNAPSHOT~9078a8c473
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.105 s) : 0, 1104760
Total [baseline] (8.733 s) : 0, 8732818
Agent [candidate] (1.087 s) : 0, 1087142
Total [candidate] (8.631 s) : 0, 8630682
section iast
Agent [baseline] (1.216 s) : 0, 1215509
Total [baseline] (9.176 s) : 0, 9175594
Agent [candidate] (1.225 s) : 0, 1225076
Total [candidate] (9.21 s) : 0, 9210023
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.222 s) : 0, 1222236
Total [baseline] (9.18 s) : 0, 9180026
Agent [candidate] (1.217 s) : 0, 1216694
Total [candidate] (9.18 s) : 0, 9179576
section iast_TELEMETRY_OFF
Agent [baseline] (1.215 s) : 0, 1214865
Total [baseline] (9.225 s) : 0, 9225033
Agent [candidate] (1.216 s) : 0, 1215885
Total [candidate] (9.217 s) : 0, 9217177
gantt
title insecure-bank - break down per module: candidate=1.43.0-SNAPSHOT~84d0501188, baseline=1.43.0-SNAPSHOT~9078a8c473
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (703.533 ms) : 0, 703533
BytebuddyAgent [candidate] (692.189 ms) : 0, 692189
GlobalTracer [baseline] (320.838 ms) : 0, 320838
GlobalTracer [candidate] (317.023 ms) : 0, 317023
AppSec [baseline] (55.6 ms) : 0, 55600
AppSec [candidate] (54.882 ms) : 0, 54882
Remote Config [baseline] (707.454 µs) : 0, 707
Remote Config [candidate] (702.819 µs) : 0, 703
Telemetry [baseline] (10.116 ms) : 0, 10116
Telemetry [candidate] (8.618 ms) : 0, 8618
section iast
BytebuddyAgent [baseline] (808.854 ms) : 0, 808854
BytebuddyAgent [candidate] (815.32 ms) : 0, 815320
GlobalTracer [baseline] (306.224 ms) : 0, 306224
GlobalTracer [candidate] (308.443 ms) : 0, 308443
AppSec [baseline] (56.36 ms) : 0, 56360
AppSec [candidate] (57.634 ms) : 0, 57634
IAST [baseline] (22.373 ms) : 0, 22373
IAST [candidate] (21.678 ms) : 0, 21678
Remote Config [baseline] (599.803 µs) : 0, 600
Remote Config [candidate] (616.376 µs) : 0, 616
Telemetry [baseline] (7.383 ms) : 0, 7383
Telemetry [candidate] (7.548 ms) : 0, 7548
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (813.644 ms) : 0, 813644
BytebuddyAgent [candidate] (809.417 ms) : 0, 809417
GlobalTracer [baseline] (307.797 ms) : 0, 307797
GlobalTracer [candidate] (306.504 ms) : 0, 306504
AppSec [baseline] (57.27 ms) : 0, 57270
AppSec [candidate] (58.04 ms) : 0, 58040
IAST [baseline] (21.605 ms) : 0, 21605
IAST [candidate] (20.841 ms) : 0, 20841
Remote Config [baseline] (650.171 µs) : 0, 650
Remote Config [candidate] (642.068 µs) : 0, 642
Telemetry [baseline] (7.475 ms) : 0, 7475
Telemetry [candidate] (7.476 ms) : 0, 7476
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (807.354 ms) : 0, 807354
BytebuddyAgent [candidate] (808.461 ms) : 0, 808461
GlobalTracer [baseline] (306.8 ms) : 0, 306800
GlobalTracer [candidate] (306.843 ms) : 0, 306843
AppSec [baseline] (57.674 ms) : 0, 57674
AppSec [candidate] (57.167 ms) : 0, 57167
IAST [baseline] (21.308 ms) : 0, 21308
IAST [candidate] (21.585 ms) : 0, 21585
Remote Config [baseline] (623.824 µs) : 0, 624
Remote Config [candidate] (622.23 µs) : 0, 622
Telemetry [baseline] (7.377 ms) : 0, 7377
Telemetry [candidate] (7.429 ms) : 0, 7429
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.43.0-SNAPSHOT~84d0501188, baseline=1.43.0-SNAPSHOT~9078a8c473
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.101 s) : 0, 1100600
Total [baseline] (10.457 s) : 0, 10456815
Agent [candidate] (1.086 s) : 0, 1085907
Total [candidate] (10.474 s) : 0, 10473736
section appsec
Agent [baseline] (1.225 s) : 0, 1224839
Total [baseline] (10.739 s) : 0, 10739066
Agent [candidate] (1.223 s) : 0, 1222773
Total [candidate] (10.694 s) : 0, 10694446
section iast
Agent [baseline] (1.223 s) : 0, 1223089
Total [baseline] (10.887 s) : 0, 10886829
Agent [candidate] (1.225 s) : 0, 1224714
Total [candidate] (10.956 s) : 0, 10955675
section profiling
Agent [baseline] (1.303 s) : 0, 1303358
Total [baseline] (10.874 s) : 0, 10873527
Agent [candidate] (1.322 s) : 0, 1322334
Total [candidate] (10.923 s) : 0, 10923138
gantt
title petclinic - break down per module: candidate=1.43.0-SNAPSHOT~84d0501188, baseline=1.43.0-SNAPSHOT~9078a8c473
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (700.11 ms) : 0, 700110
BytebuddyAgent [candidate] (691.72 ms) : 0, 691720
GlobalTracer [baseline] (320.656 ms) : 0, 320656
GlobalTracer [candidate] (316.806 ms) : 0, 316806
AppSec [baseline] (55.188 ms) : 0, 55188
AppSec [candidate] (54.436 ms) : 0, 54436
Remote Config [baseline] (684.951 µs) : 0, 685
Remote Config [candidate] (684.543 µs) : 0, 685
Telemetry [baseline] (10.094 ms) : 0, 10094
Telemetry [candidate] (8.584 ms) : 0, 8584
section appsec
BytebuddyAgent [baseline] (711.319 ms) : 0, 711319
BytebuddyAgent [candidate] (708.702 ms) : 0, 708702
GlobalTracer [baseline] (314.952 ms) : 0, 314952
GlobalTracer [candidate] (313.906 ms) : 0, 313906
AppSec [baseline] (165.816 ms) : 0, 165816
AppSec [candidate] (166.978 ms) : 0, 166978
Remote Config [baseline] (660.78 µs) : 0, 661
Remote Config [candidate] (647.188 µs) : 0, 647
Telemetry [baseline] (8.581 ms) : 0, 8581
Telemetry [candidate] (8.203 ms) : 0, 8203
IAST [baseline] (19.62 ms) : 0, 19620
IAST [candidate] (21.295 ms) : 0, 21295
section iast
BytebuddyAgent [baseline] (814.945 ms) : 0, 814945
BytebuddyAgent [candidate] (815.353 ms) : 0, 815353
GlobalTracer [baseline] (308.034 ms) : 0, 308034
GlobalTracer [candidate] (308.152 ms) : 0, 308152
AppSec [baseline] (57.508 ms) : 0, 57508
AppSec [candidate] (57.478 ms) : 0, 57478
Remote Config [baseline] (604.538 µs) : 0, 605
Remote Config [candidate] (1.442 ms) : 0, 1442
Telemetry [baseline] (7.468 ms) : 0, 7468
Telemetry [candidate] (7.488 ms) : 0, 7488
IAST [baseline] (20.793 ms) : 0, 20793
IAST [candidate] (20.959 ms) : 0, 20959
section profiling
ProfilingAgent [baseline] (95.819 ms) : 0, 95819
ProfilingAgent [candidate] (93.608 ms) : 0, 93608
BytebuddyAgent [baseline] (695.461 ms) : 0, 695461
BytebuddyAgent [candidate] (689.459 ms) : 0, 689459
GlobalTracer [baseline] (404.541 ms) : 0, 404541
GlobalTracer [candidate] (437.657 ms) : 0, 437657
AppSec [baseline] (55.732 ms) : 0, 55732
AppSec [candidate] (54.084 ms) : 0, 54084
Remote Config [baseline] (690.089 µs) : 0, 690
Remote Config [candidate] (675.072 µs) : 0, 675
Telemetry [baseline] (11.558 ms) : 0, 11558
Telemetry [candidate] (7.757 ms) : 0, 7757
Profiling [baseline] (95.842 ms) : 0, 95842
Profiling [candidate] (93.633 ms) : 0, 93633
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~84d0501188, baseline=1.43.0-SNAPSHOT~9078a8c473
dateFormat X
axisFormat %s
section baseline
no_agent (372.148 µs) : 352, 392
. : milestone, 372,
iast (484.868 µs) : 464, 506
. : milestone, 485,
iast_FULL (646.629 µs) : 625, 668
. : milestone, 647,
iast_GLOBAL (512.68 µs) : 491, 534
. : milestone, 513,
iast_HARDCODED_SECRET_DISABLED (487.053 µs) : 466, 508
. : milestone, 487,
iast_INACTIVE (451.981 µs) : 430, 473
. : milestone, 452,
iast_TELEMETRY_OFF (475.308 µs) : 454, 497
. : milestone, 475,
tracing (445.686 µs) : 425, 467
. : milestone, 446,
section candidate
no_agent (371.967 µs) : 351, 393
. : milestone, 372,
iast (488.767 µs) : 467, 510
. : milestone, 489,
iast_FULL (652.432 µs) : 631, 674
. : milestone, 652,
iast_GLOBAL (521.119 µs) : 498, 544
. : milestone, 521,
iast_HARDCODED_SECRET_DISABLED (487.801 µs) : 466, 509
. : milestone, 488,
iast_INACTIVE (454.845 µs) : 433, 477
. : milestone, 455,
iast_TELEMETRY_OFF (482.372 µs) : 460, 505
. : milestone, 482,
tracing (449.525 µs) : 428, 471
. : milestone, 450,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~84d0501188, baseline=1.43.0-SNAPSHOT~9078a8c473
dateFormat X
axisFormat %s
section baseline
no_agent (1.353 ms) : 1333, 1373
. : milestone, 1353,
appsec (1.743 ms) : 1718, 1767
. : milestone, 1743,
appsec_no_iast (1.774 ms) : 1750, 1799
. : milestone, 1774,
iast (1.505 ms) : 1481, 1528
. : milestone, 1505,
profiling (1.496 ms) : 1473, 1519
. : milestone, 1496,
tracing (1.474 ms) : 1449, 1499
. : milestone, 1474,
section candidate
no_agent (1.36 ms) : 1340, 1380
. : milestone, 1360,
appsec (1.762 ms) : 1738, 1786
. : milestone, 1762,
appsec_no_iast (1.755 ms) : 1731, 1779
. : milestone, 1755,
iast (1.492 ms) : 1470, 1515
. : milestone, 1492,
profiling (1.55 ms) : 1526, 1574
. : milestone, 1550,
tracing (1.481 ms) : 1457, 1505
. : milestone, 1481,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~84d0501188, baseline=1.43.0-SNAPSHOT~9078a8c473
dateFormat X
axisFormat %s
section baseline
no_agent (1.467 ms) : 1455, 1478
. : milestone, 1467,
appsec (2.342 ms) : 2300, 2383
. : milestone, 2342,
iast (2.081 ms) : 2029, 2133
. : milestone, 2081,
iast_GLOBAL (2.124 ms) : 2072, 2177
. : milestone, 2124,
profiling (1.966 ms) : 1923, 2009
. : milestone, 1966,
tracing (1.922 ms) : 1882, 1962
. : milestone, 1922,
section candidate
no_agent (1.465 ms) : 1454, 1477
. : milestone, 1465,
appsec (2.328 ms) : 2287, 2370
. : milestone, 2328,
iast (2.08 ms) : 2027, 2132
. : milestone, 2080,
iast_GLOBAL (2.116 ms) : 2064, 2168
. : milestone, 2116,
profiling (1.949 ms) : 1907, 1991
. : milestone, 1949,
tracing (1.914 ms) : 1874, 1954
. : milestone, 1914,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~84d0501188, baseline=1.43.0-SNAPSHOT~9078a8c473
dateFormat X
axisFormat %s
section baseline
no_agent (15.204 s) : 15204000, 15204000
. : milestone, 15204000,
appsec (15.169 s) : 15169000, 15169000
. : milestone, 15169000,
iast (18.917 s) : 18917000, 18917000
. : milestone, 18917000,
iast_GLOBAL (17.903 s) : 17903000, 17903000
. : milestone, 17903000,
profiling (14.954 s) : 14954000, 14954000
. : milestone, 14954000,
tracing (14.941 s) : 14941000, 14941000
. : milestone, 14941000,
section candidate
no_agent (15.245 s) : 15245000, 15245000
. : milestone, 15245000,
appsec (14.946 s) : 14946000, 14946000
. : milestone, 14946000,
iast (18.934 s) : 18934000, 18934000
. : milestone, 18934000,
iast_GLOBAL (17.976 s) : 17976000, 17976000
. : milestone, 17976000,
profiling (14.754 s) : 14754000, 14754000
. : milestone, 14754000,
tracing (15.167 s) : 15167000, 15167000
. : milestone, 15167000,
|
jbachorik
force-pushed
the
jb/profiling_temp_location
branch
from
6dabf93
to
0fa642e
Compare
...profiling-controller/src/main/java/com/datadog/profiling/controller/TempLocationManager.java
Outdated
Show resolved
Hide resolved
...iling-controller/src/test/java/com/datadog/profiling/controller/TempLocationManagerTest.java
Show resolved
Hide resolved
...profiling-controller/src/main/java/com/datadog/profiling/controller/TempLocationManager.java
Show resolved
Hide resolved
manuel-alvarez-alvarez
approved these changes
Nov 22, 2024
There was a problem hiding this comment.
LGTM (@richardstartin concerns at shutdown should be addressed)
...profiling-controller/src/main/java/com/datadog/profiling/controller/TempLocationManager.java
Outdated
Show resolved
Hide resolved
…va/com/datadog/profiling/controller/TempLocationManager.java Co-authored-by: datadog-datadog-prod-us1[bot] <88084959+datadog-datadog-prod-us1[bot]@users.noreply.github.com>
richardstartin
approved these changes
Nov 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
It extracts the functionality related to temporary location cleanup (previously only for JFR) to allow for shared experience for JFR recording, ddprof recording and temporary library artifacts.
Motivation
Unification of the temporary location handling. Providing cleanup of all orphaned temporary artifacts generated by the profiler.
Additional Notes
How does this work
The temp location manager will create one common root for the temporary data and that location will have the name of
pid_<pid>where<pid>is the Java process PID. Then, the users can request temporary subfolders with a specific path and these will be created under the common, process-specific, location. All the temporary subfolders are automatically registered for cleanup on JVM exit.In order to handle hard-kills and JVM crashes, each time a profiled process is started it will scan the base temporary location for any
pid_<pid>subfolders that don't correspond to any of the currently running JVM. There is an additional requirement for a file/directory to be cleaned up, though. It must have been last modified at least 2* seconds ago. This helps to exclude JVMs being currently in shutdown and therefore not showing up injpsoutput or having been created between the cleaner obtained thejpsoutput and started cleaning the files.I had to modify
PidHelperto drain the pipes asynchronously - apparently, in smoke tests and system tests the number of java processes running on a single host was large enough to cause clogging and effectively deadlocking the test process with the 'jps' subprocess.Compatibility issues
profiling.jfr.repository.cleanupis removed and ignored; cleanup will happen alwaysprofiling.jfr.repository.baseis deprecated and ignored; useprofiling.tempdirinsteadContributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: PROF-10925