Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adjust crash upload timeout #7905

Merged
merged 3 commits into from
Nov 7, 2024
Merged

Adjust crash upload timeout #7905

merged 3 commits into from
Nov 7, 2024

Conversation

dougqh
Copy link
Contributor

@dougqh dougqh commented Nov 7, 2024 •
edited
Loading

What Does This Do

Lowers the crash uploader default timeout

Motivation

We don't want the crash tracker to block process shutdown for an extended period of time

Additional Notes

While trying to make this change, I discovered that crash tracking isn't working on OS X because of the ancient version of bash that ships with OS X by default.

I've made the assertions a bit easier to debug and added some comments, so others won't spend time on this in the future.
In a later PR, it might be good to come back and make crash reporting work on OS X just to ease development.

@dougqh
Adding hamcrest matchers to the crash tracking smoke test
0132a90 
I spent a day or so digging and understanding why crash tracking smoke test wasn't working locally for me.

Eventually, I tracked it down to the ancient version of bash being used on OS X.  That version of bash doesn't support all the necessary elements used by the bash scripts.

To debug the issue, I switched to using hamcrest matchers, so I could more easily see what was happening underneath.

I've also added comments to the test class, so hopefully, others don't lose time on this again in the future.
@dougqh
Adjusting crash uploader default timeout
1475406
@dougqh dougqh requested a review from a team as a code owner November 7, 2024 15:34
@dougqh dougqh requested a review from smola November 7, 2024 15:34
@pr-commenter
Copy link

pr-commenter bot commented Nov 7, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/crashtracking-adjust-timeout
git_commit_date 1730981645 1730993794
git_commit_sha 060ab72 f330e67
release_version 1.43.0-SNAPSHOT~060ab7250f 1.43.0-SNAPSHOT~f330e675cd
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1730996306 1730996306
ci_job_id 699144908 699144908
ci_pipeline_id 48464158 48464158
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 53 metrics, 10 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.43.0-SNAPSHOT~f330e675cd, baseline=1.43.0-SNAPSHOT~060ab7250f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.09 s) : 0, 1089769
Total [baseline] (8.611 s) : 0, 8611071
Agent [candidate] (1.087 s) : 0, 1087408
Total [candidate] (8.59 s) : 0, 8590195
section iast
Agent [baseline] (1.214 s) : 0, 1213604
Total [baseline] (9.191 s) : 0, 9191372
Agent [candidate] (1.217 s) : 0, 1216666
Total [candidate] (9.145 s) : 0, 9145169
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.211 s) : 0, 1211267
Total [baseline] (9.091 s) : 0, 9090548
Agent [candidate] (1.22 s) : 0, 1219658
Total [candidate] (9.185 s) : 0, 9185471
section iast_TELEMETRY_OFF
Agent [baseline] (1.208 s) : 0, 1207680
Total [baseline] (9.176 s) : 0, 9175744
Agent [candidate] (1.216 s) : 0, 1216460
Total [candidate] (9.194 s) : 0, 9194330
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.09 s -
Agent iast 1.214 s 123.835 ms (11.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.211 s 121.498 ms (11.1%)
Agent iast_TELEMETRY_OFF 1.208 s 117.911 ms (10.8%)
Total tracing 8.611 s -
Total iast 9.191 s 580.301 ms (6.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.091 s 479.477 ms (5.6%)
Total iast_TELEMETRY_OFF 9.176 s 564.673 ms (6.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.087 s -
Agent iast 1.217 s 129.258 ms (11.9%)
Agent iast_HARDCODED_SECRET_DISABLED 1.22 s 132.25 ms (12.2%)
Agent iast_TELEMETRY_OFF 1.216 s 129.052 ms (11.9%)
Total tracing 8.59 s -
Total iast 9.145 s 554.974 ms (6.5%)
Total iast_HARDCODED_SECRET_DISABLED 9.185 s 595.276 ms (6.9%)
Total iast_TELEMETRY_OFF 9.194 s 604.135 ms (7.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.43.0-SNAPSHOT~f330e675cd, baseline=1.43.0-SNAPSHOT~060ab7250f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (691.992 ms) : 0, 691992
BytebuddyAgent [candidate] (691.265 ms) : 0, 691265
GlobalTracer [baseline] (318.822 ms) : 0, 318822
GlobalTracer [candidate] (317.588 ms) : 0, 317588
AppSec [baseline] (54.477 ms) : 0, 54477
AppSec [candidate] (54.274 ms) : 0, 54274
Remote Config [baseline] (671.398 µs) : 0, 671
Remote Config [candidate] (670.516 µs) : 0, 671
Telemetry [baseline] (9.932 ms) : 0, 9932
Telemetry [candidate] (9.797 ms) : 0, 9797
section iast
BytebuddyAgent [baseline] (806.168 ms) : 0, 806168
BytebuddyAgent [candidate] (808.753 ms) : 0, 808753
GlobalTracer [baseline] (306.679 ms) : 0, 306679
GlobalTracer [candidate] (306.729 ms) : 0, 306729
AppSec [baseline] (57.995 ms) : 0, 57995
AppSec [candidate] (57.99 ms) : 0, 57990
IAST [baseline] (20.764 ms) : 0, 20764
IAST [candidate] (20.989 ms) : 0, 20989
Remote Config [baseline] (620.983 µs) : 0, 621
Remote Config [candidate] (633.867 µs) : 0, 634
Telemetry [baseline] (7.582 ms) : 0, 7582
Telemetry [candidate] (7.708 ms) : 0, 7708
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (805.267 ms) : 0, 805267
BytebuddyAgent [candidate] (810.332 ms) : 0, 810332
GlobalTracer [baseline] (305.517 ms) : 0, 305517
GlobalTracer [candidate] (307.474 ms) : 0, 307474
AppSec [baseline] (57.107 ms) : 0, 57107
AppSec [candidate] (57.814 ms) : 0, 57814
IAST [baseline] (21.425 ms) : 0, 21425
IAST [candidate] (21.858 ms) : 0, 21858
Remote Config [baseline] (624.001 µs) : 0, 624
Remote Config [candidate] (632.298 µs) : 0, 632
Telemetry [baseline] (7.518 ms) : 0, 7518
Telemetry [candidate] (7.638 ms) : 0, 7638
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (801.755 ms) : 0, 801755
BytebuddyAgent [candidate] (807.54 ms) : 0, 807540
GlobalTracer [baseline] (305.207 ms) : 0, 305207
GlobalTracer [candidate] (307.506 ms) : 0, 307506
AppSec [baseline] (57.492 ms) : 0, 57492
AppSec [candidate] (57.824 ms) : 0, 57824
IAST [baseline] (21.288 ms) : 0, 21288
IAST [candidate] (20.729 ms) : 0, 20729
Remote Config [baseline] (634.282 µs) : 0, 634
Remote Config [candidate] (624.706 µs) : 0, 625
Telemetry [baseline] (7.511 ms) : 0, 7511
Telemetry [candidate] (8.353 ms) : 0, 8353
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.43.0-SNAPSHOT~f330e675cd, baseline=1.43.0-SNAPSHOT~060ab7250f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.09 s) : 0, 1090144
Total [baseline] (10.406 s) : 0, 10406032
Agent [candidate] (1.087 s) : 0, 1087427
Total [candidate] (10.416 s) : 0, 10415643
section appsec
Agent [baseline] (1.216 s) : 0, 1216383
Total [baseline] (10.69 s) : 0, 10689848
Agent [candidate] (1.221 s) : 0, 1221452
Total [candidate] (10.725 s) : 0, 10725052
section iast
Agent [baseline] (1.211 s) : 0, 1210815
Total [baseline] (10.934 s) : 0, 10934002
Agent [candidate] (1.221 s) : 0, 1220521
Total [candidate] (10.913 s) : 0, 10913372
section profiling
Agent [baseline] (1.281 s) : 0, 1281428
Total [baseline] (10.731 s) : 0, 10731245
Agent [candidate] (1.282 s) : 0, 1281567
Total [candidate] (10.789 s) : 0, 10789281
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.09 s -
Agent appsec 1.216 s 126.239 ms (11.6%)
Agent iast 1.211 s 120.67 ms (11.1%)
Agent profiling 1.281 s 191.283 ms (17.5%)
Total tracing 10.406 s -
Total appsec 10.69 s 283.816 ms (2.7%)
Total iast 10.934 s 527.97 ms (5.1%)
Total profiling 10.731 s 325.213 ms (3.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.087 s -
Agent appsec 1.221 s 134.025 ms (12.3%)
Agent iast 1.221 s 133.094 ms (12.2%)
Agent profiling 1.282 s 194.14 ms (17.9%)
Total tracing 10.416 s -
Total appsec 10.725 s 309.409 ms (3.0%)
Total iast 10.913 s 497.728 ms (4.8%)
Total profiling 10.789 s 373.638 ms (3.6%) 
gantt
    title petclinic - break down per module: candidate=1.43.0-SNAPSHOT~f330e675cd, baseline=1.43.0-SNAPSHOT~060ab7250f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (693.195 ms) : 0, 693195
BytebuddyAgent [candidate] (689.948 ms) : 0, 689948
GlobalTracer [baseline] (318.401 ms) : 0, 318401
GlobalTracer [candidate] (318.399 ms) : 0, 318399
AppSec [baseline] (54.76 ms) : 0, 54760
AppSec [candidate] (54.729 ms) : 0, 54729
Remote Config [baseline] (678.909 µs) : 0, 679
Remote Config [candidate] (686.366 µs) : 0, 686
Telemetry [baseline] (9.276 ms) : 0, 9276
Telemetry [candidate] (9.927 ms) : 0, 9927
section appsec
BytebuddyAgent [baseline] (704.379 ms) : 0, 704379
BytebuddyAgent [candidate] (708.406 ms) : 0, 708406
GlobalTracer [baseline] (313.66 ms) : 0, 313660
GlobalTracer [candidate] (314.907 ms) : 0, 314907
AppSec [baseline] (167.043 ms) : 0, 167043
AppSec [candidate] (165.88 ms) : 0, 165880
IAST [baseline] (18.452 ms) : 0, 18452
IAST [candidate] (19.365 ms) : 0, 19365
Remote Config [baseline] (637.193 µs) : 0, 637
Remote Config [candidate] (646.087 µs) : 0, 646
Telemetry [baseline] (7.745 ms) : 0, 7745
Telemetry [candidate] (8.163 ms) : 0, 8163
section iast
BytebuddyAgent [baseline] (804.57 ms) : 0, 804570
BytebuddyAgent [candidate] (811.823 ms) : 0, 811823
GlobalTracer [baseline] (305.608 ms) : 0, 305608
GlobalTracer [candidate] (307.467 ms) : 0, 307467
AppSec [baseline] (57.147 ms) : 0, 57147
AppSec [candidate] (56.57 ms) : 0, 56570
IAST [baseline] (21.582 ms) : 0, 21582
IAST [candidate] (22.728 ms) : 0, 22728
Remote Config [baseline] (622.269 µs) : 0, 622
Remote Config [candidate] (622.899 µs) : 0, 623
Telemetry [baseline] (7.531 ms) : 0, 7531
Telemetry [candidate] (7.435 ms) : 0, 7435
section profiling
BytebuddyAgent [baseline] (683.257 ms) : 0, 683257
BytebuddyAgent [candidate] (683.234 ms) : 0, 683234
GlobalTracer [baseline] (399.327 ms) : 0, 399327
GlobalTracer [candidate] (398.705 ms) : 0, 398705
AppSec [baseline] (54.792 ms) : 0, 54792
AppSec [candidate] (54.701 ms) : 0, 54701
Remote Config [baseline] (678.083 µs) : 0, 678
Remote Config [candidate] (683.308 µs) : 0, 683
Telemetry [baseline] (12.09 ms) : 0, 12090
Telemetry [candidate] (13.497 ms) : 0, 13497
ProfilingAgent [baseline] (92.323 ms) : 0, 92323
ProfilingAgent [candidate] (91.847 ms) : 0, 91847
Profiling [baseline] (92.347 ms) : 0, 92347
Profiling [candidate] (91.872 ms) : 0, 91872
Loading

Load

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/crashtracking-adjust-timeout
git_commit_date 1730981645 1730993794
git_commit_sha 060ab72 f330e67
release_version 1.43.0-SNAPSHOT~060ab7250f 1.43.0-SNAPSHOT~f330e675cd
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1730995731 1730995731
ci_job_id 699144910 699144910
ci_pipeline_id 48464158 48464158
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~f330e675cd, baseline=1.43.0-SNAPSHOT~060ab7250f
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.864 s) : 14864000, 14864000
.   : milestone, 14864000,
appsec (15.223 s) : 15223000, 15223000
.   : milestone, 15223000,
iast (18.794 s) : 18794000, 18794000
.   : milestone, 18794000,
iast_GLOBAL (18.136 s) : 18136000, 18136000
.   : milestone, 18136000,
profiling (15.301 s) : 15301000, 15301000
.   : milestone, 15301000,
tracing (15.079 s) : 15079000, 15079000
.   : milestone, 15079000,
section candidate
no_agent (14.971 s) : 14971000, 14971000
.   : milestone, 14971000,
appsec (15.286 s) : 15286000, 15286000
.   : milestone, 15286000,
iast (18.947 s) : 18947000, 18947000
.   : milestone, 18947000,
iast_GLOBAL (18.98 s) : 18980000, 18980000
.   : milestone, 18980000,
profiling (15.691 s) : 15691000, 15691000
.   : milestone, 15691000,
tracing (15.15 s) : 15150000, 15150000
.   : milestone, 15150000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.864 s [14.864 s, 14.864 s] -
appsec 15.223 s [15.223 s, 15.223 s] 359.0 ms (2.4%)
iast 18.794 s [18.794 s, 18.794 s] 3.93 s (26.4%)
iast_GLOBAL 18.136 s [18.136 s, 18.136 s] 3.272 s (22.0%)
profiling 15.301 s [15.301 s, 15.301 s] 437.0 ms (2.9%)
tracing 15.079 s [15.079 s, 15.079 s] 215.0 ms (1.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.971 s [14.971 s, 14.971 s] -
appsec 15.286 s [15.286 s, 15.286 s] 315.0 ms (2.1%)
iast 18.947 s [18.947 s, 18.947 s] 3.976 s (26.6%)
iast_GLOBAL 18.98 s [18.98 s, 18.98 s] 4.009 s (26.8%)
profiling 15.691 s [15.691 s, 15.691 s] 720.0 ms (4.8%)
tracing 15.15 s [15.15 s, 15.15 s] 179.0 ms (1.2%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~f330e675cd, baseline=1.43.0-SNAPSHOT~060ab7250f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1464, 1488
.   : milestone, 1476,
appsec (2.335 ms) : 2294, 2376
.   : milestone, 2335,
iast (2.069 ms) : 2017, 2120
.   : milestone, 2069,
iast_GLOBAL (2.125 ms) : 2072, 2178
.   : milestone, 2125,
profiling (1.946 ms) : 1904, 1989
.   : milestone, 1946,
tracing (1.918 ms) : 1878, 1958
.   : milestone, 1918,
section candidate
no_agent (1.469 ms) : 1458, 1481
.   : milestone, 1469,
appsec (2.324 ms) : 2283, 2366
.   : milestone, 2324,
iast (2.071 ms) : 2019, 2123
.   : milestone, 2071,
iast_GLOBAL (2.115 ms) : 2063, 2166
.   : milestone, 2115,
profiling (1.944 ms) : 1902, 1985
.   : milestone, 1944,
tracing (1.919 ms) : 1880, 1959
.   : milestone, 1919,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.476 ms [1.464 ms, 1.488 ms] -
appsec 2.335 ms [2.294 ms, 2.376 ms] 858.8 µs (58.2%)
iast 2.069 ms [2.017 ms, 2.12 ms] 592.717 µs (40.2%)
iast_GLOBAL 2.125 ms [2.072 ms, 2.178 ms] 649.039 µs (44.0%)
profiling 1.946 ms [1.904 ms, 1.989 ms] 470.23 µs (31.9%)
tracing 1.918 ms [1.878 ms, 1.958 ms] 441.969 µs (29.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.469 ms [1.458 ms, 1.481 ms] -
appsec 2.324 ms [2.283 ms, 2.366 ms] 854.864 µs (58.2%)
iast 2.071 ms [2.019 ms, 2.123 ms] 601.423 µs (40.9%)
iast_GLOBAL 2.115 ms [2.063 ms, 2.166 ms] 645.129 µs (43.9%)
profiling 1.944 ms [1.902 ms, 1.985 ms] 474.382 µs (32.3%)
tracing 1.919 ms [1.88 ms, 1.959 ms] 449.7 µs (30.6%)

@dougqh dougqh merged commit 7680873 into master Nov 7, 2024
101 checks passed
@dougqh dougqh deleted the dougqh/crashtracking-adjust-timeout branch November 7, 2024 18:16
@github-actions github-actions bot added this to the 1.43.0 milestone Nov 7, 2024
@PerfectSlayer
Copy link
Contributor

The current implementation of crash tracking doesn't work with ancient version of bash that ships with OS X by default.

Can you give a little more details about it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jbachorik jbachorik jbachorik approved these changes

@smola smola Awaiting requested review from smola smola is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: crash tracking Crash tracking type: enhancement
Projects
None yet
Milestone
1.43.0
Development

Successfully merging this pull request may close these issues.
3 participants
@dougqh @PerfectSlayer @jbachorik