Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase lambda extension request timeout #7986

Merged
merged 1 commit into from
Nov 22, 2024
Merged

Increase lambda extension request timeout #7986

merged 1 commit into from
Nov 22, 2024

Conversation

nhulston
Copy link
Contributor

@nhulston nhulston commented Nov 20, 2024 •
edited by PerfectSlayer
Loading

What Does This Do

Increases timeout for communicating with the Lambda extension.

Motivation

Customers are occasionally getting

[dd.trace 2024-11-20 18:26:27:926 +0900] [main] ERROR datadog.trace.lambda.LambdaHandler - could not reach the extension, not injecting the context
java.io.InterruptedIOException: timeout
at datadog.okhttp3.RealCall.timeoutExit(RealCall.java:108)
at datadog.okhttp3.RealCall.execute(RealCall.java:97)
at datadog.trace.lambda.LambdaHandler.notifyEndInvocation(LambdaHandler.java:162)
at datadog.trace.agent.core.CoreTracer.notifyExtensionEnd(CoreTracer.java:956)

This should resolve this.

Additional Notes

Contributor Checklist

Jira ticket: SLES-1952

@nhulston nhulston added tag: no release notes Changes to exclude from release notes inst: aws lambda AWS Lambda instrumentation labels Nov 20, 2024
@pr-commenter
Copy link

pr-commenter bot commented Nov 20, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nicholas.hulston/increase-lambda-extension-request-timeout
git_commit_date 1732122216 1732136642
git_commit_sha df51f63 d499e4e
release_version 1.43.0-SNAPSHOT~df51f63bab 1.43.0-SNAPSHOT~d499e4edd1
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1732139152 1732139152
ci_job_id 714631033 714631033
ci_pipeline_id 49549697 49549697
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 53 metrics, 10 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.43.0-SNAPSHOT~d499e4edd1, baseline=1.43.0-SNAPSHOT~df51f63bab

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.098 s) : 0, 1098190
Total [baseline] (10.584 s) : 0, 10583640
Agent [candidate] (1.091 s) : 0, 1091301
Total [candidate] (10.415 s) : 0, 10415216
section appsec
Agent [baseline] (1.221 s) : 0, 1221170
Total [baseline] (10.691 s) : 0, 10690983
Agent [candidate] (1.227 s) : 0, 1227099
Total [candidate] (10.736 s) : 0, 10735630
section iast
Agent [baseline] (1.216 s) : 0, 1215946
Total [baseline] (10.964 s) : 0, 10964120
Agent [candidate] (1.233 s) : 0, 1232713
Total [candidate] (11.009 s) : 0, 11009114
section profiling
Agent [baseline] (1.288 s) : 0, 1287547
Total [baseline] (10.849 s) : 0, 10849415
Agent [candidate] (1.293 s) : 0, 1293441
Total [candidate] (10.773 s) : 0, 10773229
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.098 s -
Agent appsec 1.221 s 122.98 ms (11.2%)
Agent iast 1.216 s 117.757 ms (10.7%)
Agent profiling 1.288 s 189.357 ms (17.2%)
Total tracing 10.584 s -
Total appsec 10.691 s 107.343 ms (1.0%)
Total iast 10.964 s 380.48 ms (3.6%)
Total profiling 10.849 s 265.775 ms (2.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.091 s -
Agent appsec 1.227 s 135.799 ms (12.4%)
Agent iast 1.233 s 141.412 ms (13.0%)
Agent profiling 1.293 s 202.14 ms (18.5%)
Total tracing 10.415 s -
Total appsec 10.736 s 320.413 ms (3.1%)
Total iast 11.009 s 593.898 ms (5.7%)
Total profiling 10.773 s 358.013 ms (3.4%) 
gantt
    title petclinic - break down per module: candidate=1.43.0-SNAPSHOT~d499e4edd1, baseline=1.43.0-SNAPSHOT~df51f63bab

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (697.826 ms) : 0, 697826
BytebuddyAgent [candidate] (691.356 ms) : 0, 691356
GlobalTracer [baseline] (320.755 ms) : 0, 320755
GlobalTracer [candidate] (318.437 ms) : 0, 318437
AppSec [baseline] (55.032 ms) : 0, 55032
AppSec [candidate] (54.919 ms) : 0, 54919
Remote Config [baseline] (714.585 µs) : 0, 715
Remote Config [candidate] (686.263 µs) : 0, 686
Telemetry [baseline] (10.051 ms) : 0, 10051
Telemetry [candidate] (12.225 ms) : 0, 12225
section appsec
BytebuddyAgent [baseline] (708.301 ms) : 0, 708301
BytebuddyAgent [candidate] (712.237 ms) : 0, 712237
GlobalTracer [baseline] (313.986 ms) : 0, 313986
GlobalTracer [candidate] (315.237 ms) : 0, 315237
AppSec [baseline] (166.483 ms) : 0, 166483
AppSec [candidate] (165.578 ms) : 0, 165578
Remote Config [baseline] (645.228 µs) : 0, 645
Remote Config [candidate] (641.837 µs) : 0, 642
Telemetry [baseline] (7.876 ms) : 0, 7876
Telemetry [candidate] (8.291 ms) : 0, 8291
IAST [baseline] (20.497 ms) : 0, 20497
IAST [candidate] (22.713 ms) : 0, 22713
section iast
BytebuddyAgent [baseline] (809.215 ms) : 0, 809215
BytebuddyAgent [candidate] (821.419 ms) : 0, 821419
GlobalTracer [baseline] (306.096 ms) : 0, 306096
GlobalTracer [candidate] (309.816 ms) : 0, 309816
AppSec [baseline] (58.096 ms) : 0, 58096
AppSec [candidate] (58.288 ms) : 0, 58288
Remote Config [baseline] (620.607 µs) : 0, 621
Remote Config [candidate] (611.454 µs) : 0, 611
Telemetry [baseline] (7.472 ms) : 0, 7472
Telemetry [candidate] (7.548 ms) : 0, 7548
IAST [baseline] (20.788 ms) : 0, 20788
IAST [candidate] (21.143 ms) : 0, 21143
section profiling
ProfilingAgent [baseline] (94.614 ms) : 0, 94614
ProfilingAgent [candidate] (92.366 ms) : 0, 92366
BytebuddyAgent [baseline] (686.31 ms) : 0, 686310
BytebuddyAgent [candidate] (692.289 ms) : 0, 692289
GlobalTracer [baseline] (401.016 ms) : 0, 401016
GlobalTracer [candidate] (401.468 ms) : 0, 401468
AppSec [baseline] (55.286 ms) : 0, 55286
AppSec [candidate] (55.177 ms) : 0, 55177
Remote Config [baseline] (668.558 µs) : 0, 669
Remote Config [candidate] (664.416 µs) : 0, 664
Telemetry [baseline] (10.678 ms) : 0, 10678
Telemetry [candidate] (12.087 ms) : 0, 12087
Profiling [baseline] (94.637 ms) : 0, 94637
Profiling [candidate] (92.389 ms) : 0, 92389
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.43.0-SNAPSHOT~d499e4edd1, baseline=1.43.0-SNAPSHOT~df51f63bab

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.096 s) : 0, 1096130
Total [baseline] (8.67 s) : 0, 8670229
Agent [candidate] (1.088 s) : 0, 1087551
Total [candidate] (8.648 s) : 0, 8648126
section iast
Agent [baseline] (1.224 s) : 0, 1223773
Total [baseline] (9.169 s) : 0, 9169017
Agent [candidate] (1.216 s) : 0, 1215947
Total [candidate] (9.169 s) : 0, 9169259
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.216 s) : 0, 1215954
Total [baseline] (9.227 s) : 0, 9227133
Agent [candidate] (1.223 s) : 0, 1222696
Total [candidate] (9.173 s) : 0, 9172500
section iast_TELEMETRY_OFF
Agent [baseline] (1.211 s) : 0, 1211422
Total [baseline] (9.162 s) : 0, 9161829
Agent [candidate] (1.212 s) : 0, 1211611
Total [candidate] (9.172 s) : 0, 9171966
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.096 s -
Agent iast 1.224 s 127.643 ms (11.6%)
Agent iast_HARDCODED_SECRET_DISABLED 1.216 s 119.824 ms (10.9%)
Agent iast_TELEMETRY_OFF 1.211 s 115.292 ms (10.5%)
Total tracing 8.67 s -
Total iast 9.169 s 498.789 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.227 s 556.904 ms (6.4%)
Total iast_TELEMETRY_OFF 9.162 s 491.6 ms (5.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.088 s -
Agent iast 1.216 s 128.396 ms (11.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.223 s 135.145 ms (12.4%)
Agent iast_TELEMETRY_OFF 1.212 s 124.061 ms (11.4%)
Total tracing 8.648 s -
Total iast 9.169 s 521.133 ms (6.0%)
Total iast_HARDCODED_SECRET_DISABLED 9.173 s 524.375 ms (6.1%)
Total iast_TELEMETRY_OFF 9.172 s 523.841 ms (6.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.43.0-SNAPSHOT~d499e4edd1, baseline=1.43.0-SNAPSHOT~df51f63bab

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (697.89 ms) : 0, 697890
BytebuddyAgent [candidate] (691.909 ms) : 0, 691909
GlobalTracer [baseline] (319.007 ms) : 0, 319007
GlobalTracer [candidate] (316.694 ms) : 0, 316694
AppSec [baseline] (54.725 ms) : 0, 54725
AppSec [candidate] (54.66 ms) : 0, 54660
Remote Config [baseline] (681.109 µs) : 0, 681
Remote Config [candidate] (683.738 µs) : 0, 684
Telemetry [baseline] (10.003 ms) : 0, 10003
Telemetry [candidate] (9.863 ms) : 0, 9863
section iast
BytebuddyAgent [baseline] (815.178 ms) : 0, 815178
BytebuddyAgent [candidate] (809.231 ms) : 0, 809231
GlobalTracer [baseline] (307.375 ms) : 0, 307375
GlobalTracer [candidate] (305.992 ms) : 0, 305992
AppSec [baseline] (58.218 ms) : 0, 58218
AppSec [candidate] (57.45 ms) : 0, 57450
IAST [baseline] (21.05 ms) : 0, 21050
IAST [candidate] (21.454 ms) : 0, 21454
Remote Config [baseline] (610.24 µs) : 0, 610
Remote Config [candidate] (609.813 µs) : 0, 610
Telemetry [baseline] (7.528 ms) : 0, 7528
Telemetry [candidate] (7.445 ms) : 0, 7445
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (808.475 ms) : 0, 808475
BytebuddyAgent [candidate] (814.519 ms) : 0, 814519
GlobalTracer [baseline] (306.453 ms) : 0, 306453
GlobalTracer [candidate] (307.803 ms) : 0, 307803
AppSec [baseline] (57.235 ms) : 0, 57235
AppSec [candidate] (57.893 ms) : 0, 57893
IAST [baseline] (21.868 ms) : 0, 21868
IAST [candidate] (20.658 ms) : 0, 20658
Remote Config [baseline] (681.543 µs) : 0, 682
Remote Config [candidate] (602.497 µs) : 0, 602
Telemetry [baseline] (7.556 ms) : 0, 7556
Telemetry [candidate] (7.397 ms) : 0, 7397
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (805.094 ms) : 0, 805094
BytebuddyAgent [candidate] (805.167 ms) : 0, 805167
GlobalTracer [baseline] (306.046 ms) : 0, 306046
GlobalTracer [candidate] (305.776 ms) : 0, 305776
AppSec [baseline] (58.284 ms) : 0, 58284
AppSec [candidate] (58.386 ms) : 0, 58386
IAST [baseline] (20.331 ms) : 0, 20331
IAST [candidate] (20.584 ms) : 0, 20584
Remote Config [baseline] (607.282 µs) : 0, 607
Remote Config [candidate] (622.689 µs) : 0, 623
Telemetry [baseline] (7.363 ms) : 0, 7363
Telemetry [candidate] (7.415 ms) : 0, 7415
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-11-20T21:13:58 2024-11-20T21:20:57
git_branch master nicholas.hulston/increase-lambda-extension-request-timeout
git_commit_date 1732122216 1732136642
git_commit_sha df51f63 d499e4e
release_version 1.43.0-SNAPSHOT~df51f63bab 1.43.0-SNAPSHOT~d499e4edd1
start_time 2024-11-20T21:13:45 2024-11-20T21:20:43
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1732138012 1732138012
ci_job_id 714631034 714631034
ci_pipeline_id 49549697 49549697
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~d499e4edd1, baseline=1.43.0-SNAPSHOT~df51f63bab
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.361 ms) : 1342, 1380
.   : milestone, 1361,
appsec (1.744 ms) : 1719, 1768
.   : milestone, 1744,
appsec_no_iast (1.754 ms) : 1729, 1779
.   : milestone, 1754,
iast (1.508 ms) : 1485, 1531
.   : milestone, 1508,
profiling (1.513 ms) : 1490, 1536
.   : milestone, 1513,
tracing (1.503 ms) : 1478, 1527
.   : milestone, 1503,
section candidate
no_agent (1.361 ms) : 1342, 1380
.   : milestone, 1361,
appsec (1.743 ms) : 1718, 1768
.   : milestone, 1743,
appsec_no_iast (1.752 ms) : 1726, 1777
.   : milestone, 1752,
iast (1.519 ms) : 1497, 1542
.   : milestone, 1519,
profiling (1.545 ms) : 1520, 1569
.   : milestone, 1545,
tracing (1.497 ms) : 1472, 1522
.   : milestone, 1497,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.361 ms [1.342 ms, 1.38 ms] -
appsec 1.744 ms [1.719 ms, 1.768 ms] 382.802 µs (28.1%)
appsec_no_iast 1.754 ms [1.729 ms, 1.779 ms] 393.264 µs (28.9%)
iast 1.508 ms [1.485 ms, 1.531 ms] 147.382 µs (10.8%)
profiling 1.513 ms [1.49 ms, 1.536 ms] 152.225 µs (11.2%)
tracing 1.503 ms [1.478 ms, 1.527 ms] 141.924 µs (10.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.361 ms [1.342 ms, 1.38 ms] -
appsec 1.743 ms [1.718 ms, 1.768 ms] 381.636 µs (28.0%)
appsec_no_iast 1.752 ms [1.726 ms, 1.777 ms] 390.309 µs (28.7%)
iast 1.519 ms [1.497 ms, 1.542 ms] 158.209 µs (11.6%)
profiling 1.545 ms [1.52 ms, 1.569 ms] 183.387 µs (13.5%)
tracing 1.497 ms [1.472 ms, 1.522 ms] 135.74 µs (10.0%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~d499e4edd1, baseline=1.43.0-SNAPSHOT~df51f63bab
    dateFormat X
    axisFormat %s
section baseline
no_agent (376.993 µs) : 357, 397
.   : milestone, 377,
iast (489.74 µs) : 468, 511
.   : milestone, 490,
iast_FULL (651.17 µs) : 630, 673
.   : milestone, 651,
iast_GLOBAL (516.844 µs) : 494, 539
.   : milestone, 517,
iast_HARDCODED_SECRET_DISABLED (492.862 µs) : 471, 514
.   : milestone, 493,
iast_INACTIVE (454.854 µs) : 434, 476
.   : milestone, 455,
iast_TELEMETRY_OFF (488.956 µs) : 467, 511
.   : milestone, 489,
tracing (447.503 µs) : 427, 468
.   : milestone, 448,
section candidate
no_agent (372.866 µs) : 353, 393
.   : milestone, 373,
iast (491.323 µs) : 470, 513
.   : milestone, 491,
iast_FULL (654.96 µs) : 633, 677
.   : milestone, 655,
iast_GLOBAL (529.188 µs) : 507, 552
.   : milestone, 529,
iast_HARDCODED_SECRET_DISABLED (501.862 µs) : 480, 524
.   : milestone, 502,
iast_INACTIVE (461.983 µs) : 441, 483
.   : milestone, 462,
iast_TELEMETRY_OFF (478.399 µs) : 457, 500
.   : milestone, 478,
tracing (448.194 µs) : 427, 469
.   : milestone, 448,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 376.993 µs [357.265 µs, 396.721 µs] -
iast 489.74 µs [468.404 µs, 511.076 µs] 112.747 µs (29.9%)
iast_FULL 651.17 µs [629.628 µs, 672.713 µs] 274.177 µs (72.7%)
iast_GLOBAL 516.844 µs [494.45 µs, 539.237 µs] 139.85 µs (37.1%)
iast_HARDCODED_SECRET_DISABLED 492.862 µs [471.496 µs, 514.228 µs] 115.869 µs (30.7%)
iast_INACTIVE 454.854 µs [433.868 µs, 475.841 µs] 77.861 µs (20.7%)
iast_TELEMETRY_OFF 488.956 µs [467.192 µs, 510.72 µs] 111.963 µs (29.7%)
tracing 447.503 µs [426.882 µs, 468.124 µs] 70.51 µs (18.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 372.866 µs [352.887 µs, 392.845 µs] -
iast 491.323 µs [469.853 µs, 512.793 µs] 118.457 µs (31.8%)
iast_FULL 654.96 µs [633.374 µs, 676.546 µs] 282.094 µs (75.7%)
iast_GLOBAL 529.188 µs [506.597 µs, 551.78 µs] 156.322 µs (41.9%)
iast_HARDCODED_SECRET_DISABLED 501.862 µs [480.002 µs, 523.722 µs] 128.996 µs (34.6%)
iast_INACTIVE 461.983 µs [440.905 µs, 483.061 µs] 89.117 µs (23.9%)
iast_TELEMETRY_OFF 478.399 µs [457.186 µs, 499.612 µs] 105.533 µs (28.3%)
tracing 448.194 µs [427.405 µs, 468.983 µs] 75.328 µs (20.2%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nicholas.hulston/increase-lambda-extension-request-timeout
git_commit_date 1732122216 1732136642
git_commit_sha df51f63 d499e4e
release_version 1.43.0-SNAPSHOT~df51f63bab 1.43.0-SNAPSHOT~d499e4edd1
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1732138666 1732138666
ci_job_id 714631035 714631035
ci_pipeline_id 49549697 49549697
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~d499e4edd1, baseline=1.43.0-SNAPSHOT~df51f63bab
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.941 s) : 14941000, 14941000
.   : milestone, 14941000,
appsec (15.202 s) : 15202000, 15202000
.   : milestone, 15202000,
iast (18.561 s) : 18561000, 18561000
.   : milestone, 18561000,
iast_GLOBAL (17.993 s) : 17993000, 17993000
.   : milestone, 17993000,
profiling (14.904 s) : 14904000, 14904000
.   : milestone, 14904000,
tracing (15.133 s) : 15133000, 15133000
.   : milestone, 15133000,
section candidate
no_agent (14.742 s) : 14742000, 14742000
.   : milestone, 14742000,
appsec (15.158 s) : 15158000, 15158000
.   : milestone, 15158000,
iast (19.003 s) : 19003000, 19003000
.   : milestone, 19003000,
iast_GLOBAL (18.073 s) : 18073000, 18073000
.   : milestone, 18073000,
profiling (14.716 s) : 14716000, 14716000
.   : milestone, 14716000,
tracing (15.306 s) : 15306000, 15306000
.   : milestone, 15306000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.941 s [14.941 s, 14.941 s] -
appsec 15.202 s [15.202 s, 15.202 s] 261.0 ms (1.7%)
iast 18.561 s [18.561 s, 18.561 s] 3.62 s (24.2%)
iast_GLOBAL 17.993 s [17.993 s, 17.993 s] 3.052 s (20.4%)
profiling 14.904 s [14.904 s, 14.904 s] -37.0 ms (-0.2%)
tracing 15.133 s [15.133 s, 15.133 s] 192.0 ms (1.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.742 s [14.742 s, 14.742 s] -
appsec 15.158 s [15.158 s, 15.158 s] 416.0 ms (2.8%)
iast 19.003 s [19.003 s, 19.003 s] 4.261 s (28.9%)
iast_GLOBAL 18.073 s [18.073 s, 18.073 s] 3.331 s (22.6%)
profiling 14.716 s [14.716 s, 14.716 s] -26.0 ms (-0.2%)
tracing 15.306 s [15.306 s, 15.306 s] 564.0 ms (3.8%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~d499e4edd1, baseline=1.43.0-SNAPSHOT~df51f63bab
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.468 ms) : 1456, 1479
.   : milestone, 1468,
appsec (2.339 ms) : 2298, 2380
.   : milestone, 2339,
iast (2.076 ms) : 2024, 2128
.   : milestone, 2076,
iast_GLOBAL (2.124 ms) : 2072, 2177
.   : milestone, 2124,
profiling (1.95 ms) : 1907, 1992
.   : milestone, 1950,
tracing (1.916 ms) : 1876, 1955
.   : milestone, 1916,
section candidate
no_agent (1.473 ms) : 1461, 1485
.   : milestone, 1473,
appsec (2.343 ms) : 2301, 2384
.   : milestone, 2343,
iast (2.076 ms) : 2024, 2128
.   : milestone, 2076,
iast_GLOBAL (2.12 ms) : 2068, 2172
.   : milestone, 2120,
profiling (1.961 ms) : 1918, 2003
.   : milestone, 1961,
tracing (1.924 ms) : 1884, 1965
.   : milestone, 1924,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.468 ms [1.456 ms, 1.479 ms] -
appsec 2.339 ms [2.298 ms, 2.38 ms] 871.486 µs (59.4%)
iast 2.076 ms [2.024 ms, 2.128 ms] 608.694 µs (41.5%)
iast_GLOBAL 2.124 ms [2.072 ms, 2.177 ms] 656.621 µs (44.7%)
profiling 1.95 ms [1.907 ms, 1.992 ms] 481.912 µs (32.8%)
tracing 1.916 ms [1.876 ms, 1.955 ms] 448.086 µs (30.5%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.473 ms [1.461 ms, 1.485 ms] -
appsec 2.343 ms [2.301 ms, 2.384 ms] 869.543 µs (59.0%)
iast 2.076 ms [2.024 ms, 2.128 ms] 602.656 µs (40.9%)
iast_GLOBAL 2.12 ms [2.068 ms, 2.172 ms] 647.024 µs (43.9%)
profiling 1.961 ms [1.918 ms, 2.003 ms] 487.496 µs (33.1%)
tracing 1.924 ms [1.884 ms, 1.965 ms] 451.352 µs (30.6%)

@nhulston nhulston force-pushed the nicholas.hulston/increase-lambda-extension-request-timeout branch from a7d44aa to d499e4e Compare November 20, 2024 21:04
@nhulston nhulston marked this pull request as ready for review November 20, 2024 21:51
@nhulston nhulston requested a review from a team as a code owner November 20, 2024 21:51
purple4reina
purple4reina reviewed Nov 20, 2024
View reviewed changes
dd-trace-core/src/main/java/datadog/trace/lambda/LambdaHandler.java
@@ -44,7 +44,7 @@ public class LambdaHandler {
private static final String START_INVOCATION = "/lambda/start-invocation";
private static final String END_INVOCATION = "/lambda/end-invocation";

private static final Long REQUEST_TIMEOUT_IN_S = 1L;
private static final Long REQUEST_TIMEOUT_IN_S = 3L;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems really weird that a 1 second timeout wouldn't be enough. Have you been able to reproduce these errors?

If you were able to reproduce the errors, did this change solve it?

Copy link
Contributor Author

@nhulston nhulston Nov 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://datadoghq.atlassian.net/browse/SLES-1952?focusedCommentId=2021461 for more details.

Maybe implementing a retry mechanism would be a better solution, WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this solution is fine 😄 Since we've been able to reproduce the error in self monitoring, in a perfect world we would build a layer from this branch and test it out for a bit. But since this change is otherwise harmless, I think just going ahead with it is fine. Though, for the customer's sake, I still think we should test this change out in self monitoring to ensure it indeed fixes the issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that's a good point. This might solve the problem, but maybe not. We only saw it in self monitoring once every few days, so we'll have to wait a while and see if the messages ever pops up again.

PerfectSlayer
PerfectSlayer reviewed Nov 21, 2024
View reviewed changes
Copy link
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Customers are occasionally getting

If customers are seeing this kind of error, we might want to add this PR in the release notes (as bug fix or enhancement, whatever fits best)

@nhulston nhulston removed the tag: no release notes Changes to exclude from release notes label Nov 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@nhulston nhulston merged commit c8030bd into master Nov 22, 2024
104 of 105 checks passed
@nhulston nhulston deleted the nicholas.hulston/increase-lambda-extension-request-timeout branch November 22, 2024 21:30
@github-actions github-actions bot added this to the 1.43.0 milestone Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer left review comments

@purple4reina purple4reina purple4reina approved these changes

@manuel-alvarez-alvarez manuel-alvarez-alvarez Awaiting requested review from manuel-alvarez-alvarez manuel-alvarez-alvarez is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
inst: aws lambda AWS Lambda instrumentation type: enhancement
Projects
None yet
Milestone
1.43.0
Development

Successfully merging this pull request may close these issues.
3 participants
@nhulston @purple4reina @PerfectSlayer