Override keepAlive time to be lower then NLB idle time (350s) #130
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
javsanbel2
reviewed
Nov 30, 2023
templates/waggledance.json
Outdated
| "systemControls": [ | ||
| { | ||
| "namespace": "net.ipv4.tcp_keepalive_time", | ||
| "value": "200" |
There was a problem hiding this comment.
The best would be to inject variables here so anyone can override based on their infrastructure.
There was a problem hiding this comment.
will there be a separate PR for K8S - https://github.com/ExpediaGroup/apiary-federation/blob/master/k8s.tf ?
There was a problem hiding this comment.
You cannot do this easily in Kubernetes, it's not allowed to set these flags.
You get errors like:
PodSecurityPolicy: unable
--
to admit pod: [pod.spec.securityContext.sysctls[0]: Forbidden: unsafe
sysctl "net.ipv4.tcp_keepalive_time" is not allowed
pod.spec.securityContext.sysctls[1]: Forbidden: unsafe sysctl
"net.ipv4.tcp_keepalive_intvl" is not allowed
pod.spec.securityContext.sysctls[2]: Forbidden: unsafe sysctl
"net.ipv4.tcp_keepalive_probes" is not allowed]
So we need to sort this out separately
| @@ -359,3 +359,21 @@ variable "datadog_metrics_enabled" { | |||
| type = bool | |||
| default = false | |||
| } | |||
|
|
|||
| variable "tcp_keepalive_time" { | |||
There was a problem hiding this comment.
variables need to be added in README
abhimanyugupta07
approved these changes
Nov 30, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See for instance this blogpost: https://paramount.tech/blog/2021/07/26/mitigation-of-connection-reset-in-aws.html and https://medium.com/tenable-techblog/lessons-from-aws-nlb-timeouts-5028a8f65dda
We've seen a reduced number of 10 mins (connection timeout) reported by waggle dance once we set this lower TCP keepalive settings. Going from 30-40 timeout calls an hour to single digit. Occurrences still happen we suspect because the server side (HMS) should also set similar TCP KeepAlive settings.