Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EDGECLOUD-5230: Envoy crashing for UDP packet size larger than 1500 #1443

Merged
merged 3 commits into from
Aug 9, 2021

Conversation

ashxjain
Copy link
Contributor

@ashxjain ashxjain commented Aug 4, 2021

Issues Fixed

  • EDGECLOUD-5230: Envoy crashing for UDP packet size larger than 1500

Description

  • This issue was faced by CloudSLAM. Their Android App was sending UDP packets of size 3088 and it was crashing envoy. We were hitting this bug:
    crash when udp packet size large than 1500 envoyproxy/envoy#14113
  • This was fixed in envoy version 1.15.3, but it was still dropping UDP packets with size > 1500. They added support to set max datagram size as part of version 1.18.x. But 1.18.x transitioned to v3 API and we are using v2 API. Have updated the code to use v3 API and also envoy version 1.18.x
  • Have added pktsize config to specify datagram size for UDP ports. Note: using value > 1500 means envoy will allocate more memory, hence set it only for those ports which require this configuration
  • We won't need UI support for this, as it will be used very rarely (as of now only by CloudSLAM). We can document it so that users can use CLI for setting it up
  • Added debug command to get the list of envoy versions used by RootLBs. Sample output:
sharedrootlb: test.test.net
  container: envoycloudslam1341-stable14ub1804
    envoy  version: 98c1c9e9a40804b93b074badad1cdf284b47d58b/1.18.3/Clean/RELEASE/BoringSSL
  container: envoycloudslam1341-stable
    envoy  version: 98c1c9e9a40804b93b074badad1cdf284b47d58b/1.15.3/Clean/RELEASE/BoringSSL

dedicatedredrootlb: test121.test.net
  container: envoycloudslam1341-stable14ub1804
    envoy  version: 98c1c9e9a40804b93b074badad1cdf284b47d58b/1.18.3/Clean/RELEASE/BoringSSL
  container: envoycloudslam1341-stable
    envoy  version: 98c1c9e9a40804b93b074badad1cdf284b47d58b/1.15.3/Clean/RELEASE/BoringSSL

@ashxjain ashxjain changed the title EDGECLOUD-5230: Envoy crashing for UDP packet size large than 1500 EDGECLOUD-5230: Envoy crashing for UDP packet size larger than 1500 Aug 4, 2021
@gainsley
Copy link
Contributor

gainsley commented Aug 4, 2021

So this will only affect new deployments, right? I might suggest adding some way for us to determine at runtime which version of envoy each AppInst is using, in case they still hit the issue, so we can verify if they recreated their AppInst and have the correct envoy version. A crm debug cmd would probably be sufficient.

@ashxjain
Copy link
Contributor Author

ashxjain commented Aug 4, 2021

So this will only affect new deployments, right? I might suggest adding some way for us to determine at runtime which version of envoy each AppInst is using, in case they still hit the issue, so we can verify if they recreated their AppInst and have the correct envoy version. A crm debug cmd would probably be sufficient.

Yes, it will only affect new deployments. For cloudSLAM, they can re-create their cluster and none of our other developers have this problem.

Sure, debug cmd makes sense. I will add it. Thanks!

util/ports.go Outdated Show resolved Hide resolved
Copy link
Contributor

@gainsley gainsley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

cloud-resource-manager/proxy/nginx.go Outdated Show resolved Hide resolved
util/ports.go Outdated Show resolved Hide resolved
cloud-resource-manager/crmutil/exec.go Outdated Show resolved Hide resolved
Copy link
Contributor

@gainsley gainsley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ashxjain ashxjain merged commit 41839e9 into master Aug 9, 2021
@ashxjain ashxjain deleted the fixenvoyvers branch August 9, 2021 05:25
gainsley pushed a commit that referenced this pull request Apr 25, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 26, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 26, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 28, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 28, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 28, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 28, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 28, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
gainsley pushed a commit that referenced this pull request Apr 29, 2022
…1443)

* add debug cmd + use envoy version 1.18.x + update envoy yaml to use v3 api
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants