Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(grpcproxy): add missing GRPCKeepAliveEnforcementMinimum #15708

Merged
merged 7 commits into from
Oct 17, 2023
Merged

fix(grpcproxy): add missing GRPCKeepAliveEnforcementMinimum #15708

merged 7 commits into from
Oct 17, 2023

Conversation

phanama
Copy link
Contributor

@phanama phanama commented Sep 28, 2023 •
edited
Loading

The keepalive MinTime parameter with value GRPCKeepAliveEnforcementMinimum is missing from apiclient's local grpcproxy when using GrpcWeb=true. This causes ENHANCE_YOUR_CALM errors for long-running requests taking >80s, due to the client having lower ping interval value (20s) than the local grpcproxy server's (default 5m ref). See grpc keepalive doc.

The same parameter was added in #9922 but probably was skipped in the grpcproxy implementation.

Fixes: #15707

Checklist:

  • Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
  • The title of the PR states what changed and the related issues number (used for the release note).
  • The title of the PR conforms to the Toolchain Guide
  • I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
  • I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
  • Does this PR require documentation updates?
  • I've updated documentation as required by this PR.
  • Optional. My organization is added to USERS.md.
  • I have signed off all my commits as required by DCO
  • I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
  • My build is green (troubleshooting builds).
  • My new feature complies with the feature status guidelines.
  • I have added a brief description of why this PR is necessary and/or what this PR solves.

@phanama phanama requested a review from a team as a code owner September 28, 2023 06:40
@codecov
Copy link

codecov bot commented Sep 28, 2023 •
edited
Loading

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (6dbd47e) 49.67% compared to head (1c7472d) 49.66%.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master   #15708      +/-   ##
==========================================
- Coverage   49.67%   49.66%   -0.01%     
==========================================
  Files         267      267              
  Lines       46383    46388       +5     
==========================================
  Hits        23039    23039              
- Misses      21084    21089       +5     
  Partials     2260     2260
Files Coverage Δ
pkg/apiclient/grpcproxy.go 0.00% <0.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@phanama
fix(grpcproxy): add GRPCKeepAliveEnforcementMinimum
c967706 
the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
@phanama phanama force-pushed the grpcproxy-keepalive branch from a441f1f to c967706 Compare September 29, 2023 02:41
@phanama
Copy link
Contributor Author

phanama commented Sep 30, 2023

The local grpcproxy is also missing some other server parameters, in particular these three:

argo-cd/server/server.go

Lines 729 to 731 in ef88d1d

grpc.MaxRecvMsgSize(apiclient.MaxGRPCMessageSize),
grpc.MaxSendMsgSize(apiclient.MaxGRPCMessageSize),
grpc.ConnectionTimeout(300 * time.Second),

I'll leave it in separate community discussion, whether or not these are needed in the local grpcproxy.
One concern might be MaxRecvMsgSize as the default might be small at 4MB (corresponding to client's max request size to the local grpcproxy). The MaxSendMsgSize (which corresponds to the response's max size) default is large at 2GB so might not be much of a concern.

todaywasawesome
todaywasawesome approved these changes Oct 5, 2023
View reviewed changes
Copy link
Contributor

@todaywasawesome todaywasawesome left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, @crenshaw-dev should we consider backporting this commit? Realistically they should have the minimums enforced.

@todaywasawesome
Copy link
Contributor

Related to #15707

ishitasequeira
ishitasequeira approved these changes Oct 17, 2023
View reviewed changes
Copy link
Member

@ishitasequeira ishitasequeira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ishitasequeira ishitasequeira merged commit f7788a7 into argoproj:master Oct 17, 2023
@ishitasequeira ishitasequeira mentioned this pull request Oct 17, 2023
Merged
13 tasks
@phanama
Copy link
Contributor Author

phanama commented Oct 18, 2023

@ishitasequeira Thank you! Can we cherrypick this bugfix into v2.8, v2.7, and v2.6?

ymktmk pushed a commit to ymktmk/argo-cd that referenced this pull request Oct 29, 2023
@phanama @todaywasawesome @ymktmk
fix(grpcproxy): add GRPCKeepAliveEnforcementMinimum (argoproj#15708)
c6b395a 
the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
@phanama phanama deleted the grpcproxy-keepalive branch November 1, 2023 02:51
jmilic1 pushed a commit to jmilic1/argo-cd that referenced this pull request Nov 13, 2023
@phanama @todaywasawesome @jmilic1
fix(grpcproxy): add GRPCKeepAliveEnforcementMinimum (argoproj#15708)
2b5d335 
the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
Signed-off-by: jmilic1 <[email protected]>
@joshuabezaleel
Copy link

@ishitasequeira @todaywasawesome Hi Ishita and Dan! 👋
Friendly check, (1) whether we have the flow in argo project to cherrypick the bugfix on the recent supported versions and (2) whether that can be done.

Since this still bites us in production that affects hundreds of our engineers and updating this on our argo dependency will be a much easier fix than disabling grpcweb. Seems like some people have also brought this fix to their forked argo 😄

Thank you lots in advance! 🙂

@todaywasawesome
Copy link
Contributor

Good call, We can do that. I'll take it on.

@joshuabezaleel
Copy link

@todaywasawesome sincerely apologize for the nudge, any update on this? 🙂

@todaywasawesome
Copy link
Contributor

Oh thanks for the reminder, I'll do today.

@todaywasawesome
Copy link
Contributor

/cherry-pick release-2.8

@todaywasawesome
Copy link
Contributor

/cherry-pick release-2.7

@todaywasawesome todaywasawesome added cherry-pick/2.7 Candidate for cherry picking into the 2.7 release branch cherry-pick/2.8 labels Dec 7, 2023
This was referenced Dec 7, 2023
Merged
Merged
maxbrunet pushed a commit to maxbrunet/argo-cd that referenced this pull request Dec 7, 2023
@phanama @todaywasawesome @maxbrunet
fix(grpcproxy): add GRPCKeepAliveEnforcementMinimum (argoproj#15708)
a7afec7 
the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
maxbrunet pushed a commit to maxbrunet/argo-cd that referenced this pull request Dec 7, 2023
@phanama @todaywasawesome @maxbrunet
fix(grpcproxy): add missing GRPCKeepAliveEnforcementMinimum (argoproj…
a9ddded 
…#15708)

the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
maxbrunet pushed a commit to maxbrunet/argo-cd that referenced this pull request Dec 7, 2023
@phanama @todaywasawesome @maxbrunet
fix(grpcproxy): add missing GRPCKeepAliveEnforcementMinimum (argoproj…
2b722b7 
…#15708)

the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
Signed-off-by: Maxime Brunet <[email protected]>
@maxbrunet maxbrunet mentioned this pull request Dec 7, 2023
Merged
14 tasks
ishitasequeira pushed a commit that referenced this pull request Dec 8, 2023
@maxbrunet @phanama @todaywasawesome
fix(grpcproxy): add missing GRPCKeepAliveEnforcementMinimum (#15708) (#…
ab3b2e7 
…16576)

the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Signed-off-by: Maxime Brunet <[email protected]>
Co-authored-by: Yudi A Phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
vladfr pushed a commit to vladfr/argo-cd that referenced this pull request Dec 13, 2023
@phanama @todaywasawesome @vladfr
fix(grpcproxy): add GRPCKeepAliveEnforcementMinimum (argoproj#15708)
3a2cf41 
the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
tesla59 pushed a commit to tesla59/argo-cd that referenced this pull request Dec 16, 2023
@phanama @todaywasawesome @tesla59
fix(grpcproxy): add GRPCKeepAliveEnforcementMinimum (argoproj#15708)
7eb502c 
the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
lukaszgyg pushed a commit to lukaszgyg/argo-cd that referenced this pull request Jan 12, 2024
@phanama @todaywasawesome @lukaszgyg
fix(grpcproxy): add GRPCKeepAliveEnforcementMinimum (argoproj#15708)
ea7a9bf 
the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
@danielkza
Copy link

Can we get a release for 2.6/2.7/2.8 series with this fix? It would be most appreciated.

@maxbrunet maxbrunet mentioned this pull request Jan 25, 2024
Open
3 tasks
Hariharasuthan99 pushed a commit to AmadeusITGroup/argo-cd that referenced this pull request Jun 16, 2024
@phanama @todaywasawesome @Hariharasuthan99
fix(grpcproxy): add GRPCKeepAliveEnforcementMinimum (argoproj#15708)
d60ae45 
the absence of the setting potentially causes ENHANCE_YOUR_CALM

Signed-off-by: phanama <[email protected]>
Co-authored-by: Dan Garfield <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ishitasequeira ishitasequeira ishitasequeira approved these changes

@todaywasawesome todaywasawesome todaywasawesome approved these changes

Assignees
No one assigned
Labels
cherry-pick/2.7 Candidate for cherry picking into the 2.7 release branch cherry-pick/2.8
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

bug: api clients with GrpcWeb=true constantly gets too_many_pings error for requests taking >80s
5 participants
@phanama @todaywasawesome @joshuabezaleel @danielkza @ishitasequeira