Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Elastic Agent] Handle 429 response from the server and adjust backoff #19918

Merged
merged 3 commits into from
Jul 14, 2020

Conversation

ph
Copy link
Contributor

@ph ph commented Jul 14, 2020

When enrolling and the server currently handle to many concurrent
request it will return a 429 status code. The enroll subcommand will
retry to enroll with an exponential backoff. (Init 15sec and max 10mins)

This also adjust the backoff logic in the ACK.

Requires: elastic/kibana#71552

What does this PR do?

Why is it important?

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

When enrolling and the server currently handle to many concurrent
request it will return a 429 status code. The enroll subcommand will
retry to enroll with an exponential backoff. (Init 15sec and max 10mins)

This also adjust the backoff logic in the ACK.

Requires: elastic/kibana#71552
@ph ph added review needs_backport PR is waiting to be backported to other branches. [zube]: In Review Team:Ingest Management labels Jul 14, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 14, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ingest-management (Team:Ingest Management)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 14, 2020
@ph ph requested a review from jfsiii July 14, 2020 19:54
backExp := backoff.NewExpBackoff(signal, 30*time.Second, 10*time.Minute)

for err == fleetapi.ErrTooManyRequests {
fmt.Fprintln(streams.Out, "Too many requests on the remote server, will retry in a moment.")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to print the wait time here? With 10min max someone might get impatient and think the beat is deadlocked

Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, agree with @urso that should add the wait time in the output.

@ph
Copy link
Contributor Author

ph commented Jul 14, 2020

@blakerouse I could add "(up to 10mins)", I would need to change the backoff package to expose the wait time. That package is used in multiple places so I feel its risky. Will make a follow up issue.

@ph ph merged commit 2db2152 into elastic:master Jul 14, 2020
ph added a commit to ph/beats that referenced this pull request Jul 14, 2020
elastic#19918)

* [Elastic Agent] Handle 429 response from the server and adjust backoff

When enrolling and the server currently handle to many concurrent
request it will return a 429 status code. The enroll subcommand will
retry to enroll with an exponential backoff. (Init 15sec and max 10mins)

This also adjust the backoff logic in the ACK.

Requires: elastic/kibana#71552

* changelog

* Change values

(cherry picked from commit 2db2152)
@ph ph added v7.9.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jul 14, 2020
ph added a commit that referenced this pull request Jul 14, 2020
#19918) (#19926)

* [Elastic Agent] Handle 429 response from the server and adjust backoff

When enrolling and the server currently handle to many concurrent
request it will return a 429 status code. The enroll subcommand will
retry to enroll with an exponential backoff. (Init 15sec and max 10mins)

This also adjust the backoff logic in the ACK.

Requires: elastic/kibana#71552

* changelog

* Change values

(cherry picked from commit 2db2152)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
elastic#19918)

* [Elastic Agent] Handle 429 response from the server and adjust backoff

When enrolling and the server currently handle to many concurrent
request it will return a 429 status code. The enroll subcommand will
retry to enroll with an exponential backoff. (Init 15sec and max 10mins)

This also adjust the backoff logic in the ACK.

Requires: elastic/kibana#71552

* changelog

* Change values
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants