Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Worker should follow instructions in HTTP 429 responses #121

Open
MichaelRoeder opened this issue Sep 20, 2019 · 1 comment
Open

Worker should follow instructions in HTTP 429 responses #121

MichaelRoeder opened this issue Sep 20, 2019 · 1 comment
Labels
component:worker priority:low low priority issue type:enhancement

Comments

@MichaelRoeder
Copy link
Member

Description

As a website owner, I expect crawlers to behave. If they are crawling too fast and my server responds with HTTP 429 (RFC 6585), the crawler should follow the instructions and slow down.
@MichaelRoeder
Copy link
Member Author

MichaelRoeder commented Oct 26, 2020 •
edited
Loading

Note: If Web API returns status code 429, it means that you have sent too many requests. When this happens, check the Retry-After header, where you will see a number displayed. This is the number of seconds that you need to wait, before you try your request again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component:worker priority:low low priority issue type:enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@MichaelRoeder