Block ahrefs bot in robots.txt #408
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This bot crawls us very very regularly (sometimes more than once a day) and causes spikes in our load and exceptions. It seems to serve no useful purpose to us, in that it's not a search engine that users would use. > 1. What is ahrefs.com > Ahrefs.com is an independent tool for SEO analysis with a wide range > of features. It is designed, first of all, for SEO specialists and > site owners but may be of interest to other concerned Internet > researchers. > https://ahrefs.com/faqs.php This change to the robots.txt is the recommended way to stop the bot crawling. See https://ahrefs.com/robot
|
In the spirit of remaining mostly open, should we set an explicit (and long, 10 seconds or so) crawl delay instead? Their site says they support it. |
bradwright
added a commit
that referenced
this pull request
May 30, 2014
Block ahrefs bot in robots.txt
alexmuller
added a commit
that referenced
this pull request
Jan 7, 2015
@bradleywright [noted][1] in #408 that it would be nice if we tried to remain mostly open. This commit sets a long Crawl-delay which will allow the bot to crawl us without impacting the site for actual users. If this causes problems for users we should revert it. [1]: #408 (comment)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This bot crawls us very very regularly (sometimes more than once a day) and
causes spikes in our load and exceptions. It seems to serve no useful purpose
to us, in that it's not a search engine that users would use.
This change to the robots.txt is the recommended way to stop the bot crawling.
See https://ahrefs.com/robot