Feature/reduce spidering #111
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Blocks all bots instead of just named bots, and blocks thumbnails since they're all dynamically generated and bots can really slow the site down
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Simplifies the robots.txt rules and adds an index to disallow spidering of /images/tiles. In chronam core, /images/tiles isn't the only URL that generates tiles (see ticket #109 ), but I believe it is the primary URL for OpenSeadragon.
Unfortunately, any sites still using the old URL style (
/lccn/snxxxxxxxxx/YYYY-MM-DD/ed-#/seq-#/image_514x514_from_1022,4094_to_2050,5122.jpg) won't benefit from this, but addressing #109, and having users update local themes, would allow this to work for everybody. (Or extra rules could be added to catch this style of URL, too)It might seem strange to disallow bots from hitting the tiles, but Apache log analysis showed us that we were getting a huge amount of traffic from bots - including thumbnails and the dynamic tiles.