From 7a53edd17874f63275fd356565e6d59ef3acc5f0 Mon Sep 17 00:00:00 2001 From: Alex Muller Date: Wed, 7 Jan 2015 17:13:30 +0000 Subject: [PATCH 1/2] Allow AhrefsBot to crawl very slowly @bradleywright [noted][1] in #408 that it would be nice if we tried to remain mostly open. This commit sets a long Crawl-delay which will allow the bot to crawl us without impacting the site for actual users. If this causes problems for users we should revert it. [1]: https://github.com/alphagov/static/pull/408#issuecomment-44185309 --- public/robots.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/public/robots.txt b/public/robots.txt index 36542f0cc..55f7b0ef6 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -16,5 +16,6 @@ Disallow: /info/* Sitemap: https://www.gov.uk/sitemap.xml Crawl-delay: 0.5 -user-agent: AhrefsBot -disallow: / +# https://ahrefs.com/robot/ crawls the site frequently +User-agent: AhrefsBot +Crawl-delay: 10 From 771d0ae2898fa7db23de33c06c39282344045259 Mon Sep 17 00:00:00 2001 From: Alex Muller Date: Wed, 7 Jan 2015 17:34:04 +0000 Subject: [PATCH 2/2] Remove unnecessary path from robots.txt This path is no longer used, so there's no need to prevent robots from crawling it. --- public/robots.txt | 2 -- 1 file changed, 2 deletions(-) diff --git a/public/robots.txt b/public/robots.txt index 55f7b0ef6..5aa9c49c4 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -9,8 +9,6 @@ Allow: /licence-finder Disallow: /business-finance-support-finder/* Allow: /business-finance-support-finder Disallow: /apply-for-a-licence -# Don't allow indexing of experimental performance platform dashboards -Disallow: /performance/experimental/* # Don't allow indexing of user needs pages Disallow: /info/* Sitemap: https://www.gov.uk/sitemap.xml