robots.txt (or sitemap) for downloads.haskell.org

[chris-martin]

Hi, I'm not sure this is the right place to open this issue, but I'm not sure where the code is for downloads.haskell.org, so I hope it's an acceptable discussion place or that somebody can direct me to the proper venue.

The SEO situation for GHC seems a bit hairy at the moment, and I was wondering if just adding a robots.txt to that site could yield some improvement.

Say, for example, you Google for "haskell type families". One of the first results I get is

https://downloads.haskell.org/~ghc/7.8.2/docs/html/users_guide/type-families.html

whereas the better result from haskell.org would be

https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html#ghc-flag--XTypeFamilies

It seems unlikely that one would want to search old versions of the manual, so could we clear this up with a robots file with Disallow entries for the older GHC versions' docs?

[gbaz]

Not a bad idea. @thoughtpolice what do you think?

[gbaz]

also @bgamari

[bgamari]

I agree that our SEO situation is quite bad. Sadly this is not an area where I have a lot of knowledge. Outright disallowing old versions seems like a rather large hammer, but perhaps the benefit to effort ratio makes it worthwhile.

[gbaz]

Would a sitemap be more fine-grained?

…

On Fri, Sep 1, 2017 at 12:27 PM Ben Gamari ***@***.***> wrote: I agree that our SEO situation is quite bad. Sadly this is not an area where I have a lot of knowledge. Outright disallowing old versions seems like a rather large hammer, but perhaps the benefit to effort ratio makes it worthwhile. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#214 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABEt2e8IauKRfgLghRbKPGyiZET1l9Acks5seDBdgaJpZM4ObzBX> .

[bgamari]

gbaz <[email protected]> writes:

Would a sitemap be more fine-grained?

This is actually the first I've heard of sitemaps. It sounds like composing a sitemap would be the right solution here.