Ensure FAQ is indexed by search engines

[simonw]

A frustrating thing about GitHub repositories is that GitHub have https://github.com/robots.txt configure such that only the contents of README.md can be crawled by search engines such as Google.

All other files in the repo - including FAQ.md - are blocked from being crawled!

See here:

• Does GitHub deliberately block indexing of code in robots.txt? community/community#20958

The two potential solutions I can see for this are:

• Include configuration for publishing a copy of the FAQ using GitHub Pages, which can be indexed by search engine crawlers
• Move the built FAQ content into the README itself

[pradyunsg]

Is including a link to the FAQ from the README sufficient?

All other files in the repo - including FAQ.md - are blocked from being crawled!

My reading is that tree pages are blocked, so file listings but not the individual files? If the search engine can get to the page, it’d index it — that’s my mental model at least.

[willmcgugan]

You can configure the FAQ.md to go where you like. For Textual I was going to put it in the docs directory, so it gets built with GitHub pages.

It would also be possible to add some kind of marker that inserted the FAQ in to the README. Although TBH I'm not sure I'd want to go to that effort. Hoping the community contributes such features!