Package robots implements robots.txt file parsing based on Google's specification:
https://developers.google.com/search/reference/robots_txt
For installation, usage and description, please see the documentation:
https://godoc.org/github.com/benjaminestes/robots
MIT