Running stack templates bumps into Github API limit

[bitemyapp]

Failed to download templates. The HTTP error was:

StatusCodeException (Status {statusCode = 403, statusMessage = "Forbidden"}) [("Server","GitHub.com"),("Date","Wed, 28 Oct 2015 23:39:51 GMT"),("Content-Type","application/json; charset=utf-8"),("Transfer-Encoding","chunked"),("Status","403 Forbidden"),("X-RateLimit-Limit","60"),("X-RateLimit-Remaining","0"),("X-RateLimit-Reset","1446077019"),("X-GitHub-Media-Type","github.v3; format=json"),("X-XSS-Protection","1; mode=block"),("X-Frame-Options","deny"),("Content-Security-Policy","default-src 'none'"),("Access-Control-Allow-Credentials","true"),("Access-Control-Expose-Headers","ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval"),("Access-Control-Allow-Origin","*"),("Strict-Transport-Security","max-age=31536000; includeSubdomains; preload"),("X-Content-Type-Options","nosniff"),("Content-Encoding","gzip"),("X-GitHub-Request-Id","9D82BA36:146C7:57FF7EF:56315CC7"),("X-Response-Body-Start","{\"message\":\"API rate limit exceeded for 157.130.186.54. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)\",\"documentation_url\":\"https://developer.github.com/v3/#rate-limiting\"}"),("X-Request-URL","GET https://api.github.com:443/repos/commercialhaskell/stack-templates/contents/")] (CJ {expose = []})

Comes by way of @stephenjbarr

I don't know how Github limits API requests here. I imagined it is partly keyed by the resource being accessed, but I don't know what else factors in (IP address?)

Does anyone know the usual way to cope with this without obligating the user to sign up for credentials?

[borsboom]

Based on the info at https://developer.github.com/v3/#rate-limiting, looks like the rate limit for unauthenticated users is 60 requests per hour, keyed by the client IP address (and not the resource being accessed). Probably wouldn't be an issue for most people, but if you're sharing a proxy or NAT with others also using the Github API it would be easy to exceed. Probably would be better to have some other way to get a list of templates. Adding a file containing a list of templates is probably easiest, but annoying to have to maintain. @chrisdone, since you added the feature, any thoughts?

[borsboom]

BTW, @bitemyapp, being authenticated with Github in the web browser shouldn't make any difference, since Stack doesn't pass any authentication info along with the API request.

[stephenjbarr]

On the plus side, if this is by IP address, I need to track down the other Haskellers in my building.

[borsboom]

@stephenjbarr The others wouldn't necessarily be Haskellers, just users of the Github API.

[stephenjbarr]

Oh I see. This happened from a large co-working space in Seattle so I can see this happening. I know I wasn't making requests at this rate.

[phadej]

Generating token by following https://help.github.com/articles/creating-an-access-token-for-command-line-use/ and teaching it to stack might be a solution, but it's not very user friendly :(

Alternatively there is an oauth authorisation api by I don't have any experience with it.

[borsboom]

Yeah, it's a lot to do for one minor feature. I think just not using the Github API would be quite a bit easier.

[mgsloan]

Revisiting this due to this PR. I'm thinking that the following would be a good way to solve this: #1595

This issue can be closed once #1595 is resolved, so marking it "blocked".

[dpwiz]

Same with stack upgrade. Is it possible to add a fallback URL or something?

[kubek2k]

Same happened for me when running stack clean on the url: https://api.github.com/repos/commercialhaskell/stack/releases/tags/v1.9.3. The result is indeed

{
message: "API rate limit exceeded for X.X.X.X (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
documentation_url: "https://developer.github.com/v3/#rate-limiting"
}

maybe it makes sense to allow people to authenticate requests using credentials provided in predefined env variables. (I am following the hub tool conventions here).
For basic auth:

• GITHUB_USER
• GITHUB_PASSWORD

and for personal token:

• GITHUB_TOKEN.

The place in the code I would apply this is here:

stack/src/Network/HTTP/Download.hs

Line 119 in 99ecb78

setGithubHeaders :: Request -> Request

.

This in theory should raise the limits to the personal ones, as stated here: https://developer.github.com/v3/#rate-limiting

[dbaynard]

Same happened for me when running stack clean on the url: https://api.github.com/repos/commercialhaskell/stack/releases/tags/v1.9.3.

@kubek2k would you explain what you mean by this?

(Note: stack templates has changed since the previous comment. See #4484)

[kubek2k]

@dbaynard sorry for not being perfectly clear. While running stack clean it stopped while trying to retrieve release information from above url, because the api rate limit was hit for the IP I was behind that day. I think the issue is the same, and it would be nice to optionally let users authenticate requests in some way.

[snoyberg]

This command no longer makes Github API calls and should no longer hit the rate limit, closing.