GitHub badges are "unresponsive": rate limiting

[ducin]

This is what I see at shields.io now:

github shields seem not to work. I've tried to open them in a separate browser tab - the same result. But the owner/repo URL is correct.

[espadrine]

Thank you for raising this issue!

I have bad news. GitHub's rate limiting has two ceilings:

• 60 requests/hour for unauthenticated requests
• 5000 request/hour for authenticated requests.

We are using authenticated requests. Indeed, the headers I receive from GitHub include the following:

Status: 403 Forbidden
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 0

However, the rate limit reset timestamp does not seem to be correct, which suggests a bug in GitHub's rate limiting system. I am writing to them for clarification.

In the meantime, I will try to adjust the cache to avoid hitting the rate limit. If that is not enough, I will try to cache GitHub badges for a longer span of time. Ideally though, I would find a common ground with GitHub to increase the rate limit.

[RIAEvangelist]

I see the issue today as well. Any word back from them?

If not, I have an idea to allow additional requests : spin up multtple authenticated "apps" and round robin the requests through them all using the a shared cache on the back end.

An additional solution would be to have a backup cache every time the primary cache expired it would replace the entry in the long term cache (1 week?). If a request was made and rejected there would be a high likelyhood that stale info was available unless the package/repo was not very popular.

[espadrine]

GitHub has told me they have opened an internal issue for the API team to investigate and track this.

Having multiple apps fails, because they track IPs as well. Switching the server's IP would be wonky.

When we hit rate-limiting, we do use our LRU cache (when there is something in it). But the requests are very varied, and the cache gets filled within an hour. (Potentially, we could have a smarter cache than LRU, but tracking popularity sounds like it would require a lot of memory.)

[arcanedev-maroc]

Same here, i've got [vendor|unresponsive] for Github issues/PR badges.

https://github.com/ARCANEDEV/Breadcrumbs

[ajermakovics]

Hi all,
Perhaps Shields could serve a blank badge in these cases. Unresponsive is more informative but doesn't look that great in a project's readme. Just a thought

[vbauer]

@ajermakovics 👍

[martincostello]

@ajermakovics 👍

[kimwalisch]

This would be a very good workaround for the time being. And please put some pressure on GitHub, why haven't they come back to you in 2 months' time?!

[lilyball]

I don't even get 'vendor | unresponsive". I get a broken image when I look at shields.io, and when I try and load the badge URL directly, I get a CloudFlare "Error 504 Gateway time-out" page.

[espadrine]

@kballard Hmm. I added the 504 because of this discussion. The hope was to configure CloudFlare to cache everything, but only use that cache if I returned a 504. However, I can't find a way to configure CloudFlare in this way. I didn't think it would cause CloudFlare to return an HTML page instead.

[davisonio]

Any news on this?

[martinpaljak]

I wish Github provided itself such badges. People obviously like them.

[calidion]

+1
@martinpaljak

[DamienDennehy]

+1. I've had to remove my latest release shield and replace it with a static svg because GitHub shields were down so often in the last week.

[brunchboy]

Yeah, this is a huge bummer. The Github-related badges on my Github-hosted documentation essentially never work. I may have to just give up on them, but they provided nice information in a pretty way when they worked. Is there a forum where raising this issue might help motivate Github to fix things?

[martincostello]

I just opened up GitHub Desktop and received this error message. Looks like the token I added has clobbered my entire API rate allowance. I was under the impression the limit was per-token/per-application. If not I'm afraid I'm going to have to revoke the token.

[brunchboy]

I was wondering if that might be an issue. If we can get enough people to contribute tokens, this should never happen though.

[espadrine]

I am awfully sorry, I substantially misinterpreted how the tokens work. I am working on a fix, don't hesitate to revoke the tokens in the meantime.

[martincostello]

No problem. I'll create a new one once the fix is deployed.

[espadrine]

There's a partial fix up. Shields no longer uses a token if it has reached 1/4th of its rate limit.

[espadrine]

The full fix is up. Shields will only use a user token that has more than 1/4th of its rate limit and is the user token with the most requests left. As a result, it self-balances between all user tokens available.

[brunchboy]

That’s a nice, elegant solution. And ¼ is conservative enough that nobody should be afraid to share their tokens.

[espadrine]

Thanks a lot for raising the issue!

I can probably raise the "1/4th of the rate limit" restriction. It corresponds to 3125 requests in an hour, which is a bit low (for tinkerers). However, that limit is only there as a safeguard. Currently, with just nine tokens, the lowest a token went in the past hour was 10288 requests, before getting reset to 12500.

[martincostello]

I've added a new token back into the app.

[espadrine]

Hi everyone,

I made a mistake.

@agboom pointed out to me in private that the GitHub user tokens were publicly accessible at https://img.shields.io/.github-user-tokens.json. (Obviously, it is no longer the case.)

The fundamental problem is that, when I first created the service, there was no sensitive information in it, and I didn't expect to ever have some. So I simply set the repository directory as the fallback location wherein the server could serve files, and forgot that I did. That was a mistake in retrospect, as when I first hit the rate limit problems a year or two later, I simply assumed that files in the main folder would not be publicly accessible.

First, correction and mitigation. I set a specific folder to contain public files, so that the server is not allowed to serve files outside of that folder. I set a specific folder to contain sensitive files, in private/.

Second, assuming the worst. The two sensitive files that we currently have are .github-user-tokens.json and secret.json. They are both very similar: they contain information that prove that shields.io is the one performing a request, and therefore, that it can be trusted with a higher rate limit. (I am sorry to have been sloppy with that trust.)

Malicious actors can have downloaded those files, and benefited from higher rate limits in Bintray, SL Insight and GitHub. For Bintray and SL Insight, anyone can obtain their own tokens to get that same rate limit by registering as a user at those companies, so there is no practical reason for them to use mine: if they wanted to be malicious, they could create fake accounts. For GitHub, it is a bit worse, as about 900 GitHub users registered to help increase the rate limit for GitHub badges, creating a rate limit of 11250000 (~ 11 million) requests per hour: virtually unlimited access to the public GitHub API. For GitHub's sake, I cannot keep using them given that they are potentially compromised, potentially giving some malicious actor a way to circumvent GitHub's rate limit mechanism, so I must revoke them.

So here is what I did:

• I regenerated all secrets,
• I revoked all GitHub user tokens for the Shields app.

Unfortunately, at peak time, I do need about 200k requests/hour. It is way below the 11 million, but way above the 12.5k that I can get from just me. I need about 20 people to re-activate the Shields app on GitHub, which may not happen in time for Monday 5pm CET, which is a typical peak time.

For users, including those that authorized Shields as a GitHub app, there was no security issue, as I originally set the app to require the minimal amount of permissions (it can only see what is already public). There is a potential denial of service issue, as a malicious entity could have used up all of your rate limit capacity, which would have prevented you from using, for instance, GitHub for Windows, up until the end of the hour.

For Shields.io, the revocation of GitHub user tokens means that GitHub badges may start failing at the end of every hour at peak time on Monday, unless about 20 people re-activate the Shields app by going to this page.

I am awfully sorry for this, and will do my best to be more worthy of your trust in the future.

[martincostello]

@espadrine Thanks for the explanation and actions you've token once you discovered the problem. I have added a new token to the service for use.

[davisonio]

Thank you for letting us know this.

Added new token!

[mathiasbynens]

Kudos for being so transparent about this, @espadrine! You have my sword. And my bow. And my token.

[agboom]

Well mitigated @espadrine. Please accept my token as a token of gratitude.

[niemyjski]

Is this why https://img.shields.io/badge/donorbox-donate-blue.svg fails to load

[niemyjski]

Could you add a section to the top of the readme about what this is and how to donate your client token (even add it to the top of shields.io?

[davisonio]

@espadrine did you get the 20 required signups? If not I can tweet or something to help.

[espadrine]

There thankfully were enough tokens. However, the volume of requests at peak time gets fairly high now, and I will need to add a server and do some performance tuning. I talk about that here: #868.

[Daniel15]

Can this issue be closed? It sounds like everything is going well with the GitHub badges 😃

[brunchboy]

As far as I can tell!

[Daniel15]

Great! Let's close it out and reopen if any issues come up 😄

[ghybs]

Hi,

Thank you for having pro-actively revoked our tokens.

Just to let you know that the page to (re-)authorize GitHub Shields App no longer works:
https://img.shields.io/github-auth

I assume you currently have enough tokens for the rate limit?
Please let us know should you need more tokens.

[mitchellkrogza]

Just ran into this issue this morning using some new badges for my repo. Tried visiting https://img.shields.io/github-auth but only get GitHub OAuth authentication failed to provide a code.

[paulmelnikow]

Thanks, let's address at #1038.

Let's open a new issue for related Github problems, so it's not buried in here.