-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Resolve API Rate limitation #117
Comments
@JafarIronclad Self assign |
GitHub's API has a rate limit of 10 searches and 60 general API calls. We are exploring ways to get a list of issues, project languages on each of these calls, but it requires us to go through those general API calls. We are hoping that we might be able to figure out a good way to increase the rate limit so we can pull the most up to date details on issues and project language. Our project typically returns about ~30 results which means that the rate limit would be burned through on someone's first 2 searches. |
Draft 11/8/2020 12:25 PM: DRAFT 11/8/2020 12:25 PM – Brent Hengeveld - Not a release candidate Dear GitHub, My name is Brent Hengeveld, and I'm contacting you as a member of the open source civic coding community and the Civic Tech Index project (https://github.com/civictechindex/CTI-website-frontend/). We would appreciate your assistance in resolving a technical hurdle we've encountered in our project. We are a team of volunteers building a user-populated online global database of civic tech organizations and projects, with the goal of offering a centralized, unified hub that any civic software or tech developer can refer to in order to find a desired initiative, as well as tags and issues. We are working through an issue where GitHub's API limits on searches (10) and general calls (60) presents a bandwidth hurdle for the end user. You can reference the following files in our repository that perform these API calls: [LIST FILES] We would like the user to be able to view a list of issues and project languages when reviewing a specific project page in our front end, which requires our database to make API calls in order to populate this data for a given project. In our testing, this limit is generally reached within 2 to 3 searches (given an average of 30 results returned per refined search query). At full operational scale, we believe this would cause search returns on project details to be unacceptably out of date for the end user. The most straightforward solution we have would be to request an increase of the search and general API limits for our project. We understand this is a very uncommon and preferential privilege granted to a GitHub project given the key function of API limits (ensuring excessive server load in one repository's does not negatively impact other Github repositories), so we would continually monitor usage and communicate our needs so as to justify the rate increases on a continuous basis. Should this not be permissible, we would appreciate assistance in reaching an alternative, workable solution that allows us to meet our design goals for this key function in our project: searchable, up to date issues and project language tracking detail for a database entry. Here are the other solutions we have attempted thus far, without success: [INSERT PRIOR SOLUTIONS ATTEMPTED WITH OUTCOMES] We appreciate your time and consideration, and are thrilled to employ GitHub's platform to realize our project vision. Please let us know if you have any further questions or require further information. Brent Hengeveld |
Of interest for when we're preparing our communication: Michael Hagger from GH's team responding on a different project, March 8th 2016 on this subject: CocoaPods/CocoaPods#4989 |
My understanding is that for an authorized user, the limits are 1,800 per hour for search API and 5,000 per hour for repos API. I see that in order to get the multiple languages we need to issue another API call https://api.github.com/repos/dyc87112/SpringCloud-Learning/languages Not sure how to find "new issues", maybe with the "since" param. Also probably we should filter on status=open |
todo is to validate what we learned so far and find the documentation that validates things like rate limits. |
https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#authentication |
This call api.github.com/search/repositories?q=topic:civictechindex
|
Check with larger result set (more popular tags). |
This API call api.github.com/search/repositories?q=topic:covid-19 returns Strangely, the same search term on github website https://github.com/search?q=covid-19&ref=opensearch In any case, it seems that up to 6,472 (probably more) results are returned in one shot. We need to check the "Link" header which in this case is saying that we need to invoke the API 34 times to get all the results (which I also guess count as 34 against the limit). Still strange, it says 34 in both cases searching covid19 (which it says to return about 2k results) and covid-19. If I ask for page=35 it says that it only returns 100 results. |
Product needs to have a discussion on search function and what the next steps are. |
Added a test. This URL https://hackforla.github.io/github-api-test/ (search for covid) returns the following headers among which the "Link" which indicates that there are many more pages. |
@giosce I'm of the opinion we should write to GitHub to request for increased API rate limit or at least a methodology to achieve it. |
@giosce will look to see if the languages api and the issue labels api per repo has same limits. See .js file from poc: https://github.com/hackforla/github-api-test/blob/master/index.js issues api would be useful for Use the POC at https://github.com/civictechindex/github-api-test (make changes there to show everything we need for the real CTI website). |
I believe that non-search API calls (like languages and issues) have these limits (I'll finish to double check). https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting |
The following 3 API calls have a 5,000 calls per hour limit Keep in mind that these calls too can return multiple pages for one response. So, let's assume that the search is returning 25 repositories. We also need to double check, I assume that every time a user looks at a contributor we load (or search) all its projects. If it is a search (for example by the organization github_tag) it is in the 30xmin limit, otherwise probably in the 5,000. Finally we'll need to evaluate the typical usage pattern and estimate the number of concurrent users before we can say we are hitting the rate limit. |
Pending implementation with GitHub API to find out if we need to write letter for exception. |
Additional reason we need to resolve Rate limitations:
|
Overview
Increase rate limits of GitHub API via dialogue with GitHub. Current limits of 10 searches and 60 general API calls are quickly reached in CTI normal use cases, which badly restricts frequency of detail updates for users.
Action Items
Resources/Instructions
[Pending]
The text was updated successfully, but these errors were encountered: