feat: improve /sites endpoint performance
#90
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
December 8, 2020 17:43
Previously, site retrieval was slow because the GitHub API for retrieving org repos was paginated, and we retrieved the data sequentially, one page at a time. This meant that it often took up to 7 or even 8 seconds each time this endpoint is accessed (each page took around 3 seconds, perhaps due to the large amount of data being sent). This commit improves performance by making these api calls concurrently, so that it now only takes around 3 seconds for the endpoint to respond. This commit also introduces an optional env var, ISOMERPAGES_REPO_PAGE_COUNT, which determines how many pages of the GitHub API to comb simultaneously. Since we know the number of repos our github org has, we can use this info to speed up our endpoint by making concurrent calls instead of stepping through the API pagination.
kwajiehao
force-pushed
the
fix/sites-performance
branch
from
e95aedd to
b0ac398
Compare
Merged
harishv7
pushed a commit
that referenced
this pull request
Feb 17, 2023
* feat: speed site retrieval up by making concurrent api calls Previously, site retrieval was slow because the GitHub API for retrieving org repos was paginated, and we retrieved the data sequentially, one page at a time. This meant that it often took up to 7 or even 8 seconds each time this endpoint is accessed (each page took around 3 seconds, perhaps due to the large amount of data being sent). This commit improves performance by making these api calls concurrently, so that it now only takes around 3 seconds for the endpoint to respond. This commit also introduces an optional env var, ISOMERPAGES_REPO_PAGE_COUNT, which determines how many pages of the GitHub API to comb simultaneously. Since we know the number of repos our github org has, we can use this info to speed up our endpoint by making concurrent calls instead of stepping through the API pagination. * refactor: remove unnecessary filter Co-authored-by: Jie Hao Kwa <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR addresses issue #83
Previously, site retrieval was slow because the GitHub API for retrieving org repos was paginated, and we retrieved the data sequentially, one page at a time. This meant that it often took up to 7 or even 8 seconds each time this endpoint is accessed (each page took around 3 seconds, perhaps due to the large amount of data being sent).
Since we know the number of repos our
isomerpagesGitHub organization has, we can use this information to speed up our endpoint by making concurrent calls instead of stepping through the API pagination. This PR improves performance by making concurrent API calls to retrieve repo info so that it now only takes around 3 seconds for the endpoint to respond, more than halving the amount of time take previously.This commit also introduces an optional env var,
ISOMERPAGES_REPO_PAGE_COUNT, which determines how many pages of the GitHub API to comb simultaneously.