Run Repo Requests in Parallel on ProjectDetail Pages #1562
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghickman
force-pushed
the
optimise-project-detail
branch
2 times, most recently
from
dc2543c to
5897d25
Compare
|
@bloodearnest – is this good to go now? |
bloodearnest
approved these changes
Jan 28, 2022
jobserver/views/projects.py
Outdated
| "name": f.path.segments[1], | ||
| "url": url, | ||
| "is_private": is_private, | ||
| } | ||
|
|
||
| futures = [repo_thread_pool.submit(get_repo, url=url) for url in repo_urls] | ||
| for future in concurrent.futures.as_completed(futures): |
There was a problem hiding this comment.
So, I think we can slightly simplify this using Executor.map(). We can also set an explicit timeout for all requests too, which is usually a good idea when doing concurrency.
So, maybe replace the last 3 lines with:
# use the threadpool to parallelise the repo requests
yield from repo_thread_pool.map(get_repo, repo_urls, timeout=30)There was a problem hiding this comment.
map basically does what you did, but handles cancellation and timeouts: https://github.com/python/cpython/blob/main/Lib/concurrent/futures/_base.py#L571
There was a problem hiding this comment.
Much nicer, and TIL!
On ProjectDetail pages with several repos the lookup for private/public status was noticably slowing the page down. This switches to ThreadPoolExecutor to run through them in parallel (mostly).
ghickman
force-pushed
the
optimise-project-detail
branch
from
5897d25 to
f481e66
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On ProjectDetail pages with several repos the lookup for private/public
status was noticably slowing the page down. This switches to
ThreadPoolExecutor to run through them in parallel (mostly).