-
-
Notifications
You must be signed in to change notification settings - Fork 30.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-74028: concurrent.futures.Executor.map: add buffersize
param for lazy behavior
#125663
base: main
Are you sure you want to change the base?
Conversation
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
1 similar comment
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
6a58c7d
to
21f7b8d
Compare
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
Thanks for the PR. First, I think this is a big behavior change for Executor. I think we need to discuss it in the https://discuss.python.org/ first. In my personal opinion, I think this is not a good choice to add the
|
9eef605
to
e5c867a
Compare
Hi @Zheaoli, thank you for your comment!
You mean big alternative behavior, right? (the default behavior when ommitting
Fair, I will start a thread there and ping you.
I'm not sure to get it, could you detail that point? 🙏🏻
You are completely right, makes more sense! I have fixed that (commit) |
8bf7be7
to
769060e
Compare
For me, the basic |
Hi @Zheaoli
There may be a misunderstanding here, the goal of this PR is precisely to make I will recap the behaviors so that everybody is on the same page: built-in
|
769060e
to
be419ed
Compare
hey @rruuaanng, fyi I have applied your requested changes regarding the integration of unit tests into existing class 🙏🏻 |
@@ -594,10 +599,21 @@ def map(self, fn, *iterables, timeout=None, chunksize=1): | |||
before the given timeout. | |||
Exception: If fn(*args) raises for any values. | |||
""" | |||
if buffersize is not None and buffersize < 1: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does it have to be None?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it has to be None OR to be greater than 0, would the addition of either make it clearer ? -> ValueError("buffersize must be either None or >= 1.")
args_iter = iter(zip(*iterables)) | ||
if buffersize: | ||
fs = collections.deque( | ||
self.submit(fn, *args) for args in islice(args_iter, buffersize) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't buffersize empty? Can you introduce it? (Forgive me for not understanding it).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
absolutely np, thank you for taking the time to review my proposal. To be sure to understand the question well, what do you mean by "Isn't buffersize empty?"
e28a0f0
to
bb0e747
Compare
Context recap (#74028)
concurrent.futures.Executor.map
is not lazy:fn
calls an external service that you don't want to overload.Proposal: add
buffersize
paramSimilar to the work of @graingert in #18566 and @Jason-Y-Z in #114975, i.e. use a queue of fixed size to hold the not-yet-yielded future results, and only iterating on input iterables if the queue is not full.
In addition this PR:
buffersize=None
timeout
andbuffersize
at the same time📚 Documentation preview 📚: https://cpython-previews--125663.org.readthedocs.build/