Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add: map support #3

Closed
metaist opened this issue May 5, 2023 · 7 comments
Closed

add: map support #3

metaist opened this issue May 5, 2023 · 7 comments

Comments

@metaist
Copy link
Owner

metaist commented May 5, 2023

It is kinda nice to be able to simply say "run this function on these inputs using parallel resources".

@metaist
Copy link
Owner Author

metaist commented May 5, 2023

Ok, what would the syntax for this look like?

result = ezq.map(f, items)

But what about additional args? Also, blowing away the builtin map function seems like a bad idea.

@metaist
Copy link
Owner Author

metaist commented May 5, 2023

We'll pick a better name later. And for now, let's say we don't care too much if there's no way to add additional args. How does this work? Oh, and we're basically re-inventing concurrent.futures at this point.

  • create input, output queues
  • create the appropriate thread / processes
  • put each item from the values into the input queue
  • end the queue and wait
  • return the result of the output queue

@metaist
Copy link
Owner Author

metaist commented May 5, 2023

That second step implies we have a few more args:

  • number of workers
  • thread or process

@metaist
Copy link
Owner Author

metaist commented May 8, 2023

Still unsure about the name. I might go with apply. In the meantime, the actual implementation is pretty straightforward.

@metaist
Copy link
Owner Author

metaist commented May 8, 2023

mypy is yelling about the return type for this function. How do you annotate a generator?

@metaist
Copy link
Owner Author

metaist commented May 8, 2023

Ah, Iterator (StackOverflow). Generator is probably more correct except that I'm still supporting python 3.8, so until that reaches end of life (did it already?), I want to be maximally compatible.

@metaist
Copy link
Owner Author

metaist commented May 9, 2023

Me: I'm debating renaming ezq.map to ezq.apply. What are reasons for and against this name change?

ChatGPT: Reasons for renaming ezq.map to ezq.apply:

  1. Familiarity: The name "apply" may be more familiar to some users, as it is used in other popular libraries like Pandas, where it is used to apply a function along an axis of a DataFrame. This familiarity can make the library more intuitive to use for those already familiar with other similar libraries.

  2. Semantics: In functional programming, "apply" is a term used for applying a function to its arguments, which is essentially what the ezq.map function is doing in a parallel manner. The term "apply" might be more descriptive of this process than "map," which is often associated with applying a function element-wise to a single iterable.

Reasons against renaming ezq.map to ezq.apply:

  1. Consistency with Python built-ins: The map function is a built-in Python function that applies a function to all items in an input list. By naming the function ezq.map, you maintain consistency with the built-in Python function, which might make it easier for users to understand its purpose.

  2. Familiarity with other parallel processing libraries: Libraries like concurrent.futures and multiprocessing use the term "map" for their parallelized version of the built-in map function (e.g., Executor.map() and Pool.map()). Keeping the name ezq.map would maintain consistency with other parallel processing libraries in Python, making it more intuitive for users who have experience with these libraries.

In conclusion, the decision to rename ezq.map to ezq.apply depends on the goals and target audience of your library. If you want to maintain consistency with built-in Python functions and other parallel processing libraries, keeping the name ezq.map might be better. On the other hand, if you believe that the term "apply" is more familiar and better conveys the purpose of the function, renaming it to ezq.apply could be a good choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant