Skip to content

Commit

Permalink
add: multiprocess to fix OSX issues (closes #9)
Browse files Browse the repository at this point in the history
  • Loading branch information
metaist committed May 18, 2023
1 parent 54fc09b commit b973929
Show file tree
Hide file tree
Showing 5 changed files with 219 additions and 157 deletions.
2 changes: 1 addition & 1 deletion Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ pytest-cov = "*"
ruff = "*"

[packages]
typing_extensions = "*"
multiprocess = "*"

[requires]
python_version = "3.8"
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ if __name__ == "__main__":

- The main process [creates queues](#create-queues) with `ezq.Q`.

- The main process [creates workers](#create-workers) with `ezq.run` or `ezq.run_thread`.
- The main process [creates workers](#create-workers) with `ezq.run` (alias for `Worker.process`) or `ezq.run_thread` (alias for `Worker.thread`).

- The main process [sends data](#send-data) using `Q.put`.

Expand All @@ -106,7 +106,7 @@ if __name__ == "__main__":

Some more differences:

- **Shared memory**: Each `Process` worker has [data sent to it via `pickle`](#beware-pickle) and it doesn't share data with other workers. By contrast, each `Thread` worker shares its memory with all other workers on the same CPU, so it can [accidentally change global state](#beware-shared-state).
- **Shared memory**: Each `Process` worker has [data sent to it via `pickle`](#beware-pickle) (actually [`dill`](https://github.com/uqfoundation/dill), a `pickle` replacement) and it doesn't share data with other workers. By contrast, each `Thread` worker shares its memory with all other workers on the same CPU, so it can [accidentally change global state](#beware-shared-state).

- **Queue overhead**: `ezq.Q` [has more overhead](#create-queues) for `Process` workers than `Thread` workers.

Expand All @@ -124,11 +124,11 @@ In the main process, create the queues you'll need. Here are my common situation

- **3 queues**: multiple stages of work are happening where workers are reading from one queue and writing to another queue for another worker to process.

**NOTE:** If you're using `Thread` workers, you can save some overhead by passing `thread=True`. This lightweight queue also doesn't use `pickle`, so you can use it to pass hard-to-pickle things (e.g., `lambda`).
**NOTE:** If you're using `Thread` workers, you can save some overhead by passing `Q("thread")`. This lightweight queue also doesn't use `pickle`, so you can use it to pass hard-to-pickle things (e.g., database connection).

```python
q, out = ezq.Q(), ezq.Q() # most common
q2 = ez.Q(thread=True) # only ok for Thread workers
q2 = ez.Q("thread") # only ok for Thread workers
```

## A worker task is just a function
Expand Down Expand Up @@ -157,7 +157,7 @@ Once you've created the workers, you send them data with `Q.put` which creates `

## Beware `pickle`

If you are using `Process` workers, everything passed to the worker (arguments, messages) is first passed to `pickle` by [`multiprocessing`][1]. Anything that cannot be pickled (e.g., `lambda` functions, database connections), cannot be passed to `Process` workers.
If you are using `Process` workers, everything passed to the worker (arguments, messages) is first passed to `pickle` (actually, [`dill`](https://github.com/uqfoundation/dill)). Anything that cannot be pickled with dill (e.g., database connections), cannot be passed to `Process` workers. Note that `dill` _can_ serialize many more types than `pickle` (e.g. `lambda` functions).

## Beware shared state

Expand Down Expand Up @@ -225,7 +225,7 @@ def collatz(q: ezq.Q, out: ezq.Q) -> None:

def main() -> None:
"""Run several threads with a subprocess for printing."""
q, out = ezq.Q(thread=True), ezq.Q()
q, out = ezq.Q("thread"), ezq.Q()
readers = [ezq.run_thread(collatz, q, out) for _ in range(ezq.NUM_THREADS)]
writer = ezq.run(printer, out)

Expand Down
Loading

0 comments on commit b973929

Please sign in to comment.