Allow using dill for pickling

[ddelange]

Hello 👋

This PR does not change any behaviour of the library, apart from when the multiprocess module is installed in the current environment (installable with pip install multiprocess or pip install aioprocessing[dill]).

dill allows for pickling locally defined functions, lambdas, and all sorts of other exotic cases where stdlib will complain.

In this gist, I solved a case of PicklingError: Can't pickle <function test4 at 0x118bb4940>: it's not the same object as __main__.test4 by monkey patching aioprocessing, and then thought 'why not PR' :)

If you have reasons to constitutionally dislike this PR, feel free to close it, otherwise I'm happy to take on suggestions!

[ddelange]

sub-optimal to skip this test... especially for people that are using concurrent.futures.ProcessPoolExecutor in combination with aioprocessing, in an environment where multiprocess happens to be installed. As the author stated in uqfoundation/pathos#90, it should be an easy PR. I've tried figuring it out multiple times as I keep running into multiprocessing problems in async contexts, but repeatedly stranded.

Here's the traceback for this test case, caused by ProcessPoolExecutor internally using multiprocessing instead of multiprocess:

test_executor (tests.queue_test.ManagerQueueTest) ... Process SpawnProcess-1:
Traceback (most recent call last):
  File "~/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "~/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "~/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/process.py", line 233, in _process_worker
    call_item = call_queue.get(block=True)
  File "~/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "~/.pyenv/versions/pi/lib/python3.8/site-packages/multiprocess/managers.py", line 959, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "~/.pyenv/versions/pi/lib/python3.8/site-packages/multiprocess/managers.py", line 809, in __init__
    self._incref()
  File "~/.pyenv/versions/pi/lib/python3.8/site-packages/multiprocess/managers.py", line 863, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "~/.pyenv/versions/pi/lib/python3.8/site-packages/multiprocess/connection.py", line 511, in Client
    answer_challenge(c, authkey)
  File "~/.pyenv/versions/pi/lib/python3.8/site-packages/multiprocess/connection.py", line 762, in answer_challenge
    raise AuthenticationError('digest sent was rejected')
multiprocess.context.AuthenticationError: digest sent was rejected

[ddelange]

note that even if uqfoundation would add compatibility, and expose pathos.futures.ProcessPoolExecutor or so, this error would still persist for people trying to combine the concurrent.futures.ProcessPoolExecutor with aioprocessing[dill].

[dano]

I think there should be a note in the readme about this incompatibility when using dill, as well as documenting the workaround that forces usage of multiprocessing. People upgrading to this version could suddenly have their code break, so it should be very clear how to fix it.

[ddelange]

Sure thing, added. Assuming semver and this being a potentially breaking change ^ I went ahead and bumped to 2.0.0 🤙

[dano]

I like the idea - my main concern is that there's no way to opt out of using dill when you have it installed. I'd be willing to take this if you add a simple way for a user to import multiprocessing-backed versions of all the aioprocessing objects, even when dill is importable.

[ddelange]

how about an env var AIOPROCESSING_DILL_DISABLED=True which we check for upon import?

[dano]

These imports should still come from threading. They're getting proxied, so we don't need the multiprocessing versions.

[dano]

I think there should be a note in the readme about this incompatibility when using dill, as well as documenting the workaround that forces usage of multiprocessing. People upgrading to this version could suddenly have their code break, so it should be very clear how to fix it.

[dano]

how about an env var AIOPROCESSING_DILL_DISABLED=True which we check for upon import?

Yes, I think that is fine.

[dano]

Thanks for the contribution, @ddelange!

[ddelange]

Hi @dano, are there any blockers for the 2.0.0 release? Are there other things you wanna do/I can help with before releasing?

[dano]

@ddelange Just inertia 🙂 I will try to do it sometime today or this weekend.

[dano]

@ddelange Done!