Add documentation and example for using a ProcessPoolExecutor

[mrocklin]

We recently added the ability to have multiple executors and annotate tasks to use those executors. A common situation where this would be useful is in farming out tasks to separate processes. This would be a good example to build.

One motivating use case is #5051

[jrbourbeau]

+1 it'd be great to see an example for this. For anyone who is interested, #4869 is where the multiple executors functionality was added. Additionally, here is a test which was added

distributed/distributed/tests/test_worker.py

Lines 2007 to 2028 in 5f01fe6

	@pytest.mark.asyncio
	async def test_multiple_executors(cleanup):
	def get_thread_name():
	return threading.current_thread().name
	async with Scheduler() as s:
	async with Worker(
	s.address,
	nthreads=2,
	executor={
	"GPU": ThreadPoolExecutor(1, thread_name_prefix="Dask-GPU-Threads")
	},
	) as w:
	async with Client(s.address, asynchronous=True) as c:
	futures = []
	with dask.annotate(executor="default"):
	futures.append(c.submit(get_thread_name, pure=False))
	with dask.annotate(executor="GPU"):
	futures.append(c.submit(get_thread_name, pure=False))
	default_result, gpu_result = await c.gather(futures)
	assert "Dask-Default-Threads" in default_result
	assert "Dask-GPU-Threads" in gpu_result

which would be a good starting point for an example in the docs

[mrocklin]

Also examples.dask.org

…

On Wed, Jul 14, 2021 at 10:40 AM James Bourbeau ***@***.***> wrote: +1 it'd be great to see an example for this. For anyone who is interested, #4869 <#4869> is where the multiple executors functionality was added. Additionally, here is a test which was added https://github.com/dask/distributed/blob/5f01fe6010a0a3621956920d0260ac5f0111f17d/distributed/tests/test_worker.py#L2007-L2028 which would be a good starting point for an example in the docs — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5059 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTCTLJKF6AE35UGE7ZDTXWVYFANCNFSM5ALWAHIA> .

[mrocklin]

I also suspect that we may have to use something with better serialization support, like loky

…

On Wed, Jul 14, 2021 at 10:44 AM Matthew Rocklin ***@***.***> wrote: Also examples.dask.org On Wed, Jul 14, 2021 at 10:40 AM James Bourbeau ***@***.***> wrote: > +1 it'd be great to see an example for this. For anyone who is > interested, #4869 <#4869> is > where the multiple executors functionality was added. Additionally, here is > a test which was added > > > https://github.com/dask/distributed/blob/5f01fe6010a0a3621956920d0260ac5f0111f17d/distributed/tests/test_worker.py#L2007-L2028 > > which would be a good starting point for an example in the docs > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#5059 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AACKZTCTLJKF6AE35UGE7ZDTXWVYFANCNFSM5ALWAHIA> > . >

[mrocklin]

I have a video for this https://www.youtube.com/watch?v=vF2VItVU5zg

This isn't yet publicized. I plan to wait until after the next release. Probably we should repurpose this into an example on examples.dask.org as well.

[mrocklin]

https://gist.github.com/mrocklin/0c01c5427358c167a89d5bd34582cd30

[pavithraes]

I'd love to work on this!