How to isolate the individual runs of a hydra multi-run?

[maxfrei750]

Is there a best practice to isolate the individual runs of a hydra multi-run, i.e. that each run starts fresh and is handled as a separate process entirely?

I think the joblib launcher might be a way to achieve this behavior, but maybe there is a simpler way?

Thanks in advance!

[Jasha10]

Hi @maxfrei750,
I think Hydra's suite of plugins is currently lacking in this regard. We would need to develop a new launcher that creates a separate process for each job. I've thought about using Python's multiprocessing module as the backend for such a purpose -- if my understanding is correct, the multiprocessing module would achieve the isolation that you're talking about.

[Jasha10]

Would it be feasible to write a custom launcher that starts each run in a separate process using subprocess.Popen or something similar? Or is that an inherently bad idea?

One requirement on launchers is that they should be able to facilitate communication from the finished job back to the sweeper. Specifically, the job's return value needs to be sent back to the sweeper (wrapped in a JobReturn object).

If the job returns a complex object (e.g. if the main function returns an instance of MyCustomReturnType), it could be hard to send that returned object via a Popen'd pipe. Popen is text-based (i.e. stdout and stderr are IO streams).

[maxfrei750]

I see. Thanks for elaborating! Then I guess that the multiprocessing module is really the easiest way... just like you wrote in your initial reply. 😅 Hence, I think I should finally formally accept your answer and the issue you opened (#2187) is a better place for further discussion. Thank you very much!

[IlyasMoutawwakil]

@maxfrei750 @Jasha10, first of all thank you both for your work on this topic.
what makes the two runs not perfectly isolated ? I followed your suggestion:

hydra:
  launcher:
    n_jobs: 2
    prefer: processes # use processes instead of threads
    backend: multiprocessing # use multiprocessing backend instead of loky
  sweeper:
    max_batch_size: 1

I'm trying to achieve perfect isolation between torch or onnxruntime benchmarks.
It seems that the two runs have different PIDs, doesn't that guarantee isolation ?

[romesco]

I solved the pickling problem with the aforementioned link, so that the joblib launcher now works. So this brings us back to the original topic of this question: run isolation 🥳

Hey @maxfrei750, what was the solution to solving the "failed to un-serialize" issue? How did you temporarily swap the system path problem in when the deserialization happens via the hydra->joblib pipeline?

[maxfrei750]

@romesco I'm not sure, if the solution to my problem can be generalized, since it was closely related to Blender's bpy module and the way it interacts with the system path (see TylerGubala/blenderpy#23 (comment), if you haven't done that already). If I remember correctly (it's been a while), I simply saved and restored the system sys.path variable like this:

ORIGINAL_SYS_PATH = list(sys.path)

import bpy

sys.path = ORIGINAL_SYS_PATH

And the problem was solved. But once again: This is probably very specific to the bpy library.