Allow Manager instances to be passed between processes

[robobenklein]

Is your feature request related to a problem? Please describe.
I would like a subprocess to be able to update a counter while there are other counters managed by the main process.

Right now a manager can't be shared because it cannot share the TextIOWrapper that is stdout/stderr, so new managers are writing over eachother.

Describe the solution you'd like
Ideally I'd like to be able to pass an instance of a Manager to a child process and let the child process create and update counters. This probably involves some kind of pickle-specific behavior to ensure the main processes' output streams aren't attempted to be copied.

Describe alternatives you've considered
A less change-heavy alternative could be to add an additional class (maybe ManagerProxy) which could be retrieved from an existing manager (manager.get_proxy()) and this new instance could pickled in order to share it with child processes. The proxy would have the same API, but all output would be sent via a pipe and read by the parent process during an update.

The proxy approach might not work well if the main process blocks for a long period, but perhaps a process just for directing terminal output could be started.

Additional context
Tried to pass the manager between processes:

TypeError: cannot pickle '_io.TextIOWrapper' object

Originally I was using tqdm, which did work between processes but every progress bar had to be positioned manually and they would interfere if processes updated the bars at the same time.

[avylove]

Thanks for your feature request! I'm not sure I agree with this approach, but happy to have the discussion. Here are my thoughts.

The preferred method for these types of situations is to have the manager in the main process and use queues to communicate changes. There is an example here.

You can probably achieve something similar to what you had in the past by specifying the positions of the progress bars, though you'd probably have to have a static status bar at the highest position to pin everything.

Both Enlighten.Manager and the underlying Blessed.Terminal require streams in order to function, so I'm not sure how you can leave them out or otherwise abstract them for pickling.

The next release of Enlighten will provide a BaseManager class that will make it easier to create your own managers, however they still have a Blessed.Terminal instance applied to them, so you'll likely have the same pickling issue.

[robobenklein]

The idea of keeping all output streams in one process is appealing to me too, though setting up queues manually for an unknown number of sub-counters feels like a bit much work.

To me it sounds like most of such an implementation might be done using python's builtin Manager and creating Proxies from a Custom Manager.

Would it be possible to pass a Counter created with one of these Proxies as the manager instance var instead of the full manager? I barely looked, but is it possible to pickle the Counter class if the self.manager reference was a multiprocessing Class Proxy?

[avylove]

I think you're already managing your subprocesses. Adding counters and queues to that should be pretty simple, but I haven't seen your code.

I just don't see how pickling gets you anything. There should be only one manager and that should be in the main queue, so even if you pickle counters or some sort of manager proxy, you still have to pass that back and forth and somehow maintain the counter-manager associations, so why not just pass the updates?

If you want to put something in a pull request, I'm definitely open to it, but, at this point, I don't see how this can be generalized.

[robobenklein]

I went ahead and tried implementing my idea and it works well enough, but it involves a hack of python's multiprocessing.Manager to trick the server into running in the main process while there are other threads as well.

Here is the code if you're interested: https://github.com/utk-se/WorldSyntaxTree/blob/3a4f69c4bc8ba843232b325755bbc8556e388d3f/tests/enlighten/idea.py

2021-05-14_10-40-51.mp4

It doesn't proxy anything apart from new counters, but it's enough to let each subprocess allocate / create as many counters as it wants dynamically, which is good enough for my use case that I might even use some of this quite hacky code. (Though I'd want to add subcounters as well)

For sure if such a feature were to be implemented upstream there's a cleaner and less error-prone way to do so, just would take more time to develop it.

[avylove]

That's interesting! I haven't needed to do anything like this, but I can see how it would be useful. That said, it's complicated code to wrap one's head around, and I'm not sure I'm going to have the time to dig into it in the near term.