Draft: Add bi-directional comms and run operations in tasks

[fleming79]

This PR has many breaking changes so will need to be a major release.

Almost all interaction between Python and the Frontend is run as a task using custom messages. Should an error occur in the Frontend the error will be raised in task. This makes it possible to control the sequence of operations and respond to errors as they occur.

Some Python classes, such as JupyterFrontEnd, are now singleton instances (per kernel) where it makes sense.

Feature summary

• By default, anything created via Ipylab will close when the kernel is closed.
• Support for execution of nested methods in the Frontend. A transformation of the result can be specified prior returning the result to the task.
• List attributes and methods of objects in the Frontend.
• Added Connection to link from Python to a disposable in the Frontend. The close method on the Python object has an option to dispose the object in the Frontend. The disposable object is specified as the base providing direct access to the object.
• Added a method to Panel & SplitPanel to add them to the shell.
• A ShellConnection is provided to the Lumino widget that is added to the shell.
• Restoration of widgets to the shell is supported when the page is reloaded and on startup when the object is evaluate (python code).
• Create new session/kernel with widgets (requires per kernel wiget manager).
• Executing code in a separate python kernel can be awaited.
• Added type hints.
• Added support for Dialogs (including embedding widgets in the custom dialog).
• Added support for creating a Launcher item.
• Added Auto start feature via pluggy (requires per kernel wiget manager).
• Added menus
• Added notifications

Notes

• Awaiting tasks in a notebook or console cell blocks indefinitely when using ipykernel, #1249 might fix it. Alternately, akernel provides a concurrent kernel that partially works.

[github-actions]

👈 Try it on Binder (branch fleming79/ipylab/main)

[fleming79]

@jtpio - There's a lot of changes in this code, but the jist of it is that I've adjusted it from a Python perspective to provide the features that I've been looking for with an app that can run in Jupyterlab (including autostart). I've spent more time than I intended on this (pretty much all of January), but it has help aid in my learning of JS (I expect some of it is a bit sloppy).

If you could find some to to have a look through the notebooks would be appreciated.

Note the major requirements are:

• Python >= 3.11
• Jupyterlab >= 4.0
• IpyWidgets >=8.0

[jtpio]

Thanks @fleming79 for all the work on this!

It looks like a nice feature indeed, will have a look soon 👍

Note the major requirements are:

The JupyterLab and ipywidgets requirements would likely be fine. For requiring Python 3.11, maybe there could be a fallback mode so ipylab is still usable on Python 3.8 - Python 3.10 for the time being?

[fleming79]

I had a look at supporting older versions. With a few changes was able to get it to work with 3.10. But anything older than that require re-writing/removing of type hints.

[fleming79]

Regarding creating a new session without a notebook. I'm wondering if the new session should be based around a document...

Currently, the function newSession defined int utils.ts creates a new KernelWidgetManager, however the base instance doesn't provide the models that get defined using the plugin system. Currently only a limited number known are provided with the function registerWidgets defined in utils. This means that models registered using the ipywidgets plugin system don't get automatically added.

It'd be better to use registerWidgetManager from IpyWidgets, but as you can see in the prototype below, the function is expecting a DocumentRegistry.IContext<INotebookModel> context.

export function registerWidgetManager(
  context: DocumentRegistry.IContext<INotebookModel>,
  rendermime: IRenderMimeRegistry,
  renderers: IterableIterator<WidgetRenderer>
): DisposableDelegate {

[jtpio]

Thanks @fleming79 for working on this!

And sorry for the delay, I'll try to have a look soon.

[fleming79]

@jtpio - no problem. I've been learning as I go and using it for my own purposes.

As an aside, I was looking at your PR for Ipywidgets jupyter-widgets/ipywidgets#3004 from 2020 (which is still open) and saw a bit of a discussion having a widgetManager on a per-kernel basis.

Jason Grout said:
I've been going over this code and thinking about this, and there's something I think this effort is exposing about limitations in how the current lab widget manager is written. This approach (properly, I think) tries to push the notion of a widget manager down to the kernel level (i.e., the widget manager is "owned" by the kernel id, for example). However, the notebook widget manager really lives above that, at the session context level (e.g., a notebook widget manager can be "owned" by several different kernel ids over its lifetime, and right now there really isn't a provision for a notebook widget manager that doesn't happen to be associated with a kernel, as Jeremy points out above when talking about the assertion operator.

This makes me think that perhaps we should restructure the code so that we have one concept of a widget manager that is tied to a kernel, and use composition to interface with the notebook rather than inheritance. In other words, we have a layer on top of the kernel widget manager that manages notebook state and interfaces with the (kernel) widget manager to deal with saving and restoring state from a notebook. In other words, the object at the notebook level HAS a kernel widget manager (not IS a widget manager), and the kernel widget manager can be swapped out or created as needed when the kernel changes. This way consoles and notebooks can cache and share kernel widget managers freely on equal footing. Still thinking through it, but this seems to address this weird disconnect we have between notebook widget managers and kernel widget managers.

I think this concept is worth pursuing and am currently investigating how difficult it may be to implement. It would solve the issue in this PR associated with needing to create a WidgetManager and manually register widgetmodels, and the problem of widget comms shutting down when the notebook is closed.

[fleming79]

I noticed that jupyter-widgets/ipywidgets#3004 has now been merged - that's great!
Edit:
I'm wondering if the legacy function registerWidgetManager here could be used.

[fleming79]

registerWidgetManager appears to work okay with the most recent jupyterlab_widgets mentioned above by creating a dummy context.

  // For the moment we'll use a dummy session.
  // In future it might be better to support a document...
  const session = sessionContext.session;
  const context = {};
  (context as any)['sessionContext'] = sessionContext;
  (context as any)['saveState'] = new Signal({});
  (context as any).saveState.connect(() => {
    null;
  });
  registerWidgetManager(context as any, rendermime, [] as any);

[fleming79]

The concept of a single widget manager per kernel looks very promising, there is a draft PR open here: single widget manager per kernel PR.

This enables creating a session (new kernel) that has it's own comms and widgets enabled. Its usage is demonstrated in autostart.ipynb.

A launcher

An example app

With the console opened

Attachments

Extract and install with pip. Install jupyterlab_widgets first.

• jupyterlab_widgets - single widget manager per kernel
• ipylab - this PR

[fleming79]

@jtpio - Just wondering if you would be able to find some time to try this PR?