Jupyverse Modularity - Workflow to add "server extension" ?

[oscar6echo]

I understand from the doc and conversation that Jupyverse is modular by design 👍

So I understand it should be feasible to add endpoints, aka server extensions. aka asphalt plugins (?).
Do such plugins need be in the jupyverse repo ? Or is there a way to add one by config before lauching the server ?

More generally is there a short demo/tutorial on how to create such a custom plugin (hello world with maybe some use of shared resources) ?

I have the following generic idea in mind:

• A Jupyterlab extension collects some data in the browser - say a token or user input data in extension frontend.
• The labextension sends it to a "POST" custom jupyverse endpoint/plugin
• This plugin saves it on the disk in a specific folder
• Then the kernel can read it from this folder

Similarly the data flow can go the other way if the plugin has a corresponding "GET" endpoint availalble to the labextension.

This would be a relatively generic way to pass data from frontend to kernel and back - which is a frequent use case - and it would be easy to customize.

NOTE: In the case of Jupyterlite, the facility to exchange data between frontend and kernel is the IndexedDB - accessible in read-write from both kernel and frontend 👍

---

Side question:
In the repo there is a lot of "fps" but there is mention (#277) that "fps" has been replaced by asphalt.
However there are still "fps-xyz" imports all over the place including after this mention.
Why ? migration on going? renaming not done ? Just to understand.

[davidbrochart]

Do such plugins need be in the jupyverse repo ?

No, they can live in another repo.

Or is there a way to add one by config before lauching the server ?

I should write more documentation about that. A plugin is an Asphalt component. Actually, Jupyverse is just a collection of plugins. You can launch Jupyverse using the Asphalt CLI and config file(s):

asphalt run config.yaml

In that case, the components in the config file must have been registered in the "asphalt.components" entry point.
But Jupyverse also has its own CLI, and in that case it will automatically start all the components registered in the "jupyverse.components" entry point, no need for a config file.

More generally is there a short demo/tutorial on how to create such a custom plugin (hello world with maybe some use of shared resources) ?

There is no tutorial for now, but you can have a look at Asphalt's documentation, or see e.g. the contents plugin. Basically, you need to require the resources you need, then add the resources provided by the plugin. If the plugin adds new endpoints (which it will probably do), you need to require the App resource, derive the added resources from jupyverse_api.Router, and initialize the Router with the App. In the resource, create a router and include it.

Regarding your mechanism to communicate between the browser and the kernel, what you are describing can work, but it also comes with drawbacks, for instance you will need to watch file changes, etc. Also, it's true that most of the time the server file system is the same as the kernel's, but they can also live on different machines.
A better solution would be to use ipywidgets. They are meant to communicate between the front-end and the kernel, using the kernel Comm protocol over WebSocket. For instance, ipyleaflet was built on top of ipywidgets. They work in JupyterLite as well.

Regarding the fps naming, yes Jupyverse doesn't use fps anymore, but all plugins are PyPI (and conda-forge) packages, so I didn't want to create new ones. Also, FPS stands for "Fast Pluggable Server", which I think is still true 😄 and it's a nice prefix to avoid name collision, so I think it will stay like that. But you can name your plugins as you wish.

[oscar6echo]

Thx a ton for the fast and precise answers 👍

No, they can live in another repo.

Is there an example of how to reference a plugin located in another repo or local folder ?

You can launch Jupyverse using the Asphalt CLI asphalt run config.yaml

So I can already use jupyverse with a completely customized set of plugins without default ones, right ? My idea would be to use jupyverse as a base for a "proxy/cors" server in the context of corp & jupyterlite, but in that case without any of the jlab plugins.

There is no tutorial for now, but ...

This is a very good starting point 👍 I'll delve into it.

A better solution would be to use ipywidgets. They are meant to communicate between the front-end and the kernel...

AFAIK ipywidgets are the prefered way to communicate between notebook frontend and kernel. But my use case is between labextension frontend (i.e. possibly full custom react component - hopefully vue too soon Cf. PR jupyterlab/extension-examples/pull/233) and kernel.
But I understand the warning about a kernel not having access to the server file system. Which triggers me to ask: Could the jupyverse server send an instruction to kernel(s) ? Maybe not all sorts of instructions but limited to settting env variables to (some) living kernels and at start of future kernels ? After all the jupyverse server keeps tabs on kernels, right ? That would open an interesting avenue in frontend in general and kernel.
The practical use case I have in mind is auth. The authentication flow is run once for the web page, not per notebook, and is natively web based. Also tokens typically have known expiry dates, so it is easy for a kernel to determine when it should go fetch the next one (first workflow described - if env var can be set by jupyverse this becomes unnecessary).

Regarding the fps naming...

Indeed, it makes sense. Clear.

---

General remark:
I gradually realize that not only jupyverse has a lot of potential but the tech stack is top notch: all toml, ruff, devcontainer, pre-commit etc - this is a lot of inspiration re best practices !
congrats 👏
This in itself could be a separate section in the doc - if you ask me.

[davidbrochart]

Is there an example of how to reference a plugin located in another repo or local folder ?

It doesn't matter if a plugin is located in another repo or a local folder, it is just a package that registers components in entry points (the "asphalt.components" and/or "jupyverse.components" that I mentioned).

So I can already use jupyverse with a completely customized set of plugins without default ones, right ?

Yes, you can do that. Just don't install any plugin shipped in plugins.

Regarding kernel/frontend communication, you're right that ipywidgets work with a notebook. In your case, I would recommend using ypywidgets/yjs-widgets. JupyterCAD has a similar use case as the one you are describing: kernels communicate using ypywidgets with the frontend, which is well authenticated with the server.

[oscar6echo]

I would recommend using ypywidgets/yjs-widgets. JupyterCAD has a similar use case...

Indeed. ypywidgets & yjswidgets look like they offer the proper com channel.
But JupyterCAD is thick !.. with many layers of very specific (hence hard to comprehend) jupyter configs.

Is there a minimal jupyterlab extension example using ypywidgets & yjswidgets ? If so it would be super if it landed in https://github.com/jupyterlab/extension-examples, ideally with jupyverse. Currently the only server-extension is with jupyter_server, it seems.

Btw JupyterCAD uses jupyter_server, not jupyverse, right ?

Also I'm bit lost with the various Jupyterlab <3.6, 3.6, 4. Do ypywidgets & yjswidgets work with all ? some ? Combined with jupyter_server/verse that's many possibilities..

What is the recommendation re versions now ? I assume longer term it would simply be jupyverse + jlab 4.

[davidbrochart]

I'll try to write a minimal example using Jupyverse and ypywidgets/yjs-widgets. They should work with JupyterLab 3, but now that JupyterLab 4 is out it's better to use it.

[davidbrochart]

I opened https://github.com/davidbrochart/yjs-widgets/pull/4.