Support (Jupyter) Debug Adapter Protocol

[bollwyvl]

ipykernel 6.0.0 just landed which, among other things, supports the Debug Adapter Protocol. We should see if:

• we somehow magically get support 🎉
  • though seems a little unlikely 😊
• we have to do some plumbing to support robot sources, etc. 🔧

This would definitely trigger a need to demonstrate a new 🤖 🔬 !

[datakurre]

😮🙀

[datakurre]

@bollwyvl Is debug adapter protocol the one used by jupyterlab-debugger? Or something new? How does it relate with xeus-robot? Is migrating to xeus still the way to go?

[bollwyvl]

the one used by jupyterlab-debugger

Yup, though it's in core now as @jupyterlab/debugger, so the "default" pip install jupyterlab will start showing it to folks. (of course running into issues on conda-forge because pypy gah)

How does it relate with xeus-robot

Nothing in particular: if just we like being in ipykernel, it's probably an option. i haven't looked into whether the robot-interpreter implements the debugger, or if it's in the kernel.

Of note, though, we're inching towards being able to support ipykernel more directly in jupyterlite which might be an easier road to getting robotkernel running in the browser, with the WASM-powered pyodide until we can start building the world (wide web) in a conda-forge wasm-32 target. await micropip.install("robot"); import robot already works there. I'm really looking forward to being able to use this kind of thing for docs/demos/workshops vs the robotlab approach (much as i love shipping the big GBs).

[datakurre]

Thanks. Now I got it!

So, the new ipykernel makes it possible for us to make ipykernel based robot kernels compatible with JupyterLab debugger by "just" implementing support for DAP-messages and related ipykernel additions. And if I'm not completely lost, the minimal set of required messages to support can be seen at:

https://github.com/ipython/ipykernel/blob/b4352b89efb0f9d938af736103b2c456ab0c6e3f/ipykernel/debugger.py#L196

You would prefer us to keeping ipykernel based robot kernel alive, because it could be easier to port to jupyterlite than xeus-robot?

I'm having a vacation after two weeks, so maybe then I should use a few afternoons to see how far this goal is. Just to get my thoughts out of the current day work projects...

Hacking xeus-robot if definitely out of my reach. Choosing that road would make robotkernel a thin wrapper around xeus-robot and robotframework-interpreter. Yet, maybe even with keeping ipykernel, trying to rebase on top of robotframework-interpreter could make sense.

[bollwyvl]

compatible with JupyterLab debugger

Well, theoretically, any jupyter client, but so far I haven't seen any other implementations (open source or proprietary).

"just" implementing

Right, here's the quick run-down of possible scenarios.

Hacking xeus-robot if definitely out of my reach.

Yep, me too, to my shame... or maybe not really... I've made it this far without having to complain about C++ aside from figuring out other peoples' make or cmake issues... but boy howdy, that's not fun for me on windows. And, like i said, there is some low-grade effort (by folks involved in xeus) to make wasm-32 a supported for conda-forge (or wasm-forge, if it comes to it, as a set of long-term patches).

having a vacation

that sounds lovely. i hope to take some time, too! this whole "using my leave to be a crappy remote kindergarten support aide" has been a real downer.

prefer us to keeping ipykernel based robot kernel alive

Welp, more kernels are... more kernels. Good and bad. I think we(you)'ve built some really nice stuff here, and we've been able to weather a number of upstream changes, and still provide something that's fun to use. The jupyterlite thing is... not yet really a thing, but probably will be a thing.

• A related technology/inspiration, basthon (also based on pyodide) is used all over France to teach folks computational basics... from their browsers, without having to deploy kubernetes.
  • If robot can be a stepping stone (or alternate path entirely) for people learning robot/python/english, i'm all for us doing what we can to support it.
• i love me some nbviewer, but nbviewer plus running kernels in the browser, with really solid, cross-session caching across a number of public content sources, would alter the landscape for real

[datakurre]

Every issue here goes out of the topic in 1, 2, 3... 😄

windows

I've heard that WSL >= 2 has been useful and even Nix has been mostly usable on that.

i hope to take some time, too

💯

The rest of the year will be terrible, but if everything goes as planned (for the next two months), I've warned by employer of a full year family leave next year. 🙌 (And definitely need some hobby project to keep me sane.)

That said, Robocon 2022 is seemingly coming in January 2022, and to have any reason to be there, I should use the few summer vacation afternoons (I almost promised yesterday) well.

i love me some nbviewer, but nbviewer plus running kernels in the browser, with really solid, cross-session caching across a number of public content sources, would alter the landscape for real

So, the first thing is to wait for ipykernel working in jupyterlite? What should be the next steps after that? Robot relies a lot on filesystem access, but you have tried it out in jupyterlite? Does pyodide virtualize file system access or should be just be careful to only use StringIO / BytesIO everywhere? And how to register a custom kernel for jupyterlite? In build time?

Also, are there examples of embedding jupyterlite tech into documentation for interactive examples (without all the jupyter chrome)?

[bollwyvl]

WSL

no comment 🙀 nix-on-win would be a substantial step forward, though MS has decided to pursue their own "package manager" 🤪 as a semi-hostile fork of an existing open source project.

pyodide virtualize file system

Yeah, emscripten gives us a full in-memory POSIX filesystem, which can have mountpoints provided by other things, including indexedDb (which we use for user contents, settings, etc.). This is not exposed in pyodide, but maybe they're waiting on me to PR it 🤣 . Because of this, jupyterlite contents aren't visible to kernels, yet, but we'll get there. As we run pyodide in a webworker (baby's heavy), there are even more constraints, like not being able to access the DOM, but i could foresee being able to run multiple kernels inside one "computer," have multiple "computers".

register a custom kernel for jupyterlite? In build time?

At present, yes, the kernels are a subset of "serverlite extensions," that provide stuff like kernelspecs and contents. These can't be easily composed outside of webpack, yet, but we're mulling over it. We'll be adding something that is (or is like) federated_extensions for these, such that one would distribute it by "normal means" (npmjs.com or pypi) and then be able to add it during jupyter lite build by CLI or config file.

interactive examples (without all the jupyter chrome)?

We have a few build things to square away at the webpack/serviceworker level, as right now every additional app flavor (right now full lab and retro) duplicate a lot of stuff, and every additional view would trigger some more requests. But yeah, the ideas range from something like https://github.com/jtpio/jupyterlite/issues/148 ...

...to something that appears more like a sphinx/mkdocs layout, with the jupyter "chrome" kinda disappearing. Probably be able to reuse existing templates with nunjucks, etc. No doubt people would (rightly) complain about the initial >=5mb slug of JS, but would be silly cool with, e.g. LSP, session-persistent widgets, etc.

O, and the jupyter 3.1 collaborative editing stuff basically works in lite: https://github.com/jtpio/jupyterlite/pull/109 🤯

[datakurre]

Thanks! I subscribed to IPython in Pyolite kernel -pull so that I know when it should be merged and there chance to get robotkernel running. Having interactive client robot would be great for learning / tutorials, and even it could not be used with most of the robot libraries, it could probably be used for some real API testing / automation too. And if that works, I could even use it to demonstrate BPMN + Camunda based microservice orchestration... 🚀

[datakurre]

Did I expect too much in expecting that there is already magic somewhere that would turn requests-library request into client side CORS-request? :)

Update: found that pyodide provides custom API for client side requests.

So, yes. Robotkernel in JupyterLite sounds suddenly much more interesting and current to me than implementing debug adapter protocol (which is also interesting, but has already been implemented in xeus-robot).

[datakurre]

Well. RobotKernel depends on ipykernel.kernelbase, which requires zmq, which cannot be installed, because it requires C-extensions. But probably there's a workaround for zmq already in JupyterLite.

@bollwyvl I see that you have already thinking this further. Which approach would prefer: should I wait until ipykernel is somehow automatically supported dependency in JupyterLite or, should we try to mock kernelbase with something that works in JupyterLite? And are there good starting points available for that already in JupyterLite codebase?

Somehow, I have already managed to isolate ipykernel dependency into https://github.com/robots-from-jupyter/robotkernel/blob/master/src/robotkernel/display.py so using alternate base kernel for JupyterLite should be possible, if there is such base.

[bollwyvl]

requests-library request into client side CORS-request?

Right, it's not the full networking stack, and a number of stdlib things aren't present dbm, lzma.

Network stuff, for now, has to go through something like:

import js
r = await js.fetch(url)
j = await r.json()   # looks delicious, but is a JS object :(
t = await r.text()
import json
j2 = json.loads(t)

note that the .json looks deliciously requests-like, but is in fact a JS object (or proxy), and doesn't just work with a lot of things that would with a normal python object, hence the import json. But given the ubiquity, we'll probably need to do a better shim... See my desire for conda-forge to make this something we can maintainbly scale. Micropip knows we're gonna do weird stuff, and leaves a lot of stuff hanging out... some of the examples do some really weird stuff.

Which requires zmq

Right, we are already shipping some shim packages for some of these ipy* things. It's unlikely that initial PR will expose enough to make our kernel usage work, but there is some stuff we'd be able to check on our own.

Robotkernel in JupyterLite

Yeah, people can pooh-pooh this stuff, but until we try it, we just won't know if it's cool.

[bollwyvl]

I'll make a standalone issue to track the RoboLite concept!