Replace thebe with jupyter-js-services, jupyter-output-area

[parente]

Thebe has served us really well in getting dashboards going. But it's actually built for a different use case from our own that includes showing code to users, letting users modify it, having run buttons for users to click, having open access to samples, etc. Fundamentally, we're using thebe to:

1. request a kernel
2. send code to that kernel
3. show output from that kernel (widgets, plots, text, etc.)
4. send widget comm messages to the kernel

As the work to refactor the Jupyter notebook frontend into a set of libraries proceeds, we should be able to build a smaller client suited specifically to the dashboard use case via the jupyter-js-services and jupyter-js-output-area packages. Some benefits of doing this:

• No extraneous deps for dashboards (e.g., CodeMirror)
• Better control over API and kernel interactions (e.g., to later add auth, code trust, etc. for Make deployed dashboards secure, scalable #13)
• Future-proof interaction with kernels by using the official client libs

At the moment, we don't know how mature these libs are yet, so initially, we'll just investigate to see how far we can get with what exists. Maybe @blink1073 can comment.

Over time, we'll want to stay in touch with @odewahn and @zischwartz because we might find that we're actually reinventing a lot of the thebe controller logic but with the input areas turned off. In that case, it might be better to work together on a single client.

[parente]

/cc @dalogsdon, @jhpedemonte

[blink1073]

I think services is stable enough and welcome a tire-kicking.  The API of output area is under review and will likely change soon, but the functionality is there. 

On Wednesday, Nov 4, 2015 at 13:43, Peter Parente [email protected], wrote:

/cc @dalogsdon, @jhpedemonte

—
Reply to this email directly or view it on GitHub.

[parente]

@blink1073 Cool. That's helpful. We'll open defects if the js-services tires deflate upon kicking, but for output we'll hold off until the API change happens.

[jhpedemonte]

Taking.

[parente]

We talked a bit today about how to approach this item. I think we agreed initially we're just looking at using the new client libs, but keeping the same model thebe uses with pre tags for code in HTML. Once that's working, we can start to look at ways of keeping the code out of the frontend as part of #13 or a new issue.

[jhpedemonte]

Currently looking at the comm support and working with IPyWidgets. It looks like Thebe is compiled with the IPyWidgets extension. At some point, the comm "targets" (such a matplotlib) are called so that they can register a comm channel. Currently trying to figure out how this would work with jupyter-js-services.

[rgbkrk]

I'd pay attention to the new decoupling of ipywidgets, as it's targeted towards making the comms be able to live outside of notebook contexts: jupyter-widgets/ipywidgets#151

[jhpedemonte]

Another issue: using jupyter/jupyter-js-output-area for several output messages, including display_data. It handles correctly for the most part, except there is no 'transformer' for JavaScript. In our current test case using IPyWidgets, one widget sends back the matplotlib JS to be run within the browser. But jupyter-js-output-area doesn't handle that.

[parente]

Guessing here: is that the transformime library's responsibility?

[jhpedemonte]

Yes, it is -- that's where all of the transformers come from. But I don't see any mention there about adding a JavaScript transformer. At least jupyter-js-output-area says // JavaScript would go here, IF I HAD ONE.

If we want to get things rolling in the short term, may need to roll our own.

[parente]

Could be this is what @blink1073 was talking about with changing output-area. @rgbkrk or @blink1073 any comments on how to proceed? We can probably do a PR for the JS renderer, but need some understanding on what you're planning to do with transformime in the long run.

[blink1073]

I believe they just haven't added support for JS rendering. The refactor just pushes the output handling to a lower level of the jupyter-js-output API. The proposed API is here.

[rgbkrk]

Let me check that out now. Transformime can and should be able to support this directly. We didn't leave it as one of the defaults, it got implemented by @jdfreder in https://github.com/nteract/transformime-jupyter-transformers.

[jdfreder]

WRT ipywidgets support, there isn't support for jupyter-js-services yet. I'll probably end up moving to the jupyter-js-services comm API and shimming the current API (notebook compatibility), for ipywidgets 5.0 (next release) or 6.0 (depends on how much time I can allocate to 5.0).

[rgbkrk]

Alright, the jupyter js output area now handles both script and svg. HTML is (mostly) ready to go, waiting on nteract/transformime#32.

[rgbkrk]

We can probably do a PR for the JS renderer, but need some understanding on what you're planning to do with transformime in the long run.

Transformime's one and only purpose is to take a mimebundle (data + mimetype) and return an HTML Element. It chooses the richest from a collection of transforms, mostly to mirror what the current notebook (as well as sidecar, Atom notebook, and Hydrogen) does.

It occurs to me now that we'd be well served by relying on the Blob API + URL.createObjectURL for quite a few of the current transforms.

[dalogsdon]

We see that matplotlib uses the old Comm API as well. Are there plans to update that to support jupyter-js-services?

[jdfreder]

I can't speak for matplotlib, but they'll probably wait till it's adopted in notebook stable.

[jhpedemonte]

@rgbkrk We noticed in our tests that maplotlib returns it's JS with mime type application/javascript, whereas transformime's script transform registers as text/javascript. Is the latter the new way going forward and matplotlib (and others) needs to be updated? Or should ScriptTransform register as ['text/javascript', 'application/javascript']?

[jdfreder]

@jhpedemonte definitely - nteract/transformime-jupyter-transformers#11

[rgbkrk]

Ok, transformime has shipped with full HTML execution in the full global context of the notebook thanks to @Karissa. I've bumped the version of transformime used in jupyter js output area here: jupyter/jupyter-js-output-area#20

[jdfreder]

jupyter-js-output-area bumped/published to 0.2.0 with the changes

[jhpedemonte]

Thanks, gents! Have verified output-area and transformime now work as expected.

In PR #127, we've updated our Thebe-replacing code to work mostly like Thebe did, even against tmpnb. I think remaining work depends on update IPyWidgets (both for IPyWidgets and our Declarative Widgets).

[okdistribute]

Thanks gent, @jhpedemonte!

[parente]

Thanks everyone who worked on this so far.

@jhpedemonte I agree with putting this on the shelf until declarative widgets works with notebook 4.x and ipywidgets works with jupyter-js-services release.

@jdfreder if you want / need some help with the widgets comm work, reach out. Clearly we have a vested interest. :)

[parente]

jupyter/dashboards_server#1 and jupyter/dashboards_bundlers#3 will track the continuation of this work form here on out.

This issue is crosslinked like mad so we don't lose the history.