Provide live documentation support in Jupyter using Thebe

[nthiery]

Thebe is a javascript library to turn static HTML pages into live documents where code cells can be edited and executed. Thebe was been introduced in #20690 as a replacement for the live documentation feature of sagenb. It covers the use case of browsing the documentation through the Jupyter notebook; in this case, Thebe connected to that notebook for computations.

However this model was not sustainable:

• Shipping Thebe is additional burden for Sage's maintainers and packagers
• Thebe is a fork of the Jupyter code base and thus not maintainable.
• In practice, Thebe got out of sync and soon became incompatible with the Jupyter we shipped. So the feature was lost.

Meanwhile, Thebe has been reimplemented as ThebeLab, a thin layer on top of JupyterLab, making it much more sustainable. In addition, an online computation backend is now available thanks to http://mybinder.org and Sage's docker container.

In this ticket, we switch gear, and focus on the use case of browsing the documentation from anywhere (within Jupyter, local static documentation, Sage's online documentation), with the assumption of an internet connection.

Implementation:

• When compiling the documentation, Sphinx adds a small header to the produced static html pages.
• If a page contains a sage code cell, an Activate button is added.
• Upon activation, and only then, the thebelab javascript library together with its dependencies in the jupyterlab javascript library are fetched from the web.
• Thebe is configured to use Sage's docker container on MyBinder as computation backend.

Bonus:

• Get rid of Thebe's package.
• A few bells and whistles of the new Thebe implementation (status field, ...)
• Use increasing versions for thebe #21680 can be closed

The feature has been tested for a couple months on
More SageMath Tutorials. It's also configured by default for any Sage package that uses sage-package; see e.g. this page.

We believe that dropping the use case of offline browsing while having a notebook running (which is broken anyway) is largely compensated by the much easier maintenance and the availability of the feature for all deployments of Sage's documentation.

thebelab tarball: https://registry.npmjs.org/thebelab/-/thebelab-0.2.1.tgz

CC: @jdemeyer @sagetrac-tmonteil @embray @videlec @jhpalmieri @kwankyu @antonio-rojas

Component: documentation

Keywords: notebook

Branch/Commit: u/jdemeyer/upgrade_from_thebe_to_thebelab_for_live_documentation_support @ c991041

Reviewer: Nicolas M. Thiéry

Issue created by migration from https://trac.sagemath.org/ticket/24593

[nthiery]

Branch: u/nthiery/upgrade_from_thebe_to_thebelab_for_live_documentation_support

[nthiery]

Commit: b4497c1

[nthiery]

New commits:

b4497c1

24593: Upgrade from Thebe to ThebeLab for live documentation support

[nthiery]

comment:5

I am currently building the whole documentation on my machine to test it locally.

[nthiery]

comment:7

Works here, either when opening the html doc files directly in the browser as file:..., or by browsing the documentation through the Help menu of the Jupyter notebook.

Hence needs review.

[nthiery]

comment:9

I had a look at the patchbot reports; one is gree. Some of the patchbot fails on compiling giac: most likely unrelated. For another the startup time is affected; again, it seems likely to be unrelated.

[nthiery]

comment:10

See sage-devel for the giac failure: apparently due to gcc that needs rebuilding (#24599) .

[sagetrac-tmonteil]

comment:11

I am not sure to completely understand how the whole stuff work, here are some questions:

• The thebelab javascript is provided by a CDN (namely unpkg.com), and since the version is hardcoded in the link, if there is an upgrade of thebelab, there will be an update within Sage source code as well, so why not providing the javascript locally (by packaging thebelab), and avoid this remote intermediary ?

• The work of forking jupyterlab to extract the thebe-like part was done already by thebelab. Why not using that interesting part locally and use the running jupyter instance instead of mybinder ?

• How could a user running the doc from her Sage installation be sufficiently warned that, while a Sage kernel is currently running on her machine, the code will however be executed remotely ?

• mybinder FAQ explicitly states * You shouldn't do anything on mybinder.org that you wouldn't mind sharing with the world! * see https://mybinder.readthedocs.io/en/latest/faq.html#can-i-push-data-from-my-binder-session-back-to-my-repository How is this issue addressed since the goal of live documentation is precisely that the user will modify the cells with stuff she wants to try ?

• The local documentation depends on the version of Sage (e.g. existence of some method, option, ...), hence there must be a corresponding kernel on mybinder:

  • How is the local Sage version transmitted to mybinder so that the correct kernel is launched remotely ?
  • This require a frequent updates on the mybinder machines (one at each release). Who takes the responsibility of long-term maintenance of remote kernels ?

• What is the point of having this into Sagemath source code if Sagemath can not run it, as it requires remote servers to work anyway ? Why not instead maintaining a livedoc.sagemath.org service ? This would be more consistent and understandable from a user perspective.

• Is it possible to disable this feature when building Sage documentation (as we did for the plot directive) ?

• [this one is sarcastic] shall we consider having sage --notebook open a webpage redirecting to some cocalc or any similar centralized, bandwidth consuming, earth warming, privacy harming, open-source alternative ?

[videlec]

comment:13

I don't understand the following

Thebe is configured to use Sage's docker container on !MyBinder as
computation backend.

This means that you need to use MyBinder in order to have live documentation? What is the point of having Sage on my computer then? To my mind, having a live documentation needing internet and a web service is not worth it.

[videlec]

comment:14

To balance comment:13: I think that using MyBinder would be good for let say livedoc.sagemath.org as Thierry suggested in comment:11.

[novoselt]

comment:15

What exactly are the differences between Thebe and SageMathCell? In particular:

• Do multiple users share the same running instance of the container?
• How many concurrent users can be supported?
• How long can a session run, i.e. if there are several linked cells, how long can one wait after executing the first one before executing the last one?
• What are CPU/memory/disk usage limits?

Regarding others comments: I also do not see the point in having my local documentation (which is build from the source on my machine, perhaps my own branch) to use for computations remote servers. Having a link (perhaps glued on the screen as the current Activate button) to some web server with live cells would be much less confusing in terms of accessing remote resources and different versions.

[nthiery]

comment:16

Hi everyone,

Thanks for the thoughtfull comments. I am done with teaching for the
week, and can finally come back to this.

I realized I forgot to state the obvious: yes, when possible, using a
locally running jupyter would be preferable for computations for all
the aforementionned reasons (use of resources, privacy, consistency
with the locally installed kernel, ...).

But this requires some additional work:

1. packaging thebelab
2. packaging jupyterlab
3. implementing the logic: "am I being served by a Jupyter server? If yes, connect there; otherwise offer binder as backup".
4. making sure that the jupyterlab client code is compatible with the Jupyter server we are running

I therefore aimed for a strategy in two stages:

• Implement something simple for now, put it in production, and assess how robust this is
• Add support for using a local Jupyter server when available

Additional motivations:

• the use of the local jupyter by our previous Thebe setup was broken
  anyway, This ticket tackles the most urgent: reinstating the feature
  in most use cases.
• we might as well wait for Sage to upgrade from Jupyter to JupyterLab
  before implementing the second step, to save on 2. and 4.

What do you think?

Now to more specific points:

• The thebelab javascript is provided by a CDN (namely unpkg.com),
  and since the version is hardcoded in the link, if there is an
  upgrade of thebelab, there will be an update within Sage source
  code as well, so why not providing the javascript locally (by
  packaging thebelab), and avoid this remote intermediary ?

The link is actually using semantic versioning (that's the ^ in
thebelab@^0.1.0). So it will fetch the latest version of thebelab
whose API is meant to be backward compatible.

• The work of forking jupyterlab to extract the thebe-like part was
  done already by thebelab.

Just for clarification: it's not a fork. thebelab is using the vanilla
jupyterlab as a library.

• How could a user running the doc from her Sage installation be
  sufficiently warned that, while a Sage kernel is currently running
  on her machine, the code will however be executed remotely ?

Yes, that's an important point. The tooltip explains this. Do you have
specific suggestions? We could e.g. easily change the Activate button
title or the messages that appear when it get pressed.

• mybinder FAQ explicitly states '' You shouldn't do anything on
  mybinder.org that you wouldn't mind sharing with the world! '' see
  https://mybinder.readthedocs.io/en/latest/faq.html#can-i-push-data-from-my-binder-session-back-to-my-repository
  How is this issue addressed since the goal of live documentation
  is precisely that the user will modify the cells with stuff she
  wants to try ?

The live documentation, is mostly meant for casually toying with small
variations of the existing examples; this sounds unlikely to leak
private information. That being said, yes, the user should be well
aware of what's happening in case she would want to play harder.

• This require a frequent updates on the mybinder machines (one at
  each release). Who takes the responsibility of long-term
  maintenance of remote kernels ?

The bulk of the work is to update the docker container, which have
many other use cases anyway. This is mostly automated. Erik and Julian
are working on fully automatizing this, toward including this in the
release process.

Beyond this, there remains on trivial item: updating the label in:

https://github.com/sagemath/sage-binder-env/blob/master/Dockerfile

I am happy to volunteer on that for the next couple years :-)

alternative the livedoc.sagemath.org approach

Our users browse the Sage documentation in many ways: on
doc.sagemath.org, through the notebook's Help menu, or browsing
directly the local html pages, or a copy of them on their department
page. With this ticket (and admittedly assuming internet), the user
can make the documentation live right where he is. Forcing her/him to
navigate to a parallel set of documentation pages is only making
things more complicated. And breaks the flow of reading.

Besides this would also forces us to setup and maintain a new service.

• Is it possible to disable this feature when building Sage
  documentation (as we did for the plot directive) ?

It should be doable, but I don't necessarily see the point. Except for
the 10 lines of javascript/css to setup the button, nothing happens
unless the user explicitly presses the activate button: no connection
to binder, no thebelab download, no network activity of any form.

About thebelab and the Sage cell server

That's a real question, as there is much overlap between the two
projects. We actually discussed this with Jason Grout last week.

ThebeLab, like Jupyter, is language agnostic and targets a much much
wider community than the Sage cell server. It's therefore likely to
get more traction in the long run, in terms of users, developers, and
computing resources. It can also use any Jupyter server (a local
jupyter, a jupyterhub, binder, ...) that the user has credentials for.

The Sage cell server on the other hand is much more advanced when it
comes to performance. For example, when used with binder, a full new
jupyter ends up being started each time, rather than just spawning a
new kernel, or even forking an existing one.

I would love to see a convergence between the two projects, with the
cool features and ideas of the Sage cell server being ported over to
ThebeLab (or presumably down the stack into Jupyter/JupyterHub). Kind
of what happened with the Sage notebook and Jupyter.

Cheers,
Nicolas

[nthiery]

comment:17

Replying to @novoselt:

What exactly are the differences between Thebe and SageMathCell? In particular:

• Do multiple users share the same running instance of the container?

No

• How many concurrent users can be supported?

I don't know for sure. Since the project has been revamped a couple months ago, Binder seems to scale relatively well. There are several ongoing efforts as well to add more deployements (like this one on the "EU cloud") though they are not federated yet.

• How long can a session run, i.e. if there are several linked cells, how long can one wait after executing the first one before executing the last one?
• What are CPU/memory/disk usage limits?

I would need to check on the binder docs.

I remember beeing really annoyed by Sage's live documentation that would loose its history in the middle of my lectures. But I have changed workflow and haven't used ThebeLab enough to have a feeling on whether this could become annoying for a typical usage.

[haraldschilly]

comment:18

Hi, I see this here for the first time. Since I'm the one maintaining the public sage documentation files, will this an impact on it? Since that last "thebe" change, I always have to apply a patch to Sage in order to make it work (basically, to disable the javascript). https://github.com/sagemath/documentation/blob/gh-pages/thebe.patch

Is there also an intention to make this work for the doc.sagemath.org pages?

[nthiery]

comment:19

Hi Harald,

Hi, I see this here for the first time. Since I'm the one maintaining the
public sage documentation files, will this an impact on it? Since that
last "thebe" change, I always have to apply a patch to Sage in order to
make it work (basically, to disable the javascript).
https://github.com/sagemath/documentation/blob/gh-pages/thebe.patch

Is there also an intention to make this work for the doc.sagemath.org
pages?

Thanks for investigating this. Yes, having this work on
doc.sagemath.org is one of the big point.

The new implementation consists of a couple lines of javascript and
css, which just assume standard internet connection to further fetch
thebelab's javascript from a cdn and connect to binder. This worked
without special action on ReadTheDoc. See e.g.:

    http://sage-package.readthedocs.io/en/latest/sage_package/sphinx-demo.html

Do you foresee anything specific in the doc.sagemath.org setup that
could break that? If useful, I can provide with a tiny tarball with
one example page and the js and css to be temporarily deployed in a
corner of docs.sagemath.org for testing.

Cheers,
Nicolas

[haraldschilly]

comment:20

Do you foresee anything specific in the doc.sagemath.org setup ...

No, there is nothing weird going on. It's just static files hosted on the "fastly" CDN.

OTOH, I also don't remember what the problem is right now. Probably that it didn't load at all or something like that. That's why I disabled it.

[nthiery]

comment:21

No, there is nothing weird going on. It's just static files hosted on the "fastly" CDN.

Good. Then this should "just work".

OTOH, I also don't remember what the problem is right now. Probably that it didn't load at all or something like that. That's why I disabled it.

Thebe is currently (i.e. before this ticket) configured in the Sage documentation to connect to a local notebook. So the feature could never have worked on doc.sagemath.org anyway. It could be that, in addition, the detection logic was faulty and causing problem.

[nthiery]

comment:22

I believe I provided tentative answers to the questions that were asked, so I am setting this back to needs review. Mostly as a ping.

Any opinions? In particular on the two stages approach?

[mkoeppe]

comment:101

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[slel]

comment:102

Requested again at

• Ask Sage question 56875: Working through the Sage tutorial in a notebook error

[slel]

comment:103

Thebelab changed its name back to Thebe.

[mkoeppe]

comment:104

Setting a new milestone for this ticket based on a cursory review.

[kwankyu]

comment:106

Nicolas, what is the status of the branch?

[mkoeppe]

comment:108

I've split out the removal of the old thebe package as #33529.

I don't think we need to ship the new thebe at all as an SPKG. As our use of it involves mybinder.org, the generated web pages can also just get the thebe from the CDN.

[kwankyu]

Reviewer: Matthias Koeppe

[kwankyu]

Changed author from Nicolas M. Thiéry, Jeroen Demeyer to none

[kwankyu]

comment:109

I agree. The feature in the ticket description is now achieved by #33507. Hence we can close this ticket.

But I would like to acknowledge that the infrastructure, namely the repo sagemath/sage-binder-env set up and maintained by Nicolas M. Thiéry and Jeroen Demeyer was very valuable for the rapid development of #33507. Thank you.

[kwankyu]

Changed reviewer from Matthias Koeppe to none

[nthiery]

comment:111

I agree that this ticket is no longer needed, and that
having a solution based on a now existing external
library (jupyter-sphinx) is better!

[kwankyu]

Reviewer: Nicolas M. Thiéry

[kwankyu]

comment:114

To be continued at #33320.