Jupyter extension for Jupytext

[mwouts]

An extension for Jupyter would make the configuration of paired notebooks easier. Rather than asking to edit the jupytext_formats metadata, we should offer the user an interface where the user can select the desired notebook extensions (among ipynb, py, jl, R, md, Rmd; the order matters), and the format for each extension (default, light, percent, sphinx...).

The documentation on how to distribute a notebook extension is here. Extensions have to be written in Javascript, which I have almost never practiced before - help is welcome!

Useful references are:

• 4 ways to extend jupyter notebook by @parente
• the gist extension by @minrk

[danieltomasz]

Maybe this interface could be used to temporaralily turn on and off the jupytext (when someone want to see .py file as textfile, so the user doesnt need to edit notebook configuration file by hand - but this I is maybe another, more complicated problem)

[mwouts]

@danieltomasz , I've not find yet how to get started on the extension. Still I have a quick tip relative to your question: if you want to open Untitled.py as a text file in Jupyter notebook, then replace notebooks in the URL with edit. You get something like http://localhost:8888/edit/Untitled.py... and I guess that is the text editor you were looking for!

[FlorianWetschoreck]

@mwouts if you want we can do a pair programming session on this one. I have quite some experience working with Javascript. Let me know on Twitter if you are interested https://twitter.com/Wetschoreck

[mwouts]

@FlorianWetschoreck , I've been thinking about what would be a good start for the extension. Here is what I found - this is just a draft, if you think otherwise please let me know!

• I would like a view on the global config of Jupytext, i.e. the entries that are found in .jupyter/jupyter_notebook_config.py. I think for the extension it would be simpler to read and edit the config from .jupyter/jupyter_notebook_config.json (there is a mention of this config file here)
  • Activate Jupytext: c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
  • Choose, if desired, a default pairing for notebooks: c.ContentsManager.default_jupytext_formats = "ipynb,py"
  • Map extensions to a default format: c.ContentsManager.preferred_jupytext_formats_save = "py:percent" # comma separated, extension:format
  • There are more options (see the code), but the above is already a good start!
• Then, I would like to offer a view on the current notebook configuration.
  • The most important entry is certainly "formats". As a start we could assume that it is made of at most two formats: ipynb, and another among the list of available formats (py:percent and many others, see jupytext --help)
  • We should also allow some space in the interface for the metadata filters, but maybe we do not need them in the first version.
  • And jupytext also has an option for commenting Jupyter magics or not (both a local and a global option)

How do you see the interface? I was thinking of a dialog with two tabs (global and local config). Also, do you think that we can offer an implementation that works for both Jupyter notebook and Jupyter Lab?

[FlorianWetschoreck]

@mwouts , sounds good so far.

For the view of the current notebook configuration:

• there should also be an option to pair/unpair the notebook (with associated formats). I guess that this was clear. I just wanted to note it down again

We can discuss any open details during our hangouts session :)

[mwouts]

Yesterday we reviewed together the Jupyter Lab extension tutorial. The tutorial seems to works very well.

The next step will be to find how to access to either the notebook metadata, or to Jupyter's configuration itself, and to write a dialog with the configuration options. Florian mentioned that a good reference on buttons, drop-downs, etc is bootstrap.

[FlorianWetschoreck]

Here are some hints about how to edit notebook metadata.
jupyterlab/jupyterlab#3379

Also, we can get inspiration from all the official JupyterLab extensions:
https://github.com/jupyterlab/jupyterlab/tree/master/packages

Also, some inspiration about how to test the extension as we talked about in the call:
https://github.com/jupyterlab/jupyterlab/tree/master/tests

[mwouts]

Thanks Florian. On my side, no real progress yet (I've been working on a few other issues), but... the more I look at the toc2 extension, the more I like it!

The extension is for jupyter notebook (but that would be a good start already). The code is not very long, and shows examples of

• modifying the notebook metadata
• adding a button to the toolbar
• adding a download entry in the menu. In the context of Jupytext we would also like to offer the option to download the text representation - see Download as .ipynb returns the associated python file #118

[FlorianWetschoreck]

Sounds good :)

[mwouts]

@FlorianWetschoreck , I have a very basic extension... The code is in the jupyter_notebook_extension branch, and it does the following:

• create a Jupytext button in Jupyter notebook's tool bar,
• the button opens a dialog that displays the Jupytext JSON config for the notebook.

Do you think you could add a few controls to the dialog ? To begin with, I would like to map the 'formats' metadata to the following choices

• does the user wants to save the notebook as ipynb (checkbox?) ?
• to which other format does he want to export? To start with: just one, among, say: markdown, py:light, py:percent.

Thanks!

[FlorianWetschoreck]

@mwouts great start. I will have a look at it on the weekend.
If I understand correctly, you want a button to enable/disable pairing.
And also a dropdown menu, where he can choose from the mentioned formats for the paired file.

Is that correct?

[mwouts]

Thanks @FlorianWetschoreck . Well, for me the difficult part is how to add the button... I think I see already how to edit the notebook metadata, so you should not bother with that part. If you can just show me how to add one dropdown to the dialog, and how to print the selected value in the console log, then that would be great already!

[FlorianWetschoreck]

@mwouts I hope that #136 includes what you have been looking for :)

[mwouts]

Thanks @FlorianWetschoreck . I have added a few more buttons and checkboxes in the dialog.

The next step will be to actually connect the buttons to the notebook metadata itself. Would you like to start with the comment magics flag ? Ideally, I'd like to see how you

• create a jupytext entry in the notebook metadata, if not existing already
• set the initial radio value in the dialog from jupytext.comment_magics (missing=default, true or false)
• delete the jupytext.comment_magics entry when default is selected, otherwise set it to true or false.

Also, I have noticed that it is more convenient to work directly on the local js file (i.e refresh does reload the updated extension). The exact location of the local file is returned by notebook.nbextensions.install_nbextension('jupytext.js', user=True). Or, if you prefer to keep both files in sync, you could even install Jupyter's copy as a symlink...

[FlorianWetschoreck]

Hi @mwouts,
I hope that #140 shows you how to achieve the tasks.
You can also check for the general jupytext entry in the notebook metadata similarly to how I checked for the comment_magics entry within the jupytext object.

[mwouts]

Yes, I think it's exactly what I wanted. Thank you Florian! I will work a bit more on the mapping of the other controls, possibly also review the interface. When I get something good enough to be tested I will share it here. Thanks again!

[FlorianWetschoreck]

Great, I am happy to help, Marc.
If you want, we can also have a small call to discuss the concept and general UI/user flow of the extension. I have experience developing multiple UIs and might have some input if you want.

[mwouts]

Thanks Florian. I am looking forward to your call!

I think we now almost have a working extension. There are two details that I would like to improve, we will also discuss them tomorrow: here and here.

[mwouts]

@FlorianWetschoreck , @fwouts , I could finally write an extension for Jupyter Notebook: https://github.com/mwouts/jupytext_nbextension. And I hope to soon be able to publish another extension for JupyterLab...

The extension adds a Jupytext menu to Jupyter Notebook. It was simpler to start that way than to offer a UI with options. It's very simple, but I believe it is still useful.

There are two small changes I'd like to make: change the location of the menu, and check the selected menu entry. I will open two issues for that at https://github.com/mwouts/jupytext_nbextension, please help if you can!

[mwouts]

I have also started working on a JupyterLab extension at https://github.com/mwouts/jupyterlab-jupytext. Unlike the Jupyter Notebook extension, that one is not functional, because I have no idea on how to access the notebook metadata from the commands. Please share you know-how at mwouts/jupyterlab-jupytext#1! Thanks

[mwouts]

Hi everyone, I'm happy to let you know that

• the jupyter notebook extension works well
• and the jupyter lab extension started working.

Both require the latest rc (now 1.0.0-rc3):

pip install jupytext --pre --upgrade

Feedback is welcome.

Also, before I let the 1.0.0 version go I'll try to bind the extensions with the python package, as nbdime or plotly seem to do.

[mwouts]

I know so little about typescript... Can anyone tell me how to read the content of
notebook_tracker.currentWidget.context.model.metadata.jupytext.formats ?

I'm asking as even notebook_tracker.currentWidget.context.model.metadata.jupytext fails with message
Property 'jupytext' does not exist on type 'IObservableJSON'.ts(2339). For that one I have seen that I can do notebook_tracker.currentWidget.context.model.metadata.get('jupytext'), but that returns a JSONValue, which if I understand correctly is not a map, and thus I cannot access to the formats field...

[fwouts]

@mwouts Sorry for the delay! It looks like JSONValue is a custom type defined by @phosphor/coreutils here: https://github.com/phosphorjs/phosphor/blob/cdb412e9dfb9a987b99143e4be79d477e5136832/packages/coreutils/src/json.ts#L21

What this is saying is basically: "this could be anything" (it's effectively untyped). In this specific case, I think the jupytext field is likely to be an object with a specific structure (including the formats field you expect).

You could do the following:

// Using an unsafe cast (tell TypeScript you know what type this is).
const jupytextConfig = notebook_tracker.currentWidget.context.model.metadata.get("jupytext") as JupytextConfig;

interface JupytextConfig {
  formats: string[];
}

Or if you want to catch any unexpected values, a more verbose approach:

const jupytextConfig = notebook_tracker.currentWidget.context.model.metadata.get("jupytext");
if (jupytextConfig === null || typeof jupytextConfig !== "object") {
  throw new Error("Invalid Jupytext config");
}
if (!("formats" in jupytextConfig)) {
  throw new Error("Invalid Jupytext config (no formats field)");
}
const formats = jupytextConfig["formats"]
if (!(formats instanceof Array)) {
  throw new Error("Invalid Jupytext config (formats is not an array)");
}
// At this point TypeScript knows that `formats` is a JsonArray. You may still want to check that
// every value is a string.
for (const format of formats) {
  if (typeof format !== "string") {
    throw new Error("Format isn't a string");
  }
  // Now TypeScript knows that `format` is a string.
}

[mwouts]

Thanks François, that's very helpful! That should be enough to get me back on the road again...

[mwouts]

Now the jupyterlab extension has a check sign on the selected format. That's great, thanks @fwouts !

I think we are now facing the last issue before the first release of the JupyterLab extension: how can I avoid deleting the full jupytext section when setting format='none'? Because of the current definition of JupytextSection, the condition for testing an empty Jupytext section on this line does not work... Should I list every possible field in the interface, or is there a more generic way?

[fwouts]

There are two things happening here:

• In general, I'd recommend to only use === instead of == in JavaScript. This will test for strict equality. See https://codeburst.io/javascript-double-equals-vs-triple-equals-61d4ce5a121a for details.
• JavaScript doesn't have object equality comparison natively, so you can't do jupytext === {} (it will always be false).

There are a few different ways to check that an object is empty:

• Check that Object.keys(jupytext).length === 0. This gets all the keys of the object and checks that the list is empty.
• Use a library like lodash, which exposes an isEmpty function: https://lodash.com/docs/4.17.11#isEmpty

[mwouts]

The two extensions are available at

pip install jupytext==1.0.0rc4

They are not yet installed automatically (issue #176). However, the manual installation process is easy and documented here.

Thanks @fwouts , @FlorianWetschoreck and @hikenace for your contributions and great support !