Create new Jupytext .md File Option

[psychemedia]

I've just been experimenting with a simple set up to try to create a basic environment that just exposes Juptext markdown files in a top level folder, and stashes notebooks in a hidden folder:

mkdir -p .jupyter
mkdir -p markdown
mkdir -p .notebooks

echo 'c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"' >> .jupyter/jupyter_notebook_config.py
echo 'c.ContentsManager.default_jupytext_formats = ".notebooks//ipynb,markdown//md"' >> .jupyter/jupyter_notebook_config.py

It seems to work, BUT:

• if I use the Jupyter homepage New menu to create a new notebook, it creates a new .ipynb file.

I can create a new text file from the same menu, set it as an .md file, and it seems to work okay, opening as a notebook and saving as a dual, but it lacks the Jupytext header yaml or comment markers.

So I wonder:

• would it be useful to be able to create a "full" (templated) Jupytext markdown file from the Jupyter home New menu, as well as from the notebook File > Jupytext menu?

Also, what do I lose out on by using a vanilla markdown file compared to a Jupytext markdown file?

FWIW, my demo repo is here. (The demo .md file was actually created from an ipynb file by Jupytext in an earlier demo. To test, run the repo via Binder and then create a new .md text file in the markdown directory.

[mwouts]

Nice challenge!

I use the Jupyter homepage New menu to create a new notebook, it creates a new .ipynb file.

Well that's a sensitive defaults I think... It should be possible to insert a new menu item to create a notebook with another extension, but the main complexity here is probably to get a meaningful UI in the end...

Also, with your config the new notebook will become a paired document only if it matches the prefix, i.e. if you create it in the .notebooks folder.

(The .md file) lacks the Jupytext header yaml or comment markers.

That's on purpose. In the earlier versions, Jupytext added this header to text files, but I got consistent feedback suggesting not to modify the files. Anyway, if you do want the header, you should just remove the filter on the notebook metadata. Starting from version 1.1.7 there is now a Include Metadata entry in the Jupytext Menu exactly for this.

Also, what do I lose out on by using a vanilla markdown file compared to a Jupytext markdown file?

Not much! You lose

• the kernel information
• jupytext's format version information

Note that in the case of paired notebook, you don't lose the kernel information because that metadata is stored in the .ipynb document anyway...

[psychemedia]

the new notebook will become a paired document only if it matches the prefix, i.e. if you create it in the .notebooks folder.

Yep.. what I'm working towards is a set up where the user doesn't have to see the*.ipynb files at all...

That is, I want a pure markdown document workflow, at least as far as files in the user's public directory listings go.

[psychemedia]

Some more observations/questions about the dual / pairing only working from .ipynb files in the .notebooks directory:

• (I need to check this...) is it possible to set a directory path watcher on one filetype, but no directory path on another? For example: c.ContentsManager.default_jupytext_formats = ".notebooks//ipynb,md", the intention being that you can then write an .md file anywhere and pair it with notebook files in a specific directory;
• will .ipynb files down a path in .notebooks get an automatic dual? For example: .notebooks/foo/notebook.ipynb? Would / could / should the path be propagated, eg creating .markdown/foo/notebook.md or would it (potentially overwrite) .markdown/notebook.md?
• and a more general question: is there any way in which Jupytext machinery can be used to automatically manage the handling of other batch conversions in a JupyterLab context where arbitrary files are saved from JupyterLab? Eg I had thought drawio might be a candidate, where saving a drawio file might trigger the generation of an SVG export, but I made a bad assumption about it... I'd guess pairing will only work where the content manager is invoked but thought it worth asking to demonstrate my naivety, if nothing else! I suppose a hacky other solution would be to just watch particular directories for files appearing and then run a command line job over them when they do.

[mwouts]

Hi @psychemedia ,

is it possible to set a directory path watcher

Jupytext does not set any watcher. The way it works is through the contents manager. The CM receives the instruction to load or save a specific file, and then actually loads or saves two files. Regarding drawio, I think that you want to duplicate the file to another format only when you save it, so a pre_save_hook should be enough.

c.ContentsManager.default_jupytext_formats = ".notebooks//ipynb,md"

That should pair every .md file (opened and saved as a notebook) to a .ipynb notebook in a local subfolder .notebooks.

will .ipynb files down a path in .notebooks get an automatic dual? For example: .notebooks/foo/notebook.ipynb?

I don't think so. The pairing format above only matches paths for which the parent folder is ".notebooks/.

There are a few tests about that in test_contentsmanager.py, especially this one. I find that these tests are a convenient way to write down our expectations about the pairing - please let me know if you want to contribute a few more of these tests...

[technic]

I don't think so. The pairing format above only matches paths for which the parent folder is ".notebooks/"

Hi @mwouts , I think this is strange. I want to have two directory trees with the same structure, but one with the ipynb files, and another one with python files. Say "notebooks" and "scripts".

Is it possible to acheive this now, or could you suggest a better way? I find it quite annoying that I have duplicates of each notebook (py + ipynb) in the jupyter interface.

[mwouts]

Hello @technic ,

I want to have to directory trees with the same structure, but one with the ipynb files, and another one with python files.

This is a nice idea. Currently it is only possible with folders, not trees, cf. https://jupytext.readthedocs.io/en/latest/using-server.html#per-notebook-configuration.

So, if I understand correctly, you have two folders at the root of your Jupyter server: notebooks and scripts, and you would like to use, say, jupytext.formats="/notebooks//ipynb,/scripts//py" (using e.g. the initial / to indicate that path is matched from the root rather than from the parent dir) to pair both document types together. Is that correct?

[technic]

Yes, that's what I want.
Now it works only for notebooks in the matched directory, but not for the sub-folders. I want to map for example /notebooks/foo/bar/qqq.ipynb to /scripts/foo/bar/qqq.py.

[mwouts]

Hi there, I'm going to close this one, as

• Jupytext 1.5.0 will have the pairing from the root suggested by @technic - use e.g.

# By default, the notebooks in this repository are in the notebooks subfolder
# and they are paired to scripts in the script subfolder.
default_jupytext_formats = "notebooks///ipynb,scripts///py:percent"

as your jupytext.toml file, see #424 (and #384 for the config file)

• Jupytext 1.5.0 will have a 'New Text Notebook' entry in the file menu, allowing the user to create new md or py notebooks ([Feature Request] Enabling ipynb-free workflows in Jupyter #443).

A RC of Jupytext 1.5.0 is already available (pip install jupytext --upgrade --pre), and the final release should be ready within a few days.