Add documentation of myst-nb format #458

chrisjsewell · 2020-03-17T03:11:28Z

Also added MyST jupyter labextension

codecov · 2020-03-17T03:16:04Z

Codecov Report

Merging #458 into master will decrease coverage by 0.02%.
The diff coverage is 95.65%.

@@            Coverage Diff             @@
##           master     #458      +/-   ##
==========================================
- Coverage   98.91%   98.88%   -0.03%     
==========================================
  Files          79       79              
  Lines        7762     7823      +61     
==========================================
+ Hits         7678     7736      +58     
- Misses         84       87       +3

Impacted Files	Coverage Δ
jupytext/formats.py	`98.38% <60.00%> (-0.31%)`	⬇️
jupytext/myst.py	`98.33% <96.55%> (-0.99%)`	⬇️
jupytext/jupytext.py	`98.67% <100.00%> (ø)`
tests/test_ipynb_to_myst.py	`100.00% <100.00%> (ø)`
tests/test_mirror.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 75449a5...c7384a8. Read the comment docs.

mwouts · 2020-03-17T07:24:51Z

Hello @chrisjsewell , thanks for this PR, which looks good to me!

I will let @choldgraf have a look at the proposed documentation, and confirm the latest changes, namely:

The default extension is now .mnb
Code cells are encoded with code-cell rather than nb-cell, etc

Also, in the previous PR (#456 ) I think that @choldgraf suggested allowing the .md extension for the MyST format. This should be possible. I think you'd have to

add '.md' (and maybe .markdown?) to the list of MyST extensions
patch formats.guess_format so that it can differenciate MyST-md files from Jupytext md files. Or maybe you could directly adapt read_metadata to the MyST-way of representing the notebook metadata - I mean, the jupytext section is at the top level in that format, and that's where you could be looking for the jupytext.text_representation.format_name metadata...

If you do that, then there will be the question of which extension should be the default: .mnb or .md? I think I'd recommend .md, but obviously that's up to you! (Note that it would be a bit harder to adapt the notebook/lab extensions, still you could seek inspiration from the light vs percent formats for scripts, which both share the .py extension).

chrisjsewell · 2020-03-17T07:42:18Z

Thanks @mwouts yes in 316bc2e, I tried adding .md to the extensions, but you'll see that it failed a number of the tests. Perhaps patching formats.guess_format will fix this.

I think I'd recommend .md, but obviously that's up to you!

IMO ideally it would be something like .myst.md. But I'm not sure how possible it is to have such a "double-barrel" extension in jupytext, without changing too much code.

Note that it would be a bit harder to adapt the notebook/lab extensions

Indeed. I did have a quick try at this but couldn't get it to working straight away. I might need some help in that respect.

chrisjsewell · 2020-03-17T08:11:37Z

Note-to-self: /usr/local/share/jupyter/nbextensions/jupytext/index.js

mwouts · 2020-03-17T08:44:26Z

Thanks @mwouts yes in 316bc2e, I tried adding .md to the extensions, but you'll see that it failed a number of the tests. Perhaps patching formats.guess_format will fix this.

Yes, I think so.

I think I'd recommend .md, but obviously that's up to you!

IMO ideally it would be something like .myst.md. But I'm not sure how possible it is to have such a "double-barrel" extension in jupytext, without changing too much code.

With the current code Jupytext will see this as a file with .md. I think indeed it would be too much work to change this. Still: 1) if you fix guess_format, then this .md file will be recognized as a MyST file, and 2) you can pair a notebook to double-barrel extensions, cf. the World Population example in which we pair the .ipynb file to a .pandoc.md file, using the format description .pandoc.md:pandoc.

Note that it would be a bit harder to adapt the notebook/lab extensions

Indeed. I did have a quick try at this but couldn't get it to working straight away. I might need some help in that respect.

Sure! First, let's see what @choldgraf thinks about having .md (or .myst.md) as the default extension, and I'll show you how to use the explicit format name (.md:mystnb) to achieve this.

Note-to-self: /usr/local/share/jupyter/nbextensions/jupytext/index.js

Yes, I saw you had issues with the README, didn't the symlink install work? I'd be happy to know how to fix the README...

chrisjsewell · 2020-03-17T10:58:33Z

Yeh I don't know if its as easy as just changing guess_format. For example, in
tests/test_active_cells.py

---
jupyter:
  jupytext:
    main_language: python
---
```python active="ipynb,py,R,Rmd"
# This cell is active in all extensions
```

There's no real way to differentiate between MyST-md and Jupytext md?
So all these are failing

The only real way to discriminate, is to parse the whole file (with myst) and check if any code cells have been discovered, which is far from ideal.
The quick and dirty way is using the same approach as for pandoc (that looks for lines starting with :::), by looking for any lines starting with ```{code-cell}, I would not that this means that a myst notebook would now require a code cell to be recognised, and there's a bunch of edge cases where this fails (for pandoc as well). But yeh, that's this is the only way. (This is why I think it would be much better to use .myst.md)

chrisjsewell · 2020-03-17T18:50:06Z

Ok I've added .md to the myst formats. @choldgraf can work out how to make that the default for nbextension and labextension

Yes, I saw you had issues with the README, didn't the symlink install work? I'd be happy to know how to fix the README...

Nope, jupyter nbextension enable jupytext --user fails, so I just used jupyter nbextension install index.js --py --user (or something like that) and edited the index.js at the path that I pasted. No idea why I'm afraid, and its why I'm now leaving it to @choldgraf, because its just way too much hassle trying to do interactive development with these extensions 😒

chrisjsewell · 2020-03-17T19:30:23Z

@mwouts if/when you are happy with this PR (do I need to work out tests to add, to cover the -0.05% decrease!?) I would suggest merging, then @choldgraf can work on the notebook/lab extensions separately

choldgraf · 2020-03-17T20:09:01Z

I'd be OK with having .mnb as the default extension. After a few weeks/months then we can revisit the question before too many users have solidified mnb in their heads. 👍

(just a note, I probably will not be working on the lab/notebook extensions, I don't know anything about developing for lab/notebook...but we don't need that to block this PR)

mwouts · 2020-03-17T21:41:52Z

Hello @chrisjsewell , @choldgraf , well this is going fast! 🚀

do I need to work out tests to add, to cover the -0.05% decrease!?

Oh no, surely not!! Actually a few months I tried to fix that with .codecov.yml, but it looks like it didn't work... 😄

I would suggest merging, then @choldgraf can work on the notebook/lab extensions separately

Well, as I user I think I'd prefer to use .md. Maybe it's not that hard to adapt the extensions, let me give it a try...

mwouts · 2020-03-17T22:42:16Z

@chrisjsewell , what do you think of this: 6bec0bc ? This gives the option to pair the document to a md:myst file, and it seems to work as expected. Would you like to give it a try ? Maybe we can also add this to the Lab extension ?

chrisjsewell · 2020-03-17T22:50:36Z

@chrisjsewell , what do you think of this: 6bec0bc ? This gives the option to pair the document to a md:myst file, and it seems to work as expected. Would you like to give it a try ? Maybe we can also add this to the Lab extension ?

Yeh cheers, if you say it works, then that's good enough for me 😄
A minor thing is that I think we can now maybe change the name from MyST-NB to MyST Markdown
(I changed it to this in formats.md),
and yes if you could also add it to the Lab extension, that would be great 👍

chrisjsewell · 2020-03-17T23:26:06Z

FYI I just added the nbextension, copied from your branch. Its working great 👍

chrisjsewell · 2020-03-17T23:39:02Z

One minor improvement that comes to mind, is to set the common sphinx metadata key orphan, and maybe tocdepth (see sphinx field-lists) to sync by default, i.e. initially setting notebook_metadata_filter: orphan.
Is this possible/advisable?

chrisjsewell · 2020-03-18T00:31:26Z

In 9735b58 I have added the option to store the original source line numbers in the cell metadata,
when converting from a text document to a notebook.
This is not directly applicable to jupytext, but is part of the fix for executablebooks/MyST-NB#83, whereby we would like to report the original source line numbers, e.g. for missing references or other sphinx errors/warnings.

mwouts · 2020-03-18T20:11:43Z

Hello @chrisjsewell , great job! I see you've changed the pairing to md:myst in the lab extension as well, so we're good to go! I should be able to find some time later on tonight to release this.

One minor improvement that comes to mind, is to set the common sphinx metadata key orphan, and maybe tocdepth (see sphinx field-lists) to sync by default, i.e. initially setting notebook_metadata_filter: orphan. Is this possible/advisable?

Yes, this is possible. I think it should have very little impact, because the filter ignores metadata when it is missing. All you need to do is to modify metadata_filter._DEFAULT_NOTEBOOK_METADATA below. As long as it has little impact on the tests (i.e. the mirror files), I am ok with any change there.

jupytext/jupytext/metadata_filter.py

Lines 6 to 12 in 2a96dfa

    
           _DEFAULT_NOTEBOOK_METADATA = ','.join([ 
        
               # Preserve Jupytext section 
        
               'jupytext', 
        
               # Preserve kernel specs 
        
               'kernelspec', 
        
               # Kernel_info found in Nteract notebooks 
        
               'kernel_info'])

mwouts · 2020-03-18T22:25:36Z

Thanks @chrisjsewell , I'll merge this, test and release a new version of Jupytext. Thank you for this great contribution !

Just to mention:

maybe I'll revert the JupyterLab extension version number to 1.2.1 (as we have not pushed that version to npm or pip yet).
and, as I like to keep link between issues and PR, this was a follow-up on Add mystnb format #456 and A book publishing project interested in extending jupytext code #447 .

chrisjsewell · 2020-03-18T22:35:54Z

Thanks @mwouts!

(Yeh I'll look into the notebook_metadata_filter for orphan at a later date.)

choldgraf · 2020-03-18T22:41:22Z

BOOM - way to go everybody 🎉 thanks @chrisjsewell for the hard work on this, and thanks @mwouts for the quick responses and turnaround :-)

mwouts · 2020-03-18T22:57:27Z

(Yeh I'll look into the notebook_metadata_filter for orphan at a later date.)

If you like I can take it (and tocdepth) - there's no visible impact here.

chrisjsewell · 2020-03-18T22:59:41Z

Dont merge

(Yeh I'll look into the notebook_metadata_filter for orphan at a later date.)

If you like I can take it (and tocdepth) - there's no visible impact here.

Yep that would be great thanks 👍

mwouts · 2020-03-18T23:40:55Z

Yep that would be great thanks 👍

Done! And jupytext-1.4.1 is now available on pip (soon on conda-forge). Let me know how it works for you... Thanks!

Add explanation of myst-nb format

fb53678

chrisjsewell added 6 commits March 17, 2020 03:27

Rename nb-cell/nb-raw to code-cell/raw-cell

d37f73a

change extension mystnb -> mnb

726e907

Addition of MyST-NB to jupyter labextension

6b5d4df

remove nbformat from front-matter

790fb00

remove accidental addition of notebook

316bc2e

remove .md extension

397d6ca

mwouts marked this pull request as ready for review March 17, 2020 06:56

move myst functions functions to top of file

424756a

chrisjsewell added 2 commits March 17, 2020 18:19

make myst's default extension .md

62351f4

use only the major version of myst-parser

0780e24

chrisjsewell added 2 commits March 17, 2020 18:57

Add to tests

3263403

Allow the notebook to be returned from matches_mystnb

72a4ac7

jstac mentioned this pull request Mar 17, 2020

Add Myst-to-NB converter executablebooks/MyST-NB#82

Closed

chrisjsewell mentioned this pull request Mar 17, 2020

Add new parser for text-based notebooks executablebooks/MyST-NB#83

Closed

chrisjsewell added 2 commits March 17, 2020 21:48

add test

b9eb714

Update test_ipynb_to_myst.py

ee2ae39

Update MyST default to .md in nbextension

6d2084e

Add store_line_numbers option for myst_to_notebook

9735b58

chrisjsewell added 6 commits March 18, 2020 00:37

propogate store_line_numbers to matches_mystnb

f80299a

minor fix

d922b14

Update MyST default to .md in labextension

f9a4a78

Update CHANGELOG.md

1a3796e

remove unintentional file commits

71a91c3

add link to myst-highlight

c7384a8

mwouts approved these changes Mar 18, 2020

View reviewed changes

mwouts merged commit 8ac346d into mwouts:master Mar 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation of myst-nb format #458

Add documentation of myst-nb format #458

chrisjsewell commented Mar 17, 2020 •

edited

Loading

codecov bot commented Mar 17, 2020 •

edited

Loading

mwouts commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020 •

edited

Loading

chrisjsewell commented Mar 17, 2020

mwouts commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020 •

edited

Loading

chrisjsewell commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020 •

edited

Loading

choldgraf commented Mar 17, 2020 •

edited

Loading

mwouts commented Mar 17, 2020

mwouts commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020 •

edited

Loading

chrisjsewell commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020

chrisjsewell commented Mar 18, 2020

mwouts commented Mar 18, 2020

mwouts commented Mar 18, 2020

chrisjsewell commented Mar 18, 2020

choldgraf commented Mar 18, 2020

mwouts commented Mar 18, 2020

chrisjsewell commented Mar 18, 2020

mwouts commented Mar 18, 2020

Add documentation of myst-nb format #458

Add documentation of myst-nb format #458

Conversation

chrisjsewell commented Mar 17, 2020 • edited Loading

codecov bot commented Mar 17, 2020 • edited Loading

Codecov Report

mwouts commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020 • edited Loading

chrisjsewell commented Mar 17, 2020

mwouts commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020 • edited Loading

chrisjsewell commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020 • edited Loading

choldgraf commented Mar 17, 2020 • edited Loading

mwouts commented Mar 17, 2020

mwouts commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020 • edited Loading

chrisjsewell commented Mar 17, 2020

chrisjsewell commented Mar 17, 2020

chrisjsewell commented Mar 18, 2020

mwouts commented Mar 18, 2020

mwouts commented Mar 18, 2020

chrisjsewell commented Mar 18, 2020

choldgraf commented Mar 18, 2020

mwouts commented Mar 18, 2020

chrisjsewell commented Mar 18, 2020

mwouts commented Mar 18, 2020

chrisjsewell commented Mar 17, 2020 •

edited

Loading

codecov bot commented Mar 17, 2020 •

edited

Loading

chrisjsewell commented Mar 17, 2020 •

edited

Loading

chrisjsewell commented Mar 17, 2020 •

edited

Loading

chrisjsewell commented Mar 17, 2020 •

edited

Loading

choldgraf commented Mar 17, 2020 •

edited

Loading

chrisjsewell commented Mar 17, 2020 •

edited

Loading