Render Elegant Scipy with Jupyter Book

[mwouts]

I'd like to see how we can use Jupyter Book to render Elegant Scipy.

That would be a nice example for mwouts/jupytext#247.

@choldgraf , do you think we could write together a short bash notebook that demoes this? I started drafting one for a Bookdown repo, but maybe this example will be simpler?

Also, we may need here an additional option to allow the execution of Python cells in Markdown documents, even if they don't have the Jupytext header.

[choldgraf]

Cool idea! What are the steps that would need to be taken (at a high level)? Are all the code cells in elegant scipy meant to be executable?

Maybe @jni or @stefanv would be interested in chiming in on this.

[stefanv]

If you succeed, I'd love to include that on our Makefile and put a hosted version on our website. I'm happy to help you understand any of the custom parts of our build, but mainly we just have a few special comments in the Markdown files that we use to identify things like examples.

[mwouts]

Thanks @stefanv for your interest. I think the chances that we get this to work are reasonable. And the example will tell how we can evolve both Jupyter Book and Jupytext to render real and challenging books like yours.

@choldgraf , the Elegant Scipy book is a great example as it is written entirely as Markdown documents which represent Jupyter notebooks using notedown, a predecessor of Jupytext. The repository seems to work very well with Jupytext, at least adding jupytext to the requirements was enough to render the .markdown files as notebooks on binder.

My initial attempt at turning Elegant Scipy into a Jupyter Book is at Elegant_Scipy_as_a_Jupyter_Book.md. A few steps there are not super user friendly yet (especially changing the kernel with jupytext seems buggy), but we will take care of that later on. @choldgraf , what do you think? It's not exactly working (I just get .md files in the _build directory), but maybe we're not that far away?

[jni]

@mwouts I don't understand from your comment whether it is working already?

[mwouts]

No, sorry it is not working yet. I need @choldgraf's feedback to see what is wrong, as I don't have much experience with Jupyter Book yet.

One thing that does work already, is to open the original Elegant Scipy repo with Jupytext on Binder.

Just add jupytext in the requirements, as here, and then you will be able to open the .markdown chapters as notebooks in Jupyter - give it a try here: https://mybinder.org/v2/gh/mwouts/elegant-scipy/binder_jupytext?filepath=markdown

[jni]

Oh wow, I get it now. That's super cool! Time to add a binder link and maybe deprecate the elegant-scipy/notebooks repository!

[stefanv]

The binder output renders the callouts correctly! I think @mwouts is on the right track here. @jni agree that we can use this as a default mechanism.

[choldgraf]

@mwouts - it should be finding the .markdown files in the master branch right now, is it still not working even though you are running from master?

[mwouts]

Thanks @stefanv , @jni. Please let me know if I can help with the binder setup. I'd recommend to check every chapter, as I tested only a few of them. Note that Jupytext expects that every ```python code chunk is executable, is that right in the case of your book? Also I believe that the compilation of the book itself could be done with jupytext --execute, tell me if anything does not go as expected.

@choldgraf , I do think I am using the master branch. Actually the code I used is in that (bash) notebook: https://github.com/mwouts/jupyter-book-experiments/blob/master/Elegant_Scipy_as_a_Jupyter_Book.md . In an ideal world I would have expected the notebook to be runnable on Binder with the bash kernel (there's a link to Binder at the top of the notebook), but apparently the bask kernel has an issue with my 'conda activate' command. I'll see if I can fix that. Unless you see a better way than a notebook to collaborate of the experiment? 😄

Also a specific question I had for you, Chris, is whether I could avoid creating a new folder for the book content. In the case of the Elegant Scipy book, I'd like to determine the minimal addition that can turn the repo into a Jupyter Book. Can I call jupyter-book and tell it that I don't want to copy the chapters, but leave them where they are found?

[mwouts]

@choldgraf , the binder now runs. It was just my conda create which missed a -y. So you can see my exact experiment at https://mybinder.org/v2/gh/mwouts/jupyter-book-experiments/master?filepath=Elegant_Scipy_as_a_Jupyter_Book.md.

At the current stage we have the following issues

• the .markdown extension is not supported
• the content is duplicated to another folder
• the notebooks are not executed
• and if I force the execution with jupytext prior to jupyter-book, I get a FileNotFoundError: [Errno 2] No such file or directory: 'data/counts.txt.bz2' (but that probably just means that I am not executing in the right directory?)

Still I am confident that it may soon start working well!

The binder link is a great place to experiment. However, if you want to edit the notebook and contribute back your modifications, I recommend that you install jupytext and bash_kernel locally, execute python -m bash_kernel.install and then edit Elegant_Scipy_as_a_Jupyter_Book.md in Jupyter. Looking forward to seeing your fixes there!

[mwouts]

I just gave another try to this, and I've reach a point where jupyter-book seems to build the book. This is something already. Now I need to find out how to activate the book on github pages...

[mwouts]

@choldgraf , thanks for your input yesterday. I was able to build the book following your deploy demo.

The book is available here: https://mwouts.github.io/elegant-scipy-as-a-jp-book.

And the actions that led to that are documented here: https://github.com/mwouts/jupyter-book-experiments/blob/8a6491767801c77e0f473246c0bbcdef4fb81716/Elegant_Scipy_as_a_Jupyter_Book.md

The good news is that there are not too many steps there: it's only a matter of

• Installing the book dependencies
• Inserting the kernel in the markdown documents
• Creating the book with Jupyter book
• Updating the ToC and the configuration (maybe these two files could be added to the original Elegant Scipy repo)
• And executing and then publishing the book.

I think that everything there (except the two config files) could go to a continuous integration file, as in Chris' deploy demo.

Now, there is probably much to do yet! Especially,

• One should check that all the book content is there...
• I had no time to update the config.yml file - we should change YOUR BOOK NAME!!
• For some reason Jupyter Book does not like the .markdown extension - I have to change it to .md.

[jni]

@mwouts this is 😍😍😍😍! Thank you! I’m sorry that I haven’t had a chance to look at all the details, but at a first glance it just looks gorgeous! I’m happy to give the rest of the to-do items a whirl from here now that you’ve done the hard work! ;)

[choldgraf]

Beautiful! Really cool @mwouts :-)

A few things:

• Supporting .markdown is easy, I'm +1 on that in Jupyter Book
• We could do a quick "find/replace" for {.callout} and instead add in a tag to that cell to make it pop into the sidebar.
  I've now seen the words "callout", "popout", and "sidebar" all used for this kind of thing...I wonder what you all think would be the most intuitive word?
• We could add an "epigraph" tag to the pages that begin with a quote to make them formatted a bit more nicely.

These could both be semi-automated with something like:

import jupytext as jpt
from glob import glob

path_files = glob('./content/ch*.md')
for ipath in path_files:
    ntbk = jpt.read(ipath)
    for cell in ntbk.cells:
        # Replace the callout cell with a popout tag
        if '{.callout}' in cell.source:
            cell.metadata.get('tags', []).append("popout")
            cell.source = cell.source.replace('{.callout}', '')
        
        # If we find what looks like a quotation syntax in the cell, make it an epigraph
        if "> —" in cell.source:
            if 'epigraph' not in cell.metadata.get('tags', []):
                cell.metadata.get('tags', []).append('epigraph')
    
    jpt.write(ntbk, ipath)

Although it'll probably miss a few things because there aren't explicit
"end-cell" pieces in the page syntax.

What else is missing?! Let's make a wish-list!

[jni]

@choldgraf 👋

I've now seen the words "callout", "popout", and "sidebar" all used for this kind of thing...I wonder what you all think would be the most intuitive word?

Software carpentry and I think O'Reilly both use callout. Who are in the other camp? =)

I'm worried that some of our callouts might be too big to fit nicely in a sidebar...

We could add an "epigraph" tag to the pages that begin with a quote to make them formatted a bit more nicely.

I think that's all of them? But, yes! =)

What else is missing?! Let's make a wish-list!

This one is kinda huge but it would be super-amazing if the functionality from @stefanv's exercise exciser was built into Jupyter Book. To summarise: exercise solutions need to be executed in the context of the original chapter because it uses a bunch of variables defined in the main text. However, we would like to copy the exercises and move the solutions to a new chapter at the end, after they've been executed, so that looking at the solution is "hard". Currently these are marked as <!-- exercise begin --> (exercise text) <!-- solution begin --> (solution text) <!-- solution end --> <!-- exercise end -->, but different tags/metadata could be used.

Also, can Jupyter Book do figure captions? If so, how? Again, for us, @stefanv rolled his own solution based on HTML comments, with possibly my favourite name for a script ever, caption-crunch.py. 😂

[choldgraf]

@jni 👋 👋

Software carpentry and I think O'Reilly both use callout. Who are in the other camp? =)

Ummmm...maybe just me? :-)

That said, I wonder if "sidebar" and "callout" are two separate things. One is more like "I want to highlight this information" and the other is more "I want to highlight this information but in a way that it doesn't break the flow of the main text". Maybe "callout" and "sidebar" are two separate things?

This one is kinda huge but it would be super-amazing if the functionality from @stefanv's exercise exciser was built into Jupyter Book. To summarise: exercise solutions need to be executed in the context of the original chapter because it uses a bunch of variables defined in the main text. However, we would like to copy the exercises and move the solutions to a new chapter at the end, after they've been executed, so that looking at the solution is "hard". Currently these are marked as <!-- exercise begin --> (exercise text) <!-- solution begin --> (solution text) <!-- solution end --> <!-- exercise end -->, but different tags/metadata could be used.

Mmmm, the way I'd do this is probably similar to how you did it. We might be able to build in some (probably hacky) ways to improve the experience with this (e.g., maybe a dont_run tag that would prevent cells from being run?) but I bet it'd require some custom code regardless...

Also, can Jupyter Book do figure captions? If so, how? Again, for us, @stefanv rolled his own solution based on HTML comments, with possibly my favourite name for a script ever, caption-crunch.py. 😂

Right now, no, but if we supported some extra cell metadata it'd be easy. Currently jupyter-book tries to only use tags for triggering certain behavior. You could imagine having some slightly more complex cell metadata (e.g. a "caption: 'mycaption'" key/value would assume that the cell outputs a PNG, and add the 'mycaption' as a div underneath it). This is the kinda stuff I'd want to run by folks in the community to figure out the right pattern etc, so this feedback is helpful!

[mwouts]

@mwouts this is 😍😍😍😍! Thank you!

Well, thank you yourself for sharing the beautiful book! And @choldgraf and the Jupyter Book team have done the hard work - As for myself I've not done much, I was just curious to see how these two wonderful projects could fit together!

Let's make a wish-list!

Great idea! What I am looking for is the following:

• Build the book from a content folder, a TOC file, and the book name/authors (rather than a long config.yml file which mostly contains default values). I would love it if transforming Elegant Scipy into a Jupyter Book was only a matter of adding two or three files to the existing repository.
• Binder integration within Jupyter Book. I see this is documented here, when time permits I will try to activate that option
• Also, the notebooks require a data subfolder. For some reason that folder is copied into the _build directory, and then breaks GitHub pages. Maybe we could ensure that this kind of resource folder is supported by JB? Maybe that will be required for the previous point?

More generally, I think we could have a look at the Elegant Scipy Makefile and integrate into the Jupyter Book project the items that are generic enough.

Also, can Jupyter Book do figure captions?

Aren't captions supported by Jupyter? I think it's a natural request and I'd love to see this in Jupyter Notebook/Lab/nbconvert.

I don't find much material about that, yet the Making publication ready Python Notebooks post by Julius Schulz suggest that one can use the following cell metadata:

"caption": "somecaption",
"label": "fig:somelabel",
"widefigure": true

And I see that the same metadata is also used by ipypublish.

This one is kinda huge but it would be super-amazing if the functionality from @stefanv's exercise exciser was built into Jupyter Book. To summarise: exercise solutions need to be executed in the context of the original chapter because it uses a bunch of variables defined in the main text.

Interesting! Maybe I can suggest another approach here: why not moving the exercises to a separate notebook, that would source the corresponding chapter before executing the exercises? I mean, maybe we could do something like this

import jupytext as jpt
nb = jpt.read('ch1.markdown')
py = jpt.writes(nb, 'py')
exec(py)

and then continue with the exercises?

[jni]

why not moving the exercises to a separate notebook, that would source the corresponding chapter before executing the exercises?

Unfortunately, in some cases, the solution to the exercise is required for the rest of the notebook to execute. Perhaps this is a bad practice, but it's there now...

I don't find much material about that, yet the Making publication ready Python Notebooks post by Julius Schulz suggest that one can use the following cell metadata:

Well, this is not standard in any way currently, as far as I know, but I agree that following prior art is a very good idea. I liked that post back in the day and thought, brilliant, surely this will be stock standard jupyter very soon... 😅

How does one define generic cell metadata in jupytext?

[mwouts]

How does one define generic cell metadata in jupytext?

Oh I see this is not documented, sorry. I'll make sure the documentation includes that in the near future... Anyway:

• code cell tags are like this:
  ```python tags=["tag1", "tag2"]
• other metadata would be like this:
  ```python caption="somecaption" label="fig:somelabel" widefigure=true
  (in both cases: name=<JSON repr>)
• metadata on markdown cells are also supported, using HTML comments - see this example

[choldgraf]

I like the idea of using ipypublish as inspiration here - I'll take a deeper dive into their API and syntax!