JEP: Add Jupyter Book as subproject

[choldgraf]

This JEP proposes incorporating Jupyter Book as a Jupyter sub-project.

See the pre-proposal issue below for conversation around this:

• Pre-proposal issue: Pre-proposal: Incorporate jupyter-book as a Jupyter sub-project #122

Voting from @jupyter/software-steering-council

• @gabalafou
  • Yes
  • No
  • Abstain
• @ivanov
  • Yes
  • No
  • Abstain
• @JohanMabille
  • Yes
  • No
  • Abstain
• @jtpio
  • Yes
  • No
  • Abstain
• @marthacryan
  • Yes
  • No
  • Abstain
• @martinRenou
  • Yes
  • No
  • Abstain
• @minrk
  • Yes
  • No
  • Abstain
• @rpwagner
  • Yes
  • No
  • Abstain
• @SylvainCorlay
  • Yes
  • No
  • Abstain
• @vidartf
  • Yes
  • No
  • Abstain

[choldgraf]

Is there any discussion or questions that we need to answer to move this forward? How can we make progress on this?

[JohanMabille]

This JEP should be discussed during the SSC working hours today (according to the triage last week). I think it has been opened long enough so that we can move it to the voting phase.

[JohanMabille]

This JEP has been discussed in the pre-JEP issue, there were Q&A sessions during the SSC working hours, it's time to move to the voting phase.

[jasongrout]

Since adding a subproject to Jupyter requires both SSC and EC approval (as Fernando mentioned here), we discussed this briefly in the EC meeting today. Since the SSC is already voting on this proposal, we (the EC) will wait to see the result of this vote. As a next step, if this is approved, the EC asks that a PR be opened to the governance repo adding the subproject to the official list of subprojects, and the EC can officially vote on the subproject addition in that governance PR.

[choldgraf]

I marked @minrk as "yes" because he approved the PR. I think only @gabalafou remains.

For the decision, am I correct in my understanding from the governance docs that this only requires a majority vote from the SSC? This is what I'm basing this on:

https://jupyter.org/governance/decision_making.html#required-aspects-of-decision-making

If that's the case then we can move forward with the steps that @jasongrout recommends

[choldgraf]

Since there is general consensus so far (and one "no vote"), I went ahead and created a PR so that the Executive Council has something to vote on if they decide that this conversation can be considered "decided":

• Add Jupyter Book to list of sub-projects governance#229

[gabalafou]

voted, sorry for the tardiness!

[rowanc1]

cc @ellisonbg from your question here jupyter/governance#229 (comment)

[jasongrout]

Since the vote here is unanimous for the SSC, and is unanimous over at jupyter/governance#229 for the EC, I think the motion passes, and Jupyter Book is now our newest subproject. I merged jupyter/governance#229, but I'll leave it up to someone in the SSC to merge this PR.

Congratulations!

[choldgraf]

Wow, so exciting! Thanks everybody 🙂 it has been a long road back home for the jupyter book project!

[SylvainCorlay]

Congrats @choldgraf @rowanc1 and everyone involved.

[westurner]

Jupyter Book is now forking and eliminating support for Sphinx, Sphinx extensions, and RestructuredText in Jupyter Book 2.

Jupyter Book 2 requires installing NodeJS to build Markdown (to PDFs).

It was already possible to use various non-latexpdf sphinx.builder Builders to build PDFs from Sphinx docs in ReStructuredText and Markdown.

This fragments the existing open ecosystem, selfishly foregoes support for lots of already-done ReStructuredText docs, and requires rewriting every Sphinx extension from Python to JS.

There is a open core vendor with a product built on myst-md, which they currently contribute to, which wants this.

Jupyter Book 2 certainly has the right to port away from Sphinx and all the code they've now copied. Jupyter Project certainly has the right to support whichever projects, regardless of fragmentation by (later) 'hostile' forks/clones/ports and open-core interests.

I'm obviously late to committee on this.

MyST Markdown (myst-md) was created to implement the Roles and Directives features of ReStructuredText (docutils).

nbsphinx also allows for embedding notebooks within docs, and roles and directives docs within notebooks.
nbsphinx implements support for ReStructuredText (and Markdown) and Sphinx Roles and Directives in Jupyter Notebooks with Sphinx and docutils, but does not have Jupyter in the name?

Jupyter Notebook actually originally supported RST in Notebooks, but the ReStructuredText cell type was removed early on by @takluyver IIRC, IIRC due to there having been no good way to prevent arbitrary file inclusion and RST injection with the .. include: directive and similar (which is also possible with the %pfile magic command or read() or MyST-MD given no OS process isolation with e.g. containers and VMs, which are more mature these days with e.g. repo2jupyterlite).

Docutils (@goodger) was moved from SourceForge to GitHub in 2015; but they didn't import Issues like @chrisjsewell's /docutils;

• https://github.com/docutils/docutils
• https://github.com/chrisjsewell/docutils/issues

So, Jupyter in this context means backward-incompatible fork of docutils and sphinx and restructuredtext that doesn't support Python or RST anymore?

It's too late to ask for justification for backward-incompatibility and rework and fragmentation: https://executablebooks.org/en/latest/blog/2024-05-20-jupyter-book-myst/

Which of these things are within the scope of the Jupyter Project and/or the Jupyter Book project, or does there need to be another "Jupyter Book" -like project with a different name? :

• Maintain docutils (@goodger (Dave Goodger))
• Port docutils for performance
  • "Port Docutils and Sphinx to Rust or Go for performance or WASM, implement myst-parser for better Markdown support in Sphinx, and maintain backward compatibility with Sphinx extensions written in Python," (just like sphinxcontrib-bibtex,)
    • @ericholscher at @readthedocs can probably justify supporting such an effort for net cost reduction
    • (edit)
    • existing plugins
      • https://pypi.org/search/?q=sphinxcontrib
      • https://github.com/sphinx-contrib
      • https://bitbucket-archive.softwareheritage.org/projects/bi/birkenfeld/sphinx-contrib.html
    • porting for performance stories
      • pydantic ported to Rust from v1 to v2 and maintains v1 Python API compatibility
        • https://docs.pydantic.dev/latest/migration/#patterns-regex-on-strings
          • import re, regex -> rust regex crate regexes (without backreferences,)
• Maintain sphinxdoc (@birkenfeld (Georg Brandl))
• Add myst-parser support to sphinx
• Rewrite docutils and sphinx and forego backward compatibility
• Port tests from Docutils and Sphinx to myst-parser, myst-md, and myst engine, and Jupyter Book 2, and myst-nb
• Maintain myst-md
• Maintain jupyter-book

How can Jupyter users continue to use myst-parser with existing Sphinx docs?

[ivanov]

@westurner thanks for taking the time to write down your assessment. I am not sure if commenting on this proposal vote from half a year ago will give it the best visibility, but unfortunately there is not an obvious "better" place that comes to mind, either, as Jupyter has struggled for several years now in engaging both users and contributors.

[Zsailer]

Yes, thank you @westurner. Now that the Jupyter Book Team is formed, I would suggest using their team-compass page to raise this issue: https://github.com/jupyter-book/team-compass

In principle, this is a good place to raise issues to a subprojects team members who are ultimately responsible for shepherding the project.