[WIP] notebook as a jupyter server extension

[Zsailer]

This is obviously a massive change and should not be merged without tons of review+testing.

This PR begins refactoring the classic notebook into a jupyter_server extension. This was proposed in a Jupyter Enhancement Proposal here and discussed at the Jupyter Server Workshop.

It uses ExtensionApp that has not yet been merged into the jupyter_server. See this PR for discussion and documentation on the new extension entry point. To test out this PR, you'll need to clone and install this branch of jupyter_server.

There's still work to done to make this transition smooth for the end-user. I'll try to keep this thread updated with progress. In the meantime, try it out and let me know what you think.

Pinging @SylvainCorlay @blink1073 @kevin-bates @lresende @takluyver @rolweber @mpacer

[Zsailer]

Besides removing a bunch of server specific configuration+modules, this chunk of logic is the biggest change. It uses the patterns defined by the ExtensionApp to make the NotebookApp an extension.

[Zsailer]

Also pinging @rgbkrk, @willingc, @ellisonbg, @jasongrout, @minrk, and @ivanov

[echarles]

Can you list (or link to) the commands to install and test this (which jupyter_server branch, how to enable this server extension, how to start the server with the notebook extension...)? thx.

[minrk]

I thought the plan for this was to do this in a separate package, rather than making notebook itself an extension?

[echarles]

@minrk You mean leaving this notebook repo as-it-is and creating a separated repo like jupyter_server_notebook (or jupyter_notebook, expect confusions...) which would contain the extension to jupyter_server?

[rolweber]

At the moment, this PR should be considered a Proof of Concept. It shows that Notebook could build on Jupyter Server and thereby get rid of a bunch of files that currently need to be maintained. But Jupyter Server itself is undergoing changes, and we'll need a few releases to get it solid and stable. So actually merging this PR now would be a bad idea.

During the Jupyter Server Workshop in May, we talked about Notebook and how it could use Jupyter Server. The general feeling was that we should move forward with Jupyter Server, and leave it up to the Notebook maintainers whether they eventually want to build on it, or whether they prefer to keep the original backend implementation that's already here.

To the other participants of the workshop: Please correct me if I got that wrong :-)

[echarles]

The general feeling was that we should move forward with Jupyter Server, and leave it up to the Notebook maintainers whether they eventually want to build on it

It is true that jupyter_server can not force notebook to migrate, but to keep the community strong and self-reinforced, it can be good to have a consensus on where things are going.

Speaking for me, I have spent those last days trying to anticipate the near- and middle- future of jupyter landscape to know on which layer I should build and what I could expect from it.

So why not trying to formalize the expected future landscape? My understanding from the reading so far is a ideal target architecture such as the following (the names are repositories). Whether notebook embraces that kind of architecture is solely the notebook core maintainers decision.

jupyterlab <--- jupyterlab_server <---+
                                       |
                                       +<---  jupyter_server
                                       |
notebook <----------------------------+
(ui + handlers extensions)

(or)

jupyterlab <--- jupyterlab_server <---+
                                       |
                                       +<---  jupyter_server
                                       |
notebook <--- jupyter_notebook <-------+
(ui)          (handlers extensions)

[minrk]

The general feeling was that we should move forward with Jupyter Server, and leave it up to the Notebook maintainers whether they eventually want to build on it

Yes, absolutely! I 100% agree that jupyter-server should happen and ought to be the future default. To me, it's just a question of whether this repo solidifies as the previous generation to be replaced by the next generation, or continues to be updated in active, backward-incompatible development.

You mean leaving this notebook repo as-it-is and creating a separated repo like jupyter_server_notebook

Yes, that's been my understanding of the plan.

So why not trying to formalize the expected future landscape?
My understanding from the reading so far is a ideal target architecture such as the following...

Great idea! For me, it looks like this:

jupyterlab <--- jupyterlab_server <----+
                                       |
                                       +<---  jupyter_server
                                       |
nbclassic <----------------------------+
(ui + handlers extensions)


notebook (as-is, independent, no longer in development)

which is exactly as you have it, just with a fork of notebook called nbclassic (or whatever) that builds on jupyter_server instead of updating this repo in-place. This makes the new backward-incompatible structure opt-in by being a new package, and communicates more clearly that it's a major change that might require updating by extension developers, etc.. It also should significantly improve the maintainability of all packages involved, I think. It also allows the notebook repo that bundles server and client features to more easily enter into a strict maintenance mode, which we haven't managed to enforce, I think mostly because the successors haven't been ready for replacement.

That doesn't necessarily mean we shouldn't update this repo, but it tips the scales for me. I think making this change here would mean that we need members of the jupyter-server team to commit to being active maintainers on this repo, though, to keep up with the new efforts.

[willingc]

@minrk's suggestion makes good sense. nbclassic would be much easier to maintain and will be less likely to confuse existing notebook users since the new issues related to this development would be on a new repo.

[Zsailer]

Wow, thank you everyone for the feedback.

@rolweber At the moment, this PR should be considered a Proof of Concept

Yes, this really is a proof of concept. I wanted to demonstrate how this could affect the notebook. If we decide to go in the direction of creating a fork nbclassic (or whatever name choose), this PR would just port over there.

@echarles but to keep the community strong and self-reinforced, it can be good to have a consensus on where things are going.

Absolutely. And I don't want to add confusion while we're moving towards some sort of consensus. This requires lots of discussion. As @rolweber said, this is a proof of concept.

@minrk whether this repo solidifies as the previous generation to be replaced by the next generation, or continues to be updated in active, backward-incompatible development.

This is the crux. @minrk, thanks for clearly articulating previous discussion about splitting repos. It makes a lot of sense to me. I do think its going to require that we strongly enforce the end of notebook, and I don't think that will be easy. notebook is an iconic repo name, and I'd bet most contributors will go there first to contribute.

I also want to mention another common pattern I've seen from end users:

Many end users launch JupyterLab (or nteract, voila, etc.) side-by-side with notebook. This won't be possible with notebook anymore (under the nbclassic case). We will have to explain to users that they have to install another "classic notebook" project in parallel that works with this thing called jupyter_server. Yes, it's opt-in, but it's still very confusing to an end user.

My hope was that we could keep the notebook repo as the "supported notebook client", and make the jupyter_server split go unnoticed. It would be a major release of the notebook. Users update and the server gets installed as a dependency. Everything just works after that. Users won't realize notebook switched to using jupyter_server. Is this too ambitious? 😃 I realize this is backwards incompatible, but can we address most of those issues with shims or something like that?

(Also, realize that wasn't around for the last Big Split, so I'm speaking out of total ignorance here 😆)

[SylvainCorlay]

notebook (as-is, independent, no longer in development)

This may make things a bit complicated for the case of server extensions that want to support both the notebook server and jupyter server.

For example, the jupytext server extensions checks for actual types with e.g. issubclass(app.contents_manager_class, TextFileContentsManager)... But depending on which version of TextFileContentsManager is imported it may be true or not....

[echarles]

@Zsailer My hope was that we could keep the notebook repo as the "supported notebook client", and make the jupyter_server split go unnoticed.

There is a huge risk to have unexpected side effects (even with extensive test and validation), resulting in user confidence lost and developer burden.

@SylvainCorlay But depending on which version of TextFileContentsManager is imported it may be true or not....

This is why leaving the current notebook repo as-it-is is is the only way to avoid breaking other things. Should we build a nbclassic not 100% backward compatible? Who will use i?

Making nbclassic 100% compatible may still be possible providing proxy handlers (I believe). Existing configuration jupyter_notebook_conf.py backwards compatibility should anyway also be ensured.

@Zsailer I do think its going to require that we strongly enforce the end of notebook, and I don't think that will be easy.

On a technical level, we can make notebook an extension. But if we think that finally this would give more trouble than benefit, and easy way to go forward is to concentrate on the new repos and simply plan maintenance and EOL of notebook in favor of the new stuff.

[Zsailer]

Ok, here is a summary of the two ways forward (feel free to suggest more points if I left anything out).

1. We fork jupyter/notebook to jupyter/notebook_classic (or whatever we want to name it).

   • Two Notebook-based repos would exist.
     • notebook_classic becomes an extension of jupyter_server.
     • notebook continues to provide it's own notebook server.
   • jupyter_server will evolve while notebooks server will not; eventually, notebook will reach EOL.
   • notebook and jupyter_server become incompatible; users must install notebook_classic to have a Notebook interface on top of jupyter_server.
   • If users want a classic notebook interface next to Jupyter Lab, nteract_on_jupyter, Voila, etc., they have to use notebook_classic.
   • We (Jupyter devs) have to keep notebook and notebook_classic in sync manually.
   • notebook is backwards compatible; classic_notebook is forwards-compatible and backward-incompatible.

2. We refactor jupyter/notebook (inplace) to become a jupyter_server extension.

   • A single Notebook repo would exist (notebook).
   • The old notebook server would go away; jupyter_server becomes a dependency.
   • Less confusing for maintainers and contributors.
   • Users run notebook next to Jupyter Lab, nteract_on_jupyter, Voila, etc. (no extra install is needed.)
   • notebook becomes backwards incompatible (many traits change class names); may be risky for users? Will need extensive testing.

I'd love to make a decision on this, so we can move forward. I'm happy with either outcome; I'm more interested in progressing the jupyter_server.

If we decide to make a notebook_classic (or nbclassic) repo, we need to propose it to the steering council and ask them to create the repo under the jupyter github org.

Let's vote with emojis here :)

🚀 = Option 1
👀 = Option 2

[rolweber]

I still don't see what a second repo achieves, that a separate branch in this repo couldn't.

[lresende]

I still don't see what a second repo achieves, that a separate branch in this repo couldn't.

Same here. I believe that the best option, and even to avoid confusion, is to have one repository for notebook. As the split will be a new major release, we can treat this as Python 2.x support, where if you want the fully monolithic Notebook + Server in one package use NB5.x otherwise use NB6.x or whatever release number we decide to have.

If we decide to have two notebook repositories, I believe that this will cause confusion, and will make things harder to maintain as we will have to try to keep fixes in sync (e.g. security, and blocker issues) between jupyter_server and this other repository.

As for issues, where users might come and create an issue in one of the new clients, where it actually belong to the server, Github now offers a way to transfer issues between repositories and we should leverage that.

[kevin-bates]

When I read the two options, this from Option 2 struck me as a no-brainer against it...

• The old notebook server would go away; jupyter_server becomes a dependency.

I took this to mean a hard cutover, which I oppose.

My brain still has not baked in the power of git and branching, etc. That said, I believe having two repos is actually less confusing, particularly to our users (and potential contributors). It creates a clear line of demarcation. I also believe docs will be confusing if we use a shared repo since the primary linked docs equate to "stable" - so I'd be confused immediately. Is that 'master' or '5.x'? And what is 'master' - NotebookClassic or NotebookAll?

Regarding security or blocking issues, these are always 'all hands on deck' events and the community will make the necessary changes - whether that's in multiple branches or multiple repos. Those events are infrequent.

Given we're entering a world of a physically separate (and somewhat hidden) server, there will be a myriad of server issues opened against the various front-ends (much like kernel-specific issues in Notebook). However, if we used a shared repo, we cannot (potentially) forward Notebook issues to jupyter_server until we're absolutely sure which notebook "client" they're opened against. If the "clients" are visibly different entities, I believe that problem will be less severe since notebook-related issues will tend to land in the correct repo. In short, having a single repo will always require further qualification - which won't necessarily be easy for users to determine.

Other naming ideas for the Notebook classic front-end ... NotebookFE, NotebookClient. Of these, I prefer NotebookFE since it seems less ambiguous and kind of new sounding.

I also want to give a shout out to @Zsailer for driving this effort - thank you!

[Zsailer]

Thanks, @kevin-bates :)

It looks like we still haven't reached consensus so far.

We may need to pose this dilemma to a broader audience. I don't think this issue is getting a lot of traffic outside of us. Maybe we post in discourse and the mailing list?

The challenging thing here is that this is a deeply technical question that affects a large number of less-technical users. We want them to vote and be informed about the consequences without overwhelming them with the details. I'm not sure how to do that effectively.

[minrk]

I still don't see what a second repo achieves, that a separate branch in this repo couldn't.

Branch vs repo doesn't make a huge difference, the main question is whether this package is changed in a major, backward-incompatible way, or kept stable. If the branch created a different package, it would be the same as making a new repo, but I think more confusing because now one repo contains the code for two packages.

The main question is whether we want to bring existing folks using the notebook package into the new way of doing things (including forcing upgrades as dependencies/apis change), vs letting them stay with the 'old' way until they opt-in to the new way by installing nbclassic (or just jupyterlab) instead. From a code standpoint, there isn't really a difference. Just what things are called, and the upgrade path for users / maintenance strategy for us.

Some corrections for @Zsailer's summary (thanks for driving this!):

notebook and jupyter_server become incompatible

I would say they are independent. You can definitely have both installed at the same time, they just wouldn't interact with each other. To be clear, only the fork approach allows one env to have the new way and the old way present at the same time.

If users want a classic notebook interface next to Jupyter Lab, nteract_on_jupyter, Voila, etc., they have to use notebook_classic.

To be precise, this is for a single running server with multiple frontends, not an environment. Having notebook in the same environment would be AOK, but if you want to launch one server with multiple frontends available at the same time, that would be notebook-classic for the classic frontend.

We (Jupyter devs) have to keep notebook and notebook_classic in sync manually.

Yes, though I think a contingent for this proposal is that the notebook package does not move forward. It is semi-frozen and never receives new features, only security updates. If the notebook package is to be actively developed, I think we should not go for the fork; that's only if we want to continue with the idea of the classic notebook being in a "maintenance mode".

To me, the big question is: is the legacy notebook package actively maintained and continuing to make backward-incompatible changes or in a 'maintenance mode' with developer resources focused elsewhere. If this repo becomes active again (it is not right now) by making these huge changes in-place, then the easiest way forward is to make notebook depend on jupyter_server (option 2), eliminating the duplicate server implementations, to minimize the backports that @lresende pointed out. If it continues to be in 'maintenance mode', then I think the least disruptive option is to make a new package for the frontend-only (option 1), avoiding the need to update this repo with the deprecations, etc. that are required for a smooth transition to backward-incompatible changes of widely used software. The fork also gives more freedom to jupyter-server to make incompatible changes that wouldn't otherwise be acceptable for an in-place upgrade of notebook.

[blink1073]

I'm with Min here. I'd prefer a clean break and a conscious migration for users. However, there are now a large amount of users who expect notebook and lab to interop cleanly, so we'd need to give clear advice to those folks.

[rolweber]

@minrk Thanks for the clarification, that makes sense. From this point of view, I'd say the best way "forward" is to neither fork nor change notebook, but to leave it in maintenance mode as it is.

It's not the idea of jupyter_server to revive development of notebook and create new features in the frontend, which might give users an incentive to actively switch. And there's a ton of tutorials and blogs out there that currently tell users to install notebook. Users will just keep coming to the classic notebook. If it works for them, why should they switch? Without adding new features, the only way I can see a fork getting a relevant user base is if we could convince Anaconda to include the fork instead of the classic notebook in its releases.

I see the sweet spot of jupyter_server as a dependency for projects that need a backend with kernels, but that don't want to depend on notebook with its classic frontend. We don't have to fork notebook to provide that alternative, nor do we have to update it. If the repository has effectively become inactive, let's just keep it that way. The idea to reduce maintenance effort for notebook by removing the backend code makes no sense if the maintenance effort is lower than the effort for replacing the backend.

[minrk]

@blink1073 that's an excellent point. I think the existing jupyterlab+nb deployments that do exist are going to need to have some updates, and we should be conscientious about having a good upgrade story there as JupyterLab switches over to jupyter-server. Let's try to keep that use case in mind as we move forward to make sure we aren't dropping users through the cracks.

Users will just keep coming to the classic notebook.

Absolutely. No need to upgrade to new software if you don't need new features.

Without adding new features, the only way I can see a fork getting a relevant user base is if we could convince Anaconda to include the fork instead of the classic notebook in its releases.

I view the nbclassic jupyter-server-extension as something more for hosted deployments like JupyterHub once they switch over to JupyterLab as the primary UI, but have reasons to keep legacy UI available for a smoother transition of longtime users as it will be the way to have multiple UIs with one server instance. I don't imagine it would get or need a whole lot of use from desktop users.

If we do come to a consensus that this repo is in maintenance mode (I don't want to presume that this has been established yet), then the main action items on this repo are to clearly state the no-new-features policy and direct folks to the new repos for further development. We'll also have to figure out the right way to resolve what's in master now.

[Zsailer]

Ok, I'm with @minrk (and others) here.

I propose that we create a new repo under the jupyter org called something like notebook_classic or notebook_extension. This will start as a fork of the notebook repo. I'll resubmit this PR over there, stripping the repo of server-specific logic and refactoring the notebook client to use jupyter_server's new ExtensionApp API.

Then, I think we clearly state the no-new-features policy (maybe in a pull-request template and the main README).

What does everyone else think about that idea?

@minrk can you request permission to create this new repo from the steering council?

[ellisonbg]

@Zsailer thanks for working on this. The main parts of the decision that come to mind for me are:

• I don't see any way we can make a decision that disables users from running the classic notebook alongside (same server) other frontends (lab, nteract, viola). That will create a lot of pain, especially for organizations running Hub that need to support both for a period of time.
• What is the maintenance load? In spite of the amazing hard work of the classic notebook maintainers, we are already under-resourced in this area. Taking steps that increase this load are going to be painful to absorb.
• What is the migration path for users? There are different types of migration pain: 1) no bug fixes or security patches on the old version, 2) breakage of the old version, 3) time needed to port things to the new new solution, 4) confusion. I doubt there is a path that minimizes all of these things at the same time, but we should be aware of tradeoffs and think about ways to mitigate the pain points we choose to live with.

I am not sure how these points translate to the options on the table.

[echarles]

@Zsailer Built your branch and jupyter server returns a simple HTML page with A Jupyter Server is running. - Any URL I should invoke to get the list of notebooks?

[Zsailer]

@echarles Oops, yes. Change to /tree. This branch doesn't have the redirect setup yet. Thanks for reminding me!

Let me know if this doesn't work.

[echarles]

@Zsailer oh so obvious, I just have guessed that... Good news, notebook can use the latest jupyter-server master! Notebooks are listed and cells can be executed.

[echarles]

@Zsailer jupyter_server should be listed in install_requires in setup.py.

[Zsailer]

I have a couple of announcements for people following this thread.

1. We've created a simple RoadMap for the Jupyter Server. That roadmap is still up for discussion and review. Feel free to raise concerns there. There are some references to Notebook 7.x in that roadmap.
2. I've created Zsailer/nbclassic. This is a thin package that provides the Jupyter Notebook UI on top of the Jupyter Server. It leverages the ExtensionApp API in jupyter_server to make a classic Notebook server extension. You can install this package side-by-side with jupyter/notebook and it will use the Javascript and HTML templates in jupyter/notebook to serve the classic notebook UI.
   • You can install from PyPI today, pip install jupyter_server notebook nbclassic.
   • And run by calling jupyter nbclassic and going to http://localhost:8888/tree?token=

[ellisonbg]

I think the nbclassic idea is brilliant!

…

On Thu, Oct 24, 2019 at 4:15 PM Zachary Sailer ***@***.***> wrote: I have a couple of announcements for people following this thread. 1. We've created a simple RoadMap <https://github.com/jupyter/jupyter_server/issues/127> for the Jupyter Server. That roadmap is still up for discussion and review. Feel free to raise concerns there. There are some references to Notebook 7.x in that roadmap. 2. I've created Zsailer/nbclassic <https://github.com/Zsailer/nbclassic>. This is a thin package that provides the Jupyter Notebook UI on top of the Jupyter Server. It leverages the ExtensionApp API <https://jupyter-server.readthedocs.io/en/latest/frontends.html> in jupyter_server to make a classic Notebook server extension. You can install this package side-by-side with jupyter/notebook and it will use the Javascript and HTML templates in jupyter/notebook to serve the classic notebook UI. - You can install from PyPI today, pip install jupyter_server notebook nbclassic. - And run by calling jupyter nbclassic and going to http://localhost:8888/tree?token= — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4653?email_source=notifications&email_token=AAAGXUDS5ZJCT4CBYD477UTQQIUAXA5CNFSM4HSQVME2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECGWTUA#issuecomment-546138576>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGXUCR7ZAHAFHVCOHQ2M3QQIUAXANCNFSM4HSQVMEQ> .

-- Brian E. Granger Principal Technical Program Manager, AWS AI Platform ([email protected]) On Leave - Professor of Physics and Data Science, Cal Poly @ellisonbg on GitHub

[blink1073]

Closing in favor of the nbclassic approach. Thanks @Zsailer!