Token set to an empty string as /servers returned an empty string - why?

[consideRatio]

I think this may not be an issue with this extension, but it may influence people using it. Anyhow, I figure I can write down notes while debugging this here.

The issue

I tried to use this extension on my recently deployed JupyterHub (v1.4.1) with quite recent versions of most things, including jupyterlab==3.1.0a10. It works visually and so, but a web response from /jupyterlab_link_share/servers results in a token set to an empty string which in turn results in a link that doesn't include a token.

Why?

1. Why does /jupyterlab_link_share/servers return an empty string token for me on hub.jupytearth.org while it returns a token on mybinder.org?
   No clue.

2. What code base is responsible for returning a response when visiting /jupyterlab_link_share/servers ?
   It seems like this code:

   jupyterlab-link-share/jupyterlab_link_share/handlers.py

   Lines 6 to 17 in a1bf019

   	from jupyter_server.serverapp import list_running_servers as list_jupyter_servers
   	from jupyter_server.utils import url_path_join
   	from notebook.notebookapp import list_running_servers as list_notebook_servers
   	class RouteHandler(APIHandler):
   	@tornado.web.authenticated
   	def get(self):
   	servers = list(list_notebook_servers()) + list(list_jupyter_servers())
   	# sort by pid so PID 1 is first in Docker and Binder
   	servers.sort(key=lambda x: x["pid"])
   	self.finish(json.dumps(servers))

   That code references notebookapp list_running_servers... Have that function changes between versions? No it had not.

   I wonder if list_running_servers have started returning different details because of a change in JupyterHub's script entrypoint jupyterhub-singleuser has changed between between 1.3.0 and 1.4.1?

   Hmmmm it seems like jupyter --runtime-dir returns a runtime directory path, and within that path I find nbserver-<pid>.json with details representing the JSON structure returned.

   cat $(jupyter --runtime-dir)/nbserver-*.json
   

   What sets this file, and why is it set without a token or not?

Differences observed

jupyter --version differences

jupyter --version and misc	on mybinder.org	on hub.jupytearth.org
ipykernel	5.5.0	5.5.5
ipython	7.20.0	7.23.1
ipywidgets	7.6.3	7.6.3
jupyter client	6.1.11	6.1.12
jupyter core	4.7.1	4.7.1
jupyter lab	3.0.7	3.1.0a10
jupyter-notebook	6.2.0	6.3.0
nbconvert	6.0.7	6.0.7
nbformat	5.1.2	5.1.3
qtconsole	not installed	not installed
traitlets	5.0.5	5.0.5
jupyter-server	1.4.0	1.7.0
jupyterlab-server	2.3.0	2.5.1
jupyterhub	1.3.0	1.4.1

JSON Response from /server at mybinder.org

When pressing the share button I observed a request like this https://hub.gke2.mybinder.org/user/jupyterlab-cont-rlab-link-share-ati8onmk/jupyterlab_link_share/servers?1622512360400 result in the following response.

[
    {
        "base_url": "/user/jupyterlab-cont-rlab-link-share-ati8onmk/",
        "hostname": "0.0.0.0",
        "notebook_dir": "/home/jovyan",
        "password": false,
        "pid": 18,
        "port": 8888,
        "secure": false,
        "sock": "",
        "token": "QkLNLf04RQyJSldyJMzu0A",
        "url": "http://0.0.0.0:8888/user/jupyterlab-cont-rlab-link-share-ati8onmk/"
    }
]

JSON Response from /server at hub.jupytearth.org

When pressing the share button I observed a request like this https://hub.jupytearth.org/user/consideratio/jupyterlab_link_share/servers?1622512267115 result in the following response.

[
    {
        "base_url": "/user/consideratio/",
        "hostname": "0.0.0.0",
        "notebook_dir": "/home/jovyan",
        "password": false,
        "pid": 1,
        "port": 8888,
        "secure": false,
        "sock": "",
        "token": "",
        "url": "http://0.0.0.0:8888/user/consideratio/"
    }
]

ps aux at mybinder.org

$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jovyan         1  0.0  0.0  25928  9360 ?        Ss   01:27   0:00 python3 -u /usr/local/bin/repo2docker-entrypoint jupyter-notebook --ip=0.0.0.0 --port=8888 --NotebookApp.base_url=/user/jupyterlab-cont-rlab-link-share-ati8onmk/ --NotebookApp.token=QkLNLf04RQyJSldyJMzu0A --NotebookApp.trust_xheaders=True --NotebookApp.allow_origin=* --NotebookApp.allow_origin_pat=.*
jovyan        18  0.5  0.1 402240 81532 ?        Rl   01:27   0:10 /srv/conda/envs/notebook/bin/python /srv/conda/envs/notebook/bin/jupyter-notebook --ip=0.0.0.0 --port=8888 --NotebookApp.base_url=/user/jupyterlab-cont-rlab-link-share-ati8onmk/ --NotebookApp.token=QkLNLf04RQyJSldyJMzu0A --NotebookApp.trust_xheaders=True --NotebookApp.allow_origin=* --NotebookApp.allow_origin_pat=.*
jovyan        65  0.0  0.0  20320  3796 pts/0    Ss   01:29   0:00 /bin/bash -l
jovyan       146  0.0  0.0  36092  3316 pts/0    R+   01:59   0:00 ps aux

ps aux at hub.jupytearth.org

$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jovyan       1  0.8  2.0 1211852 326404 ?      Ssl  01:12   0:23 /srv/conda/envs/notebook/bin/python /srv/conda/envs/notebook/bin/jupyterhub-singleuser --ip=0.0.0.0 --port=8888 --SingleUserNotebookApp.default_url=/lab
jovyan     296  0.0  0.3 454444 48908 ?        Ssl  01:13   0:00 /srv/conda/envs/notebook/bin/python -m ipykernel_launcher -f /home/jovyan/.local/share/jupyter/runtime/kernel-de0ba441-a81c-43af-84d3-5734b7101e50.json
jovyan     310  0.0  0.0   6132  4148 pts/0    Ss+  01:13   0:00 /bin/bash -l
jovyan     951 14.5  0.0   6132  4120 pts/1    Ss   02:00   0:00 /bin/bash -l
jovyan    1489  0.0  0.0   7660  3308 pts/1    R+   02:00   0:00 ps aux

[jtpio]

Thanks @consideRatio for the detailed breakdown 👍

a change in JupyterHub's script entrypoint jupyterhub-singleuser has changed between between 1.3.0 and 1.4.1?

Out of curiosity, do you have a link to a diff or PR related to that change? Thanks!

[jtpio]

Also this link share extension was initially made to share links to test RTC on the dev Binder for JupyterLab, which starts a separate Jupyter Server with jupyter-server-proxy:

https://github.com/jupyterlab/jupyterlab/blob/1c8ff104a99e294265e6cf476dcb46279b0c3593/binder/jupyter_notebook_config.py#L39

In that case when running on Binder, there are 2 running servers:

• the front server started by Binder (currently a notebook server)
• the jupyter server started by the config posted above

The extension should list both, but the end user needs the token from the (first) front server to access the instance.

[consideRatio]

Thanks @jtpio for your input! I've made no progress since last but is not done trying.

Out of curiosity, do you have a link to a diff or PR related to that change? Thanks!

These are two PRs that I was thinking could potentially have something to do with this. But I'm not sure at all.

jupyterhub/jupyterhub#3437
jupyterhub/jupyterhub#3347

[fperez]

Just a quick note for context - in a conversation with @ellisonbg about this, he suggested something might be happening by virtue of the Github authenticator we're using where this problem manifests, that has to monkeypatch the base authentication classes in JupyterHub, and changes/eliminates the token generation. Binder doesn't use Github auth, so that difference might be the source of the problem.

Not 100% sure it is, but I wanted to capture Brian's point here in case it helps tracking down the problem...

@yuvipanda might also have some thoughts here perhaps?

[consideRatio]

For reference, these are logs when starting up a user server that doesn't have a token set. My guess is that we observe a difference because of the difference of using SingleuserNotebookApp vs NotebookApp (Or ServerApp).

An important clue to guess this for me has been that @fperez clarified that mybinder.org doesn't start user pods with jupyterhub-singleuser which mean they will run NotebookApp rather than SingleUserNotebookApp, and hence they may not experience the same thing even though they are started by a JupyterHub.

[I 2021-06-14 11:17:01.004 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
[I 2021-06-14 11:17:10.925 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1
[I 2021-06-14 11:17:10.933 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /home/jovyan
[I 2021-06-14 11:17:10.933 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at:
[I 2021-06-14 11:17:10.933 SingleUserNotebookApp notebookapp:2302] http://jupyter-someuser:8888/user/someuser/
[I 2021-06-14 11:17:10.933 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-06-14 11:17:10.954 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds

So, what is the difference between SingleUserNotebookApp and NotebookApp then? I'm not sure, here are some traces of code that may be relevant.

• @fperez also mentioned that GitHubOAuthenticator may be doing things that influences this, and looking at this code I now think that could be technically reasonable.
• A helper function about a token
• The JSON files read by this extension are written by this function in NotebookApp's start function!
• The write JSON file function is called here: NotebookApp.start
• Here is SingleUserNotebookAppMixin.start declared

[ivanychev]

@consideRatio did you manage to make it work in the end? I'm using the latest (1.1.1) zero-to-jupyterhub-k8s helm chart and trying to figure out how to make it work with the newly released RTC in jupyterhub.

The extension indeed returns a link with no token and I could not manage to share any notebook with user is not allowed in the logs.

[steverweber]

not sure if this helps any..

found: https://github.com/jupyterhub/jupyterhub/blob/98ec8991f9ca4f2a95e2714abff1c6d536b20c0a/jupyterhub/singleuser/mixins.py#L361

if you comment out token="" in the SingleUserNotebookAppMixin class I'm now getting a token set when viewing

jupyterlab_link_share/servers however the jupyterhub frontend does not seem to respect or know the token so it does not allow it to bypass auth.

I should add to get that token showing up I first login to jupyterhub with the token='' in the SingleUserNotebookAppMixin class then I jupyterlab > hub contol > stop my server .. then change source so remarked #token='' in the SingleUserNotebookAppMixin and then start the user server again without logging out.

If I have token='' not set in SingleUserNotebookAppMixin im getting an error on login. The error could be the hub proxy not including the token in the url.

[yuvipanda]

In JupyterHub, you can just set the token to be the value of the env variable JUPYTERHUB_API_TOKEN, and that works. Gives you full control over the hub until the original user restarts it.

[dlukes]

I've updated to 0.2.3, but I'm still getting a link with an empty token in a JupyterHub (1.5.0) context. Is there anything else I need to do?

[consideRatio]

@yuvipanda you got this to work. What versions of jupyterlab and jupyterhub was used?

Note that its also relevant to know the version of jupyterhub used in the singleuser-server in this case I think.

[yuvipanda]

jovyan@jupyter-yuvipanda:~$ pip list | grep -P 'jupyter|notebook'
jupyter-client                    6.1.12
jupyter-contrib-core              0.3.3
jupyter-contrib-nbextensions      0.5.1
jupyter-core                      4.9.1
jupyter-highlight-selected-word   0.2.0
jupyter-latex-envs                1.4.6
jupyter-nbextensions-configurator 0.4.1
jupyter-resource-usage            0.6.0
jupyter-rsession-proxy            1.4
jupyter-server                    1.11.2
jupyter-server-proxy              3.1.0
jupyter-shiny-proxy               1.1
jupyter-telemetry                 0.1.0
jupyter-tree-download             1.0.1
jupyterhub                        1.5.0
jupyterlab                        3.2.4
jupyterlab-geojson                3.1.2
jupyterlab-link-share             0.2.3
jupyterlab-pygments               0.1.2
jupyterlab-server                 2.8.2
jupyterlab-widgets                1.0.2
notebook                          6.4.5

is the versions on the user container image. I'm running JupyterHub 1.5 on the hub image as well

[dlukes]

Thanks for those details, @yuvipanda! Here's the diff with the same command on my server:

--- yuvipanda-deps.txt  2021-11-22 16:53:28.306201127 +0100
+++ dlukes-deps.txt     2021-11-22 16:53:48.021584625 +0100
@@ -1,22 +1,16 @@
-jupyter-client                    6.1.12
-jupyter-contrib-core              0.3.3
-jupyter-contrib-nbextensions      0.5.1
+jupyter                           1.0.0
+jupyter-archive                   3.2.1
+jupyter-client                    7.0.6
+jupyter-console                   6.4.0
 jupyter-core                      4.9.1
-jupyter-highlight-selected-word   0.2.0
-jupyter-latex-envs                1.4.6
-jupyter-nbextensions-configurator 0.4.1
-jupyter-resource-usage            0.6.0
-jupyter-rsession-proxy            1.4
 jupyter-server                    1.11.2
 jupyter-server-proxy              3.1.0
-jupyter-shiny-proxy               1.1
 jupyter-telemetry                 0.1.0
-jupyter-tree-download             1.0.1
 jupyterhub                        1.5.0
+jupyterhub-systemdspawner         0.15.0
 jupyterlab                        3.2.4
-jupyterlab-geojson                3.1.2
 jupyterlab-link-share             0.2.3
 jupyterlab-pygments               0.1.2
 jupyterlab-server                 2.8.2
 jupyterlab-widgets                1.0.2
-notebook                          6.4.5
+notebook                          6.4.6

I'm guessing the problem might be that I'm using SystemdSpawner, whereas you're using Docker? The rest of the differences either look like non-essential packages, or I have newer versions.

[dlukes]

I'm guessing if I whitelist the token env var in c.Spawner.env_keep, it might work? Trouble is, I don't know that env var's name. I'll try to figure it out by inspecting JupyterHub's environment, but if someone knows, a hint would be appreciated :)

[dlukes]

JUPYTERHUB_API_TOKEN? (Can't test right away as I currently have users logged in.)

[yuvipanda]

the API token env var is to test manually, the lab extension doesn't need that. I'm not sure what could be happening. If you view source on your jupyterlab page, is there a json blob in the HTML (page config)? If so, does it have a key for hubUser? That's what is used by #21 to determine if it's on a hub, and maybe it isn't being set? That's what @minrk is referring to

[dlukes]

Thanks for the pointers! Well hubUser is a non-empty string set to my username, so that part is alright and the condition used by #21 to determine whether it's on a hub succeeds. However, the token is then fetched by calling PageConfig.getToken, which -- AFAICS -- tries two places to get the token: that JSON blob you mentioned, using the key "token", and document.body.dataset, using the key "jupyterApiToken". In my case, the first one is set to an empty string, and the second one is undefined.

[dlukes]

So whitelisting JUPYTERHUB_API_TOKEN in my jupyterhub_config.py doesn't change anything, the token in the generated link is still empty. But I can at least access it in a notebook via os.environ and manually paste it into the link, which works.

[yuvipanda]

@dlukes ah, you're probably running into the case @minrk mentioned in #21 (comment)

[dlukes]

Oh, OK, thanks :) So it looks like I might get it to work if I downgrade JupyterHub to <1.5? But it also sounds like the entire token/permissions system is still in flux. I think I might hold off on link sharing for a bit more in that case, it's not such a core piece of functionality for me, and revisit it once it settles down a bit.

[yuvipanda]

@dlukes I am super curious why it doesn't work for you, while it does work for me even though we have very similar hub versions...

[dlukes]

Me too! I realize you're also running JupyterHub 1.5. The big difference seems to be SystemdSpawner vs. Docker; I thought whitelisting JUPYTERHUB_API_TOKEN might do it, but it didn't. I'm all out of ideas on what to investigate further at the moment, but if you have any suggestions, let me know, I'm happy to try them when I get a chance :)

[minrk]

The token may be available if you run a notebook-based server in jupyterhub 1.5, while a jupyter-server-based server should (correctly) not put $JUPYTERHUB_API_TOKEN in PageConfig.token.

[JeanMarie2608]

could you be more specific what you mean with "notebook-based server" and "jupyter-server-based server"

[minrk]

Ah, sorry. There are currently two implementations of the jupyter server. The old one, in the notebook package, and a new one, in the jupyter-server package. JupyterHub's single-user server is a 'wrapper' of one of these to integrate authentication with JupyterHub. The behavior is almost the same, but there can be subtle differences (and of course, new features in the new server not available in the old one).

In JupyterHub 1.x, the default behavior of jupyterhub-singleuser is to extend notebook (this changes to jupyter-server in 2.0). If you launch jupyter-labhub as the single-user entrypoint (e.g. c.Spawner.cmd = 'jupyter-labhub'), it will always be jupyter-server.

In both 1.x and 2.x, you can specify which to use with the environment variable:

# notebook, default in 1.x
JUPYTERHUB_SINGLEUSER_APP="notebook.notebookapp.NotebookApp"
# jupyter_server, default in 2.0
JUPYTERHUB_SINGLEUSER_APP="jupyter_server.serverapp.ServerApp"

[dlukes]

In JupyterHub 1.x, the default behavior of jupyterhub-singleuser is to extend notebook (this changes to jupyter-server in 2.0). If you launch jupyter-labhub as the single-user entrypoint (e.g. c.Spawner.cmd = 'jupyter-labhub'), it will always be jupyter-server.

@yuvipanda Maybe that's the reason your setup works while mine doesn't? I'm using labhub.

[yuvipanda]

@dlukes that seems the most likely explanation

[yuvipanda]

This isn't actually fixed.

[jkitchin]

I also have this issue. We are running jupyterhub ((it is the littlest jupyterhub setup) with Google authentication, and when I click the share button it leads to a link like this with no token: https://jupyterhub-dev.cheme.cmu.edu/user/[email protected]/lab?token=

In a terminal, there is a $JUPYTERHUB_API_TOKEN, which works fine if I add it to that url.

Is there some setting that will put hte token in the URL?