How does routing work with JupyterLabHub?

[krinsman]

I have a JupyterLab extension and corresponding notebook server extension which I am trying to use with JupyterLabHub. However, the behavior is substantially different from running JupyterLab on a remote host and using SSH port forwarding. In particular, my extension works in the latter case, but it does not work with JupyterLabHub.

The reason seems to be that the routing mechanisms are confused.

Below I give some details so that the source of my confusion is understandable. My goal was to avoid overloading with unnecessary information, but I apologize in advance if there is still too much.

On the other hand, please also let me know if you require more information from me.

Description of Problem(s):
There are two commands, call them command1 (GET) and command2 (POST).

When I run the JupyterLab extension on a remote host and then open it in my browser via SSH port forwarding (ssh -L 8888:0.0.0.0:8888 remote_host) everything works.

With SSH port forwarding on the remote host, according to Google Chrome, the GET requests for command1 have the name command1 which expands to (according to Chrome when hovering over it in the Network tab of Developer tools) the route http://localhost:8888/command1. Similarly, the POST requests for command2 are have the name command2 which expands to the route http://localhost:8888/command2.

However, when running the extension with JupyterLabHub on a website (with a different remote host) with JupyterHub correctly installed (it was recently upgraded from vanilla JupyterHub to JupyterLabHub), neither command1 nor command2 works. According to Google Chrome:

• there is a 404 error for command1, the name is still command1, but the route is now expanded to https://hubwebsite/hub/command1.
• there is a 405 error for command2, the name is still command2, but the route is now expanded to https://hubwebsite/lab/user/command2.

Question: Why do command1 and command2 suddenly have different "base routes"?

(Not sure what the correct terminology is here -- my point is simply that, given the behavior on the remote host with SSH port forwarding, I would have expected the routes to have expanded to either:

• https://hubwebsite/hub/command1 and https://hubwebsite/hub/command2 OR
• https://hubwebsite/lab/user/command1 and https://hubwebsite/lab/user/command2.

Instead, it's a mix-and-match for some reason.)

Note:
Hypothetically there could be a problem with the authentication/token mechanism I'm using not extending to the second configuration, and the routing is working correctly for command2. However, my understanding is that Tornado describes any 404 errors for POST requests as being 405 errors for security reasons. Also, since the routing isn't working for command1, it seems more likely that this is just a routing problem for command 2, and not actually an authorization problem, as Tornado would have me believe. Yet since I am not sure, I say "Problem(s)" above, not "Problem".

(Additionally still, since JupyterLabHub uses PageConfig from coreuitls successfully, it seems unlikely that the same code/module would fail for authentication with JupyberLabHub.)

Background/Details:
In the Python code for the notebook server extension, there are custom Tornado request handlers (both inheriting from IPythonHandler) for each, Command1Handler and Command2Handler. Command1Handler accepts GET requests, and Command2Handler accepts POST requests.

Here is (essentially) my load_jupyter_server_extension function:

from notebook.utils import url_path_join

def load_jupyter_server_extension(nb_server_app):
    web_app = nb_server_app.web_app
    host_pattern = '.*$'

    def create_full_route_pattern_from(end_of_url_pattern):
        return url_path_join(web_app.settings['base_url'], end_of_url_pattern)

    command1route_pattern = create_full_route_pattern_from('/command1')
    command2route_pattern = create_full_route_pattern_from('/command2')

    web_app.add_handlers(host_pattern, [
         (command1route_pattern, Command1Handler),
         (command2route_pattern, Command2Handler),
    ])

The TypeScript code for the JupyterLab extension part is complicated, so it's not clear if what follows is a sufficient summary. Basically, Ajax is used to make a GET request to /command1 (so Chrome says that jquery.js is the initiator for an xhr type request), while XMLHttpRequest is used to make a POST request to /command2 (so Chrome says that index.js is the initiator for an xhr type request).

The Javascript code of the Ajax request is on line 9600 of jquery.js (according to Chrome). The TypeScript code of the XHttpRequest is essentially the following:

import { PageConfig } from '@jupyterlab/coreutils';
import {  Widget  } from '@phosphor/widgets';
...
class ExtensionWidget extends Widget {
...
private _submit_request(command: string, body: string) {
    let xhttp = new XMLHttpRequest();
    xhttp.onreadystatechange = () => { if (xhttp.readyState === xhttp.DONE) {
        alert("Submitted command"+ xhttp.responseText.toString());}};
    xhttp.open('POST', command, true);
    xhttp.setRequestHeader('Authorization', 'token ' + PageConfig.getToken());
    xhttp.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
    xhttp.send(body);}
... }

Failed Attempts to Diagnose:
(For the website) JupyterLabHub says the following in the console:

jupyterlab-hub: Found configuration {hubHost: "", hubPrefix: "/hub/"}

I don't think the hub prefix was manually set in page_config.json (I am not the site admin, which makes debugging harder for me) or jupyterhub_config.py, so is just the default value.

Clicking the JupyterLabHub 'Control Panel' button takes the user to https://hubwebsite/hub/home. Clicking the JupyterLabHub 'Logout' button (ultimately) takes the user to https://hubwebsite/hub/login. (I am not sure which route/URL it points to initially.)

I tried to compare all of this with the JupyterLabHub code and figure out what the values of hubHost, hubPrefix, and baseUrl are. Based on the URLs/routes I would have guessed hubHost=https://hubwebsite/hub/ and hubPrefix=''=baseUrl. But obviously the console says that it must actually be hubHost=https://hubwesite/ and hubPrefix=hub/=baseUrl.

When logged in (i.e. at the front page of the JupyterLab part of JupyterLabHub), the URL is https://hubwebsite/lab/user/krinsman.

Comparing the URL of the login page with the URL of the logout button displayed in the JupyterHub control panel https://hubwebsite/hub/logout (not the JupyterLabHub logout button -- I can't figure out which URL that button points to) and with the JupyterHub source code, it seems that JupyterHub thinks the "base route" is https://hubwebsite/hub. (I don't say base URL because I think that is /, so that the "root web page/route" is "base route" + "base URL", but really the terminology is unclear to me.)

On one hand, it seems like JupyterLab (also JupyterLabHub?) thinks that the "base route" is https://hubwebsite/lab/user based on the JupyterLab front page URL as well as the route for command2. On the other hand, it seems like the "base route" for JupyterLab is hubwebsite/hub based on the route for command1.

As for what the notebook server extension thinks the "base route" is, or what the value of nb_server_app.web_app.settings['base_url'] is, I have no idea. (It obviously doesn't agree with what JupyterLab or JupyterLabHub or JupyterHub thinks it is though, or else everything would still be hunky-dory like it is in the SSH port forwarding case.)

Even more confusing is that there is a base_url or base_Url or baseURL or baseUrl (I don't remember exactly) variable somewhere in the JupyterHub source code too, but I don't understand what it refers to (except that it apparently can't be the same as the base URL of the JupyterLab or of the notebook server or someone else).

I couldn't think of anything else to check besides this, and still being confused I figured it would be best to post this issue here, if for no other reason so that it can be documented and indexed by Google.

[krinsman]

Here's what seems to be a convincing guess to me right now:

The key/relevant distinction in moving from the SSH port forwarding setup to the JupyterLabHub set up is that the number of remote hosts increases by 1.

Specifically, for the JupyterLabHub setup, there are now two remote hosts, not just one: one remote host running the JupyterHub instance, and another remote host running the JupyterLab instance.

Based on the information written above, as well as experimentation with JupyterHub I did today, it seems most likely to me that the "base route" for the remote host running the JupyterHub instance is "https://hubwebsite/hub/".

My guess then is that the "base route" for the remote host running the JupyterLab instance is "https://hubwebsite/lab/user", but I have less reason to be certain of that.

Anyway, command1 is a valid operation (corresponds to a binary in the path) on the remote host where the JupyterLab instance is located. However, it is not a valid operation on the remote host where the JupyterHub instance is located.

Therefore, my guess is that the problem is the following: the HTTP requests sent via Ajax for whatever reason only connect with/run on the remote host running the JupyterHub instance, while the XMLHttpRequest have the expected behavior of sending the HTTP requests to the remote host running the JupyterLab instance. Therefore, command1 and command2 are being run on two different remote hosts, explaining their different routes. Since they command1 and command2 need to run on the same remote host for things to work, this would seem to explain the observed errors.

Note that this wasn't a problem in the SSH port forwarding case, where there was only one remote host (the same remote host for which command1 is a valid operation), so both Ajax and XMLHttpRequest sent their HTTP requests to it, so the resulting routes looked the same, and command1 was ran on a remote host that could actually execute it.

If this is correct (is it correct based on your experience?), it still seems to leave at least two questions unresolved:

1. Why do Ajax and XMLHttpRequest send their HTTP requests to different remote hosts?
2. What is the "base route" that the notebook server extension sees? Is it "https://hubwebsite", in which case the following changes to the code might ensure that both of the necessary routes are handled:

command1route_pattern = create_full_route_pattern_from('/hub/command1')
command2route_pattern = create_full_route_pattern_from('/lab/user/command2')

Or does the notebook server extension see "https://hubwebsite/lab/user" or "https://hubwebsite/hub/" as the "base route"? (In which case the required modifications of the route patterns would be a little trickier to implement.) Or does the notebook server extension see something entirely different (compared to both of the remote hosts) as the "base route", making it much more difficult to choose the correct routing patterns.

I suppose that the answer to 2. doesn't matter as much if 1. remains unanswered, because even if the notebook extension has the correct routes to handle both command1 and command2, the extension still won't work until both commands are being run on the same remote host (the one where the JupyterLab instance is).

Right now I really don't have any idea how I could change the Ajax code to change where it sends the HTTP requests to. (It's not my choice or preference to use Ajax to send the requests, per se, but it Ajax is what is used by the JavaScript library needed to implement the extension.)

[krinsman]

Since the code in question is public now, I can link to it: https://github.com/NERSC/jupyterlab-slurm

Calls to squeue are being sent via Ajax through the DataTables library, while calls to sbatch (and also scancel and scontrol, but for simplicity I didn't mention them) are being sent via XMLHttpRequest.

_{(The ideal would be in the future to update the ones using XMLHttpRequest to use fetch, but as has been noted before that was something there hasn't been time for me to figure out yet. Changing the other call to use something besides Ajax seems like it would involve forking the DataTables code and doing major surgery on it which of course would be substantially unpleasant.)}

TL;DR: command1 = squeue and command2 = sbatch.

That being said, my impression is that what was originally written above is probably simpler to understand than the issue as it unfolds in the actual code. @blink1073

[blink1073]

I don't have the bandwidth to maintain this library other than bumping the version along. Happy to accept any PRs.