Jupyter plugin hides some custom kernelspecs when the executable defined in the argv cannot be found

[iakovn]

Environment data

• VS Code version: 1.79.2
• Jupyter Extension version (available under the Extensions sidebar): v2023.4.1001091014
• Python Extension version (available under the Extensions sidebar): v2023.10.1
• OS (Windows | Mac | Linux distro) and version: Linux Ubuntu 22.04 (same on CentOS7)
• Python and/or Anaconda version: n/a
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): n/a
• Jupyter server running: Local (same container as vscode)

Expected behaviour

All kernel spec visible in JupyterLab interface should appear in VSCode/Jupyter Kernels list as well. Eventually explicit error message (or warning) is shown if kernelspec is incomplete or broken in some way.

Extend documentation in https://github.com/microsoft/vscode-jupyter/wiki/General-overview-of-Kernel-Discovery-&-Execution-in-Jupyter-(&-extension)#1-global-kernelspecs and/or https://code.visualstudio.com/docs/datascience/jupyter-kernel-management#_jupyter-kernels to explicitly state what paths are searched and how broken or incomplete kernelspec are handled.

I was giving up trying jupyter plugin after not being able to use my kernel with it until I realized why it skips my kernel spec.

Actual behaviour

Some kernels are silently skipped during detection.

Steps to reproduce:

Put a kernel spec into ~/.local/share/jupyter/kernels/custom/kernel.json with a command on system path:

{
    "display_name": "CUSTOM",
    "language": "python-custom",
    "argv": [
        "mycommand",
        "{connection_file}"
    ]
}

The kernel will be picked up by JupyterLab and will work as expected.
KernelPicker will not show the kernel in the list.

Resolution on my side was to change "mycommand" into "/full/path/mycommand" and it worked! Thanks for the great plugin!
EDIT: Per discussion below it's language set to be different from python (like "python-custom") that does the trick. With language set to "python", which is the default, the code makes some assumptions that are not valid for custom kernels.

[DonJayamanne]

Thank you for filing this issue and sorry you are running into this.

I've tested this at my end (with the exact same kernelspec json) and it works as expected. please can you test this with the latest version of Jupyter exetnsion
You do not have the latest version.
Also, please can you sahre the logs from the Jupyter extension

• Open your command palette and select the command Jupyter: Show Output
• Copy all of the output and paste that here
• This generate contains errors or warnings when kernels are hidden from the list

[iakovn]

Ok, my bad. It appears that I changed "language" also and that was what affected things. My kernel spec that was not detected:

{
    "display_name": "CUSTOM",
    "language": "python",
    "argv": [
        "custom-start",
        "{connection_file}"
    ],
    "interrupt_mode": "message"
}

Thanks for the tip about "Jupyter: Show Output":

20:50:10.735 [warn] Kernel Spec for 'CUSTOM' (~/.local/share/jupyter/kernels/custom/kernel.json) hidden, as we cannot find a matching interpreter argv = 'custom-start'

The message is a bit cryptic though.
As per: https://jupyter-client.readthedocs.io/en/stable/kernels.html

language: The name of the language of the kernel. When loading notebooks, if no matching kernelspec key (may differ across machines) is found, a kernel with a matching language will be used. This allows a notebook written on any Python or Julia kernel to be properly associated with the user’s Python or Julia kernel, even if they aren’t listed under the same name as the author’s.

It feels weird that the field affects how kernel is getting detected, don't you think? My use case is similar to "docker kernel" as in https://stackoverflow.com/a/63715102, i.e., it is python kernel but not locally installed.

[DonJayamanne]

It feels weird that the field affects how kernel is getting detected, don't you think?

Thanks, I agree.
However we found that most users end up with just python in that path or pyspark or the like, and when using VS Code, such commands are not in the path (env variable) of VS Code, as a result these kernels cannot be launched from within VS Code.

Outside vscode users open terminals and have conda activated or the like, hence those executables are in the current path
But go into vscode they fail, and thats why we hide them.

I understand that this doesn't work for you, i just wanted you to understand why we decided to exclude them (to avoide users frolm running into issues where the kernel might not start)

I'll discuss this with the team

[iakovn]

Thanks for getting back to this!
Just to be clear: I'm completely fine that VSCode requires full path in the kernelspec file. It was just not clear for someone like me trying it out that:
(a) kernel spec was found but hidden [ since I didn't know I should look into the "output" and even then message was pretty cryptic - it should have said that absolute path is required ]
(b) not a word about the need to specify the full path in the docs about "generic kernelspec", nor about specifics for the "python" kernels, nor about potential "hiding" of the kernelspec (I did read all the docs before reporting, sorry if I missed that).

Another way that would work in my case: since I work with devcontainer I'd prefer to be able to explicitly specify the kernelspec(s) that I need instead of autodetect + filter option to exclude the ones that I don't need.

[matthewfeickert]

Like @iakovn, myself (and actually multiple other physicists in my collaboration but I'm the one debugging it on our behalf) are hitting this. We'd like to be able to use VS Code's remote explorer to interact with Jupyter on remote environments but given some requirements for the software that we need in the environment (we need to be able to see and interact with the CernVM File System) we need to mutate the default kernelspec generated from something like

(lcg-example) [23:01] login01.af.uchicago.edu:~ $ python -m ipykernel install --user --name="lcg-example" --display-name="Example LCG view + venv kernel"
Installed kernelspec lcg-example in /home/feickert/.local/share/jupyter/kernels/lcg-example
(lcg-example) [23:10] login01.af.uchicago.edu:~ $ cat ~/.local/share/jupyter/kernels/lcg-example/.bak.kernel.json
{
 "argv": [
  "/home/feickert/.venvs/lcg-example/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Example LCG view + venv kernel",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}(lcg-example) [23:10] login01.af.uchicago.edu:~ $ 

to something that would allow for the environment manipulation suggested by the Jupyter team in jupyterhub/jupyterhub#847 (comment)

(lcg-example) [23:11] login01.af.uchicago.edu:~ $ cat ~/.local/share/jupyter/kernels/lcg-example/kernel.json 
{
 "argv": [
  "/home/feickert/.local/share/jupyter/kernels/lcg-example/startup.sh",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Example LCG view + venv kernel",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}
(lcg-example) [23:11] login01.af.uchicago.edu:~ $ cat ~/.local/share/jupyter/kernels/lcg-example/startup.sh 
#!/usr/bin/env bash

export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
# Allows for working with wrappers as well
source "${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh" --quiet || echo "~~~ERROR: setupATLAS failed!~~~"

lsetup 'views LCG_103 x86_64-centos7-gcc11-opt'

. ~/.venvs/lcg-example/bin/activate

exec python -m ipykernel_launcher $@
(lcg-example) [23:11] login01.af.uchicago.edu:~ $ 

Resolution on my side was to change "mycommand" into "/full/path/mycommand" and it worked!

@iakovn, can you elaborate on this? My experience with the valid kernelspec above is failure with VS Code failing to find the kernel at all.

@DonJayamanne while I appreciate that the VS Code team might not want to make things visible by default, can it be made possible so that there is at least some dev option to still give a full path to a valid kernel?

---

Environment:

• VS Code version: 1.79.2
• Jupyter Extension version (available under the Extensions sidebar): v2023.6.1101941928
• Python Extension version (available under the Extensions sidebar): v2023.12.0
• OS (Windows | Mac | Linux distro) and version: Linux Ubuntu 22.04 (remote server is CentOS7)
• Python and/or Anaconda version: CPython 3.9.12
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
• Jupyter server running: Remote

Ouput after enabling Jupyter: Show Output:

Visual Studio Code (1.80.0, ssh-remote, desktop)
Jupyter Extension Version: 2023.6.1101941928.
Python Extension Version: 2023.12.0.
Platform: linux (x64).
Workspace folder /home/feickert, Home = /home/feickert
23:30:52.840 [info] User belongs to experiment group 'FastKernelPicker'
23:30:53.125 [info] Start refreshing Kernel Picker (1689654653125)
23:30:53.135 [info] Using Pylance
23:30:54.641 [warn] No interpreter with path ~/.local/share/jupyter/kernels/lcg-example/startup.sh found in Python API, will convert Uri path to string as Id ~/.local/share/jupyter/kernels/lcg-example/startup.sh
23:30:54.645 [warn] No interpreter with path ~/example/.venv/bin/python found in Python API, will convert Uri path to string as Id ~/example/.venv/bin/python
23:30:55.194 [info] Process Execution: ~/.venvs/lcg-example/bin/python -m pip list
23:30:56.757 [warn] Kernel Spec for 'Local Test' (~/.local/share/jupyter/kernels/local-test-venv/kernel.json) hidden, as we cannot find a matching interpreter argv = '~/example/.venv/bin/python'
23:31:12.764 [warn] Kernel Spec for 'Example LCG view + venv kernel' (~/.local/share/jupyter/kernels/lcg-example/kernel.json) hidden, as we cannot find a matching interpreter argv = '~/.local/share/jupyter/kernels/lcg-example/startup.sh'
23:31:12.847 [info] End refreshing Kernel Picker (1689654653125)

---

ATLAS Experiment internal reference link: https://atlas-talk.sdcc.bnl.gov/t/running-jupyter-notebooks-within-vscode/445

[iakovn]

Using /bin/bash as the first element in argv and changing "language" to "python-custom" should solve the problem for you as far as I understand.

[matthewfeickert]

changing "language" to "python-custom" should solve the problem for you as far as I understand.

Wow, indeed this works! Amazing and thank you!

Using /bin/bash as the first element in argv

I had previously run chmod +x on my startup script, but you're right that explicitly putting this in the kernel is probably a better approach to be safe.

$ cat ~/.local/share/jupyter/kernels/lcg-example/kernel.json
{
 "argv": [
  "/bin/bash",
  "/home/feickert/.local/share/jupyter/kernels/lcg-example/startup.sh",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Example LCG view + venv kernel",
 "language": "python-custom",
 "metadata": {
  "debugger": true
 }
}

Thanks for your help @iakovn.

@DonJayamanne Are there any official docs on using python-custom?

[matthewfeickert]

EDIT: Per discussion below it's language set to be different from python (like "python-custom") that does the trick

@iakovn Sorry one final question, from this do I correctly understand that python-custom" is just an arbitrary string choice and the only thing that matters for the kernel to get picked up is that "language" is not "python"? That is, "python-custom" isn't something from IPython, correct?

[iakovn]

EDIT: Per discussion below it's language set to be different from python (like "python-custom") that does the trick

@iakovn Sorry one final question, from this do I correctly understand that python-custom" is just an arbitrary string choice and the only thing that matters for the kernel to get picked up is that "language" is not "python"? That is, "python-custom" isn't something from IPython, correct?

That's my understanding, yes. Seems that the plugin has some undocumented heuristics for "python" aiming to automatically handle venvs.

@DonJayamanne would be great if you explain the logic either here or in the docs.

[matthewfeickert]

That's my understanding, yes. Seems that the plugin has some undocumented heuristics for "python" aiming to automatically handle venvs.

Thanks. Yes, from my limited understanding of kernelspecs from the jupyter_client kernel sepcs docs (which is where I've found the best user docs on IPython kernelspecs so far)

language: The name of the language of the kernel. When loading notebooks, if no matching kernelspec key (may differ across machines) is found, a kernel with a matching language will be used. This allows a notebook written on any Python or Julia kernel to be properly associated with the user’s Python or Julia kernel, even if they aren’t listed under the same name as the author’s.

there's no constraint on "language". So we're just use "language": "python-custom" to make a human readable note that we're getting to Python by another means.

But very much agreed that user docs from the VS Code team on this would be amazing!

[DonJayamanne]

Hi everyone, I'm sorry about this issue, I will review the current workflow next iteration.
Perhaps what the Jupyter extension can do is,

• Not hide items that have paths such as /bin/bash or the like
• And if we do hide items (which have ambiguous paths such as python), then we display a more informative message in the logs and how it can be resolved.

[matthewfeickert]

Perhaps what the Jupyter extension can do is,

• Not hide items that have paths such as /bin/bash or the like
• And if we do hide items (which have ambiguous paths such as python), then we display a more informative message in the logs and how it can be resolved.

Thanks for these suggestions @DonJayamanne. I think both of these sound reasonable, but even just documentation on what the expected user behavior is would be great.

[iakovn]

Hi everyone, I'm sorry about this issue, I will review the current workflow next iteration.
Perhaps what the Jupyter extension can do is,

• Not hide items that have paths such as /bin/bash or the like
• And if we do hide items (which have ambiguous paths such as python), then we display a more informative message in the logs and how it can be resolved.

Those are good fixes!
I think that docs are still the higher prio (inform that some kernels may be hidden, point out dependency on python interpreter setting, recommend "show output" to see warnings, etc).

If you can spend more time, then a new setting to explicitly specify kernels (instead of autodetect + hide not needed) would be great. In my devcontainer I need to exclude 5 choices out of 6...

[amunger]

@DonJayamanne - are the original repro steps valid to verify this, and what is the expected behavior now?

[DonJayamanne]

We continue to hide them but have a better message in the logs

[amunger]

It seems to now attempt to start the kernel, and fail:

 kernelConnectionMetadata: Cs {
    kind: 'startUsingLocalKernelSpec',
    kernelSpec: {
      specFile: 'c:\\Users\\<username>\\AppData\\Roaming\\jupyter\\kernels\\incompleteSpec\\kernel.json',
      name: 'incompleteSpec',
      argv: [Array],
      language: 'python',
      executable: 'custom-startup',
      display_name: 'incomplete spec',
      metadata: [Object]
    },
    interpreter: undefined,
    id: '.incompleteSpec.custom-startup.\\.'
  },

[DonJayamanne]

It seems to now attempt to start the kernel, and fail:

@amunger I'm not sure i understand the issue,
is the problem that the kernel appears in the list and it does not work, or that jupyter appears to launch this process and it fails?
Or that we still do not display this in the list?
Also what is the metadta value? or does that not matter?

[DonJayamanne]

oh sorry, we will still display these custom python kernel specs, and if it fails, its on the user, basically you have a bogus kernel spec, and we have no idea its bogus or not.
In the issue you can see users have custom kernel.specs and just because we fail, doesn't mean its invalid, the language can be python but they are not really pyton kernels, e.g. they can be launched through some custom executable.

So its correct to display these, thats whats expected.