Kernelspecs with non-default arguments don't work.

[saisasidhar]

Environment data

• VS Code version: 1.59.0
• Jupyter Extension version (available under the Extensions sidebar): 2021.8.11
• Python Extension version (available under the Extensions sidebar): 2021.8.1
• OS (Windows | Mac | Linux distro) and version: Ubuntu 20.04.02 LTS (WSL2)
• Python version: 3.7.11
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): poetry, same issue with system python environment
• Jupyter server running: Remote

Expected behaviour

According to jupyter documentation kernel.json must contain certain keys and values.

argv: A list of command line arguments used to start the kernel. The text {connection_file} in any argument will be replaced with the path to the connection file.

I expect that my custom kernel.json (language=python) can also be launched in Interactive Window mode of this extension with the following argv.

python main.py --something --something-else -f {connection_file}

Note that {connection_file} argument exists as described in kernel spec.

Actual behaviour

Extension fails with the following notification:

Unsupported KernelSpec file. args must be [<pythonPath>, '-m', <moduleName>, arg1, arg2, ..]. 
Provied python /path/to/main.py --something --something-else --ip=127.0.0.1 --stdin=9058 
--control=9056 --hb=9055 --Session.signature_scheme="hmac-sha256" 
--Session.key=b"d6623136-342f-460e-816c-edc0fcdb5712" --shell=9057 --transport="tcp" 
--iopub=9059 --f=/tmp/tmp-32343hby0MHCz6WSc.json

The extension expects that a python kernel should always be started using python -m module_name which as far as I understand does not adhere to the generic nature of kernelspec definitions.

---

• main.py is just an example to show that I want to use python script rather than a python module
• main.py accepts -f argument as well as --ip, --iopub etc
• main.py script handles --something and --something-else arguments first and then starts an ipython kernel by calling IPython.embed_kernel(cfg).
  • --something* does some network and logging customization I need, and that is the reason why I want to start up the kernel as a script
  • Where cfg is an instance of traitlets.config.loader.Config that contains ip, ports, transport, key etc that the script received from {connection_file} or from command line arguments directly.

I know the embed_kernel() works with this extension because when I symbolically link this main.py file to a python file in venv/lib/python3.7/site-packages/main_link.py directory and use python -m main_link in kernel.json, the extension and interactive window works just as expected.

However, I think it would be a correct implementation if vscode-jupyter extension handles argv as it was defined in a kernel.json instead of assuming all python kernels would be started with python -m module_name.

Steps to reproduce

Assuming jupyter package is installed in a virtual environment.

1. Create a new directory called custom_kernel at ./venv/share/jupyter/kernels
2. Copy the contents of ./venv/share/jupyter/kernels/python3 to ./venv/share/jupyter/kernels/custom_kernel (copies kernel.json and icon*.png)
3. Edit ./venv/share/jupyter/kernels/custom_kernel/kernel.json to have the following
   { "argv": [ "python", "/path/to/main.py", "--something", "--something-else", "--f", "{connection_file}" ], "display_name": "custom-kernel", "language": "python", "metadata": { "debugger": true } }
4. Restart VS Code, create new interactive window and change kernel to custom-kernel

Logs

mainThreadExtensionService.ts:63

[[object Object]]Unsupported KernelSpec file. args must be [<pythonPath>, '-m', <moduleName>, arg1, arg2, ..]. Provied python main.py --something --something_else --ip=127.0.0.1 --stdin=9071 --control=9064 --hb=9063 --Session.signature_scheme="hmac-sha256" --Session.key=b"3a80d972-d49b-4064-b0b2-a3a3750cd411" --shell=9065 --transport="tcp" --iopub=9072 --f=/tmp/tmp-323431mTrva1GuxNz.json
$onExtensionRuntimeError @ mainThreadExtensionService.ts:63
_doInvokeHandler @ rpcProtocol.ts:418
_invokeHandler @ rpcProtocol.ts:403
_receiveRequest @ rpcProtocol.ts:319
_receiveOneMessage @ rpcProtocol.ts:246
(anonymous) @ rpcProtocol.ts:111
fire @ event.ts:577
fire @ ipc.net.ts:513
_receiveMessage @ ipc.net.ts:866
(anonymous) @ ipc.net.ts:705
fire @ event.ts:577
acceptChunk @ ipc.net.ts:286
(anonymous) @ ipc.net.ts:247
(anonymous) @ browserSocketFactory.ts:197
fire @ event.ts:577
_fileReader.onload @ browserSocketFactory.ts:81

mainThreadExtensionService.ts:64

Error: Unsupported KernelSpec file. args must be [<pythonPath>, '-m', <moduleName>, arg1, arg2, ..]. Provied python main.py --something --something_else --ip=127.0.0.1 --stdin=9071 --control=9064 --hb=9063 --Session.signature_scheme="hmac-sha256" --Session.key=b"3a80d972-d49b-4064-b0b2-a3a3750cd411" --shell=9065 --transport="tcp" --iopub=9072 --f=/tmp/tmp-323431mTrva1GuxNz.json
	at f.launch (vscode-file://vscode-app/home/tars/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.1195043623/out/client/extension.js:52)
$onExtensionRuntimeError @ mainThreadExtensionService.ts:64
_doInvokeHandler @ rpcProtocol.ts:418
_invokeHandler @ rpcProtocol.ts:403
_receiveRequest @ rpcProtocol.ts:319
_receiveOneMessage @ rpcProtocol.ts:246
(anonymous) @ rpcProtocol.ts:111
fire @ event.ts:577
fire @ ipc.net.ts:513
_receiveMessage @ ipc.net.ts:866
(anonymous) @ ipc.net.ts:705
fire @ event.ts:577
acceptChunk @ ipc.net.ts:286
(anonymous) @ ipc.net.ts:247
(anonymous) @ browserSocketFactory.ts:197
fire @ event.ts:577
_fileReader.onload @ browserSocketFactory.ts:81

[saisasidhar]

Also I noticed that the file path (/tmp/tmp-32343hby0MHCz6WSc.json in the above example) sent as argument to the default python kernel (-m IPython.kernel) does not exist anymore by the time the kernel starts up. Ofcourse, the extension is sending key, ip, ports etc via args which ensures that the kernel starts up properly.

Is there any particular reason why vscode-jupyter sends individual arguments instead of using a valid existing connection file ?

[rchiodo]

This is a result of our kernel daemon we use to speedup kernel launching (we have a process already running that we pass the arguments to, and it starts ipykernel for us). It's not handling this case and should probably be skipped in this scenario.

The line throwing the exception is here:

vscode-jupyter/src/client/datascience/kernel-launcher/kernelLauncherDaemon.ts

Line 60 in bd63750

const args = kernelSpec.argv.slice();

[DonJayamanne]

@saisasidhar If i get a VSIX ready, are you able to test this out at your end?

[saisasidhar]

Thanks for the update @DonJayamanne
yes, I can test the VSIX on my setup

[DonJayamanne]

@saisasidhar I'm sorry for the delay in getting back to you, attached is the VSIX, please use VS Code insiders to install & test this VSIX
Let me know how this goes.

ms-toolsai-jupyter-insiders.vsix.zip

[saisasidhar]

@DonJayamanne Thank you very much for providing a VSIX file for testing. Unfortunately, I get the same "Unsupported KernelSpec file" error.

Here are the last two stack traces for brevity:

mainThreadExtensionService.ts:63 [[object Object]]Unsupported KernelSpec file. 
args must be [<pythonPath>, '-m', <moduleName>, arg1, arg2, ..].
Provied python /home/tars/workbench/pandora/ipy.py --ip=127.0.0.1 --stdin=9021 --control=9019 --hb=9018 --Session.signature_scheme="hmac-sha256" --Session.key=b"fdef0ff8-297d-459f-8c66-a1e3868fd4d1" --shell=9020 --transport="tcp" --iopub=9022 --f=/tmp/tmp-15208sfM2DW1WJUzk.json

$onExtensionRuntimeError	@	mainThreadExtensionService.ts:63
_doInvokeHandler	@	rpcProtocol.ts:472
_invokeHandler	@	rpcProtocol.ts:457
_receiveRequest	@	rpcProtocol.ts:373
_receiveOneMessage	@	rpcProtocol.ts:295
(anonymous)	@	rpcProtocol.ts:160
fire	@	event.ts:577
fire	@	ipc.net.ts:513
_receiveMessage	@	ipc.net.ts:871
(anonymous)	@	ipc.net.ts:710
fire	@	event.ts:577
acceptChunk	@	ipc.net.ts:286
(anonymous)	@	ipc.net.ts:247
(anonymous)	@	browserSocketFactory.ts:197
fire	@	event.ts:577
E._fileReader.onload	@	browserSocketFactory.ts:81

mainThreadExtensionService.ts:64 Error: Unsupported KernelSpec file. 
args must be [<pythonPath>, '-m', <moduleName>, arg1, arg2, ..]. 
Provied python /home/tars/workbench/pandora/ipy.py --ip=127.0.0.1 --stdin=9021 --control=9019 --hb=9018 --Session.signature_scheme="hmac-sha256" --Session.key=b"fdef0ff8-297d-459f-8c66-a1e3868fd4d1" --shell=9020 --transport="tcp" --iopub=9022 --f=/tmp/tmp-15208sfM2DW1WJUzk.json
at f.launch (vscode-file://vscode-app/home/tars/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.10.1001332376/out/client/extension.js:52)

$onExtensionRuntimeError	@	mainThreadExtensionService.ts:64
_doInvokeHandler	@	rpcProtocol.ts:472
_invokeHandler	@	rpcProtocol.ts:457
_receiveRequest	@	rpcProtocol.ts:373
_receiveOneMessage	@	rpcProtocol.ts:295
(anonymous)	@	rpcProtocol.ts:160
fire	@	event.ts:577
fire	@	ipc.net.ts:513
_receiveMessage	@	ipc.net.ts:871
(anonymous)	@	ipc.net.ts:710
fire	@	event.ts:577
acceptChunk	@	ipc.net.ts:286
(anonymous)	@	ipc.net.ts:247
(anonymous)	@	browserSocketFactory.ts:197
fire	@	event.ts:577
E._fileReader.onload	@	browserSocketFactory.ts:81

VS Code Insiders Version: 1.62.0-insider (user setup)
System: Python project and interpreter located on WSL2
I cross-checked to ensure the VSIX you sent was indeed installed. Here's some information from the extensions page

Released on 11/11/2020, 20:14:18
Last updated 10/12/2021, 11:23:47
Identifier ms-toolsai.jupyter

[DonJayamanne]

Let me have a look, i think the insider version of the jupyter extension wasn't release to the marketplace.

[DonJayamanne]

I'm sorry for some reason the extesnion was't published, and should be in the marketplace now. Please update the Jupyter extesnion installed to the latest (currently latest is 2021.10.1001338521)

Appreciate your patience in this matter and do let me know if this fixes it.

[saisasidhar]

Thank you @DonJayamanne for releasing the insider version to marketplace. The extension is now working as expected, and it is able to start interactive kernel based on my kernel spec.

Just a minor thing I noticed while testing. This insider version does not display names of kernels (defined in the kernel spec file), i.e all kernels in my virtual environment instead have the same name,

[DonJayamanne]

This insider version does not display names of kernels (defined in the kernel spec file), i.e

Please could you file a bug and ping me on that issue.
I'll get that fixed as well. Thanks a lot for your help.
Closing this issue.

[vibhavagarwal5]

Hi, still facing this same issue.
Unsupported KernelSpec file. args must be [<pythonPath>, '-m', <moduleName>, arg1, arg2, ..]. Provied bash --ip=127.0.0.1 --stdin=9027 --control=9025 --hb=9024 --Session.signature_scheme="hmac-sha256" --Session.key=b"ff7757cc-8d70-4b85-9cdb-75065346aed7" --shell=9026 --transport="tcp" --iopub=9028 --f=/tmp/tmp-127568xvBtxbZQUyx.json
@DonJayamanne