Acts weird with not pre-defined magics

[skakker]

Firstly, thanks to the amazing Jupyterlab-LSP community for always being pro-active in helping folks resolve issues quicker :)

Description

While using a "%%" in Python kernel, the "lsp" part of autocomplete returns all the possible results (189 total) irrespective of the token with which its invoked.

Attaching a video here for reference.

Screen.Recording.2021-03-15.at.9.08.02.PM.mov

Reproduce

1. In the Python kernel, just do a %%random_magic \n anythi and invoke autocomplete.

Expected behavior

Like shown in the video's second half, it only shows relevant results. Thats the expected behavior.

Side note: If you can point to some examples on how to define custom magics and their behavior, that will also work. I recently enabled Metals server and if I can specify somehow that metals server should be invoked on %%spark magic, that will also work.
I really tried finding examples and documentation, but was unable to find concrete examples on how to achieve that.

Context

• Browser and version:
• JupyterLab version: 2.2.4
• jupyter-lsp version: 0.9.2
• Language Server: Python/Metals

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output

Paste the output from your browser Javascript console here.

[krassowski]

I think that this is because there is no extractor for a generic cell magic (but there is a code override for a generic cell magic) which leads to the misalignment of positions in virtual documents. I do NOT think that we can create one because we do not know what language will be inside the cell magic. For example %%manim, %%pypy and %%timeit all use Python, but %%R uses R. Probably getting this information from the user would be the best.

Side note: If you can point to some examples on how to define custom magics and their behavior, that will also work

See https://github.com/krassowski/jupyterlab-lsp/pull/553 for an example on how to contribute magics. If you are thinking about a popular magic (say thousands of users or hundreds of GitHub stars), then it would be probably reasonable to include it by default - please consider contributing. For more examples of simple magics see: https://github.com/krassowski/jupyterlab-lsp/tree/master/packages/jupyterlab-lsp/src/transclusions/ipython

I am looking into exposing the regular-expression based transclusions into the user interface (in Advanced Settings Editor). I am slightly reluctant as I do not think that this API is final (there is #347 for example).

Maybe it would make sense to auto-detect unknown magics and prompt user to choose language for those (and in the background to add them into the settings).