Have Jedi look at additional paths outside of sys.path for code completion

[AadBh]

When using Python stubs files in Spyder you have to add the stubs to sys.path for Jedi to look for code completion, however this means when the code is run it will try and execute the stubs which do not have any implementation.

In Visual Studio Code there is a setting to have Jedi search in additional locations for code completion but it won't be run when executing the code.

Environment

Spyder 3.3.3

Expected Behaviour

some setting which has paths for jedi to look for e.g.

from jedi import settings
settings.additional_autocomplete_path = ["StubsDirectory"]

Workaround

There is a workaround solution for this by adding the stubs to the sys.path but putting it in unreachable code which won't be executed, jedi will have code completion and it won't try and execute code in the stubs.

if false:
    sys.path.append("StubsDirectory")

[davidhalter]

I believe this is only possible by settings sys_path:

sys_path = jedi.get_default_environment().get_sys_path()
jedi.Script(..., sys_path=sys_path + ["StubsDirectory"])

Is this enough?

Also why are your stubs not on sys path? Did you know that you can put them there using a proper setup.py? Also did you know about the format package_name-stubs?

[AadBh]

The stubs I am using are created from the ironpython-stubs project, they are .NET
stubs to use when using pythonnet to load .NET DLLs in Python. So the stubs look like System.Collections.Generic

If they are on the sys path then it tries to execute the stubs as it takes that instead of what pythonnet does to use the .NET DLL

[davidhalter]

You just have to put it on path with the name package-stubs. Keeping this open, but this is probably not going to change until we have proper support for projects (which is an unofficial API at this moment).

[AadBh]

I tried it and it doesn't seem to work in Spyder.

Is that jedi specific for package-stubs or is it trying to keep inline with PEP-561? I thought that uses .pyi which Jedi doesn't support yet as per #839 .

I think if the .pyi issue is done then that might solve the issue as well and can just have the .pyi stubs in the path hopefully, that seems to work for PyCharm at least.

[davidhalter]

@AadBh You need to use the master branch of Jedi. It's just not released, yet.

[davidhalter]

Jedi 0.14.0 is out. Please try and put your stubs on path and think about if this issue is still needed.

[AadBh]

If I try and use the latest version of jedi with Spyder 3.3.3 it then thinks that Jedi is missing. There is still some autocomplete functionality but still not for the stubs I am using.

I'm also not sure if the renaming the folder works. For Example if I am using .NET System dlls and have .pyi files as System stubs, if I rename that folder as "System-stubs" and add it on to the path then should I get stubs?

The .NET System dll is referenced to the script by using Pythonnent CLR so for example given the following

import clr # pythonnet module to import .NET DLLs
clr.AddReference("System") # Adds .NET System
from System import Double # imports Double from System.dll
Double. # expecting jedi autocomplete

so would having "System-stubs" folder with a file inside called System.pyi which has a stub for the Double class allow autocomplete because that is what I have at the moment but it is not working.

Spyder via Anaconda doesn't appear to be using the updated version of Jedi yet, I'm not sure when they will update. I'm happy if you want to close the issue now and I can re-open it at a later date if the updated Spyder/Anaconda version doesn't resolve the issue.

[davidhalter]

For Example if I am using .NET System dlls and have .pyi files as System stubs, if I rename that folder as "System-stubs" and add it on to the path then should I get stubs?

You have to use System-stubs/__init__.pyi.

It's fine, we can keep the issue open - at least for a while. The request of adding additional paths is kind of another issue anyway.

[kkpattern]

We need this feature too. We have a python project that runs in a python environment that is embedded in a C++ project. And the packages are located in several different directories. Something Like this:

project
    |-subproject_a
    |           |-package_a
    |           |-package_b
    |-subproject_b
    |           |-package_c
    |-common_packages
    |           |-package_d
    |           |-package_e

subproject_a and subproject_b are independent projects but they share some common packages in common_packages directory. For now, when we're working on subproject_a or subproject_b Jedi can't complete for packages in common_packages directory.

YouCompleteMe have a nice .ycm_extra_conf.py that can be used to supply additional paths for completion. SublimeJEDI also has a python_package_paths in settings for this purpose.

However, it will be much better if Jedi can support this itself. For example, python-language-server uses Jedi for completion, but it doesn't support additional package paths. Jedi.vim also lacks this feature. At least I can't find documents about this feature in python-language-server and Jedi.vim. If Jedi can support this feature, those two projects(and many more) will benefit from it immediately.

I looked into the project code, it seems that the project already try to load data from .jedi/project.json, I think it would be very nice if we can supply additional packages paths in .jedi/project.json file.

[davidhalter]

Why don't you properly install those packages as stubs within the system (with pip)?

[kkpattern]

Why don't you properly install those packages as stubs within the system (with pip)?

The python project is embedded in a C++ project. It's a little weird to install those packages within the system. And we don't want to activate a virtual environment just for coding. Especially when you use gvim on Windows.

[davidhalter]

You don't have to install those packages in the system, you could just install them for your user: pip install --user.

[kkpattern]

You don't have to install those packages in the system, you could just install them for your user: pip install --user.

Still, since the python project is embedded, it can never be used directly in normal python environment so it makes no sense to install it in the system or for my user.

Also, if we set the extra paths in a project setting file like YouCompleteMe's .ycm_extra.conf.py without actually installing, it can further reduce the chance that two different projects have packages with the same name.

It's a very handy feature for big and complex python projects. like YouCompleteMe for example. This is how they set packages paths for completing their own project.

[davidhalter]

I see. Thanks for explaining this in detail.

[davidhalter]

This was implemented in the project branch. The branch will be merged for 0.17.0.

Now you can do something like:

project = jedi.get_default_project()
project.added_sys_path = ['/foo/bar']

# or

project = jedi.Project('/my/project', added_sys_path=['/foo/bar'])
jedi.Script(..., project=project)

[davidhalter]

Please let me know if you guys think this API should be improved or if there is a name that is better than added_sys_path.

The paths are added at the end of the sys path not at the beginning, so if you think this is wrong, please also let me know so we can discuss.