Support PEP-0484 .pyi stub files

[patrys]

These are not currently supported but it would be useful to have them.

Support is tricky as Jedi seems to do definition discovery and type discover in the same parsing/evaluation pass. My initial ideas was to load a .pyi file instead of the .py file when looking for a module and it seems to work for type inference but can't possibly work for providing actual definitions as in the stub all of the definitions are dummy ellipsis objects (which means names match but things like offsets won't).

I think it's safe to follow what mypy does and ignore the .py file entirely when doing completion if a .pyi file exists but it's unclear to me how to decouple completion from the definition provider (go to assignment, go to definition etc.).

[davidhalter]

Good points. I'm not sure yet where we want to go with them. I'm against just ignoring the .py file, because that makes it pretty much impossible to use goto_definitions properly. I would probably just add a hook into the function execution logic of Jedi (and maybe one for param inference). That's the biggest problem AFAIK.

Support is tricky as Jedi seems to do definition discovery and type discover in the same parsing/evaluation pass.

What do you mean by that? By the way, I've almost completely rewritten type inference the dev branch. Just saying in case you have looked at master.

[patrys]

My initial idea was to separate the definition lookup from type lookup.

Type lookup could theoretically parse just the .pyi files where available and otherwise use the current parser/evaluator to find where all imports come from.

Definition lookup could theoretically reuse completion info to find which module a particular symbol comes from and only re-parse that single file if all completion had was a stub.

Tricky stuff happens when completions are needed within the same file. I'm not sure what PyCharm does here: should it assume that the stub is up to date and use that possibly missing a signature for a newly defined function? Should it ignore the stub entirely and rely on the evaluator to give it a correct signature? Should we somehow combine both of these relying on symbol names matching?

[patrys]

Another question is what to do with type signature comments such as # type: (int, int) -> str. If I wanted to have them supported, should that be up to the parser (I think that's how Python 3 type annotations are handled) or evaluator? I think these share a fair amount of scope with stub files: they provide an alternative source of signatures for functions and classes that Jedi possibly already has an (in)complete signature based on the inference engine.

[davidhalter]

Are you planning to implement this feature yourself? I'm happy to help you with those questions, because they are really good. However if you're not planning to add this feature, I'll leave them open for a future discussion.

[patrys]

I'm happy to work on this as I've already worked on a number of tools using Jedi's API (as contributions to plugins for Sublime, Atom and most recently VSCode). I would obviously need your guidance but in exchange would be happy to contribute more docstrings for Jedi internals.

[patrys]

Another question: how to store overloaded signatures?

@overload
def foo(a: int) -> str: ...
@overload
def foo(a: str) -> None: ...

[davidhalter]

You need to give me a bit of time, I need to work out a plan first (after the next release).

[davidhalter]

Ok. I suggest we ignore the issue with overload for now. I think that is something that we have to track separately. (Because it may appear in normal files as well?!)

There's a check that looks for the pep 484 parameters when Jedi executes functions. I think for return values this is good enough. For params there's a similar check I think. At both of these places we could just try to lookup a file that has the .pyi name.

I'm not exactly sure where we would need to search for those files.

I think one remaining though would be: It shouldn't slow down Jedi too much.

Let me know if you need to know more.

[gopar]

Any updates on this?

[lazka]

I've added stubs to pycairo recently (the latest release on pypi): https://github.com/pygobject/pycairo/blob/master/cairo/__init__.pyi if someone needs a package to test with

[tristan957]

@lazka thank you for your GTK documentation. It is amazing.

[idchlife]

Wow! Jedi would support .pyi stubs files? This is insanely wonderful.
Seeing how TypeScript ecosystem benefits from actual .ts files and .d.ts (only definition files, adding or replacing type definitions for existing symbols, modules) I imagine that jedi support for .pyi files will fire interest for using and writing types in python community and we will end up with huge typeshed with many types, like typescript has DefinitelyTyped.

Edit: there will be a problem as I can understand from OP post with .py files of existing module and .pyi files of stubs.
So actually there will be less benefit of incomplete .pyi files than lack of them completely, yes?
(if .pyi will not have new variable and it will be inside "real" .py file - jedi will not show new variable in .py file)

[Goom11]

Any updates on this? I would greatly benefit from support for .pyi files. I also have the use case where I have folders with only .pyi files. So go to definition would hopefully go to the .pyi files themselves as there is no .py file to go to.

[davidhalter]

We're actually pretty far on this. Please try the typeshed branch and let me know if there are still issues with it.

Once the typeshed branch is working, we can get stubs working in general, because stdlib stubs are already working on that branch. This means that they should also be working in general, only the lookup is missing on PYTHONPATH.

So I would say this is 80% done, just needs a bit more effort and a merge to master.

[Goom11]

@davidhalter, I verified that I am able to go to definition to .pyi files in stdlib.

I used a simple script.

import asyncio
async def call():
    pass
loop = asyncio.get_event_loop()
loop.run_until_complete(call())
loop.close()

If I go-to-definition on the get_event_loop, it takes me to asyncio/events.py.
If I go-to-definition on the run_until_complete, it takes me to .../stdlib/3/asyncio/events.pyi.

Is this expected behavior or should the go-to-definition on run_until_complete take me to run_until_complete in AbstractEventLoop in asyncio/events.py? Without the typeshed branch, go-to-definition doesn't do anything for run_until_complete. Same for close. Nonetheless, this is likely unrelated to this issue and this behavior is an awesome improvement.

Thanks so much! Let me know if I can do anything to add support for general stubs as that is my primary use case.

[Goom11]

In fact, I would appreciate a code pointer for adding support for general stubs.

[davidhalter]

IMO stubs are not there to get you to the stubs (for goto/completions), but to assist type inference and for builtins. So with that branch you should be able to goto the stub of math.cos, but not to os.walk, because you're mostly going to be more interested in the implementation than its types. However, the result of os.walk() should be the result of the stub.

It might at the moment not be possible to use goto properly in all Jedi plugins with the typeshed branch. I have therefore started davidhalter/jedi-vim#915. If you want to test stubs you can checkout the typeshed branch and copy your stubs to jedi/third_party/typeshed/third_party/2and3.

[patrys]

Sorry, I confused declaration with type definition. Typescript extension actually provides those two where "go to definition" opens a .js file and "go to type definition" the corresponding .d.ts file (the Typescript counterpart of .pyi files) if case of code with external typing. If types are inline, both open the .ts file.

I guess declaration/definition distinction is useful for languages with either header files or where public APIs need to be declared explicitly (like Pascal "units").

Implementation request is for code "specializing" the symbol at hand (extending an abstract base class, implementing a virtual method, implementing the highlighted interface etc.). I think for Python it would be useful to get a list of locations sub-classing the highlighted class or overloading the highlighted method. I'm not sure there's any documentation that describes this in depth other than discussions around the LSP spec mentioning some intended uses or examples of application.

[Goom11]

@davidhalter, could we support stubs for goto_definitions / goto_assignments & autocomplete?

For example, suppose there is a file, helper.pyi, with the contents:

def greeting(name: str) -> str: ...

and main.py with the contents:

import helper
x = helper.greeting("hello")

Facebook Thrift (https://github.com/facebook/fbthrift) is an example of a use case where it generates .pyi files for ease of development.

1. I would like for completions on helper. to provide greeting.
2. I would like for goto_definitions / goto_assignments on the helper and greeting and x to provide something as well.

I'm willing to help in whatever way I can.

cc: @mostafaeweda or @kcaze may be able to answer questions about the differences between the LSP commands.

[davidhalter]

@Goom11 Please try stuff like import os; os.walk on the typeshed branch. If that works, it will also work for other stubs. Currently only the stdlib is supported.

If anything doesn't work for you, please let me know. I'm happy to improve some things.

[davidhalter]

How do you guys want to use stubs? Like foo.py with a foo.pyi file right next to it? I will implement that if that's all. I feel like in PEP 561 there are other approaches as well. Are those all needed for you guys? You may speak now, otherwise it might not make the next release. :)

BTW: @patrys I'm almost done with supporting both goto declaration and goto definition. Still some work is needed, but I'm pretty far.

[patrys]

Two use cases for me:

1. Being able to use .pyi files that came with packages (or with a typeshed project) to tell me whether I use my imports correctly.
2. Being able to validate my own implementation against the .pyi stub sitting next to it (so I can distribute both).

[davidhalter]

So for your use case (1), where are the files? Can you give me an example of the original library and the stubs? If possible also with the github links. I'm still not 100% sure how these work and I would like to have examples (that I can test against).

Typeshed is of course included by installing Jedi.

[tonyfinn]

So one case where I would have liked this was when I was doing some stuff with Kivy a year or so ago. Kivy doesn't provide type stubs and at least at the time was not covered by typeshed. It's also written in many parts in cython. I built some stubs here for the parts I was using: https://gitlab.com/tonyfinn/mypy-types-kivy which were useful for typechecking with mypy but we're not picked up for autocomplete because jedi did not support them. (Kivy itself can be found at: https://github.com/kivy/kivy )

[The-Compiler]

There are two usecases for me:

• Improving what jedi does based on the typeshed project (though at least for the stdlib, I'm not sure if there is much to be improved)
• Using PEP 561-like stub packages (in my case, PyQt5-stubs) for type information for C/C++ extensions.

[Goom11]

My use case is similar to @The-Compiler 's PyQt5-stubs use case. My use case is using PEP 561 stub-only packages for definitions and completions. These .pyi files would not be included in Typeshed but rather the location of the library would be passed via the sys_path argument to jedi.Script.

[hoefling]

My use case is foo.pyi stub right next to a foo.cpython-36m-x86_64-linux-gnu.so extension (or foo.cp36-win_amd64.pyd on Windows if that matters), would it be also covered by the impl?

[Goom11]

I actually also have the same / similar use case as @hoefling . I have a foo.pyi file next to a foo.so file.

[davidhalter]

I guess we can cover your cases as well @hoefling and @Goom11.

Do you have docstrings in any of those files?

Or even more generally: Does someone write docstrings in they stubs?

[fredrikaverpil]

There's a shim called Qt.py which selects either PySide, PySide2, PyQt4 or PyQt5 depending on what is available. This is used as so:

import sys
from Qt import QtWidgets

app = QtWidgets.QApplication(sys.argv)
button = QtWidgets.QPushButton("Hello World")
button.show()
app.exec_()

Because of how Qt.py works, autocompletion/intellisense won't work properly of the selected Qt binding but it could work if sidecar stub files were used.

I tried these stub files (modified PyQt5 stubs to work with Qt.py) and they worked great in PyCharm, but not in vscode. When contacting the vscode devs, they said this was because stubs were not supported by jedi.

@davidhalter Would your additions possibly make this work better now in vscode?

---

@ctismer @cmaureir Are there any stubs bundled with PySide2 which could potentially be used out of the box?
EDIT: awesome, seems the .pyi files for PySide2 is located under venv/lib/python3.7/site-packages/PySide2/*.pyi

[davidhalter]

@fredrikaverpil Not sure if Qt.py is going to work initially, but if it doesn't you can create a separate issue. I'm pretty sure the fix is trivial once PySide2 stubs are working.

[fredrikaverpil]

@davidhalter I am guessing I need to be running a different version of jedi inside vscode (than the one bundled) to test this?
I created, maybe prematurely, an issue here: #1318

PySide2 stubs (via Qt.py imports) works fine inside PyCharm with PySide 5.12.3.

[hoefling]

@davidhalter indeed, I include docstrings in stubs; I use them to generate Sphinx docs for C extension modules with a custom Sphinx extension similar to this one. Not sure if it's a common usecase though.

[cmaureir]

late to the party, but yeah, we are shipping .pyi files since a couple of versions ago. If you find any issue with them, let us know!

[davidhalter]

The whole typeshed/stubs support is now on master. Please test and report errors. Release coming soon.

[davidhalter]

v0.14.0 is released 🎆 In the end stubs were way more work than I originally thought - I ended up rewriting all "PEP 484" related things. It took a year and was probably like 3-4 months of work in 8 hour work days.

Please test and report bugs.

I hope all your requirements are working. Please let me know if they are not. I kind of lost track about all the different use cases.

BTW I usually announce releases on twitter as well: https://twitter.com/jedidjah_ch (no spam to be expected)

[hoefling]

@davidhalter ran my tests now with 0.14 and my workarounds (ugly stuff like monkeypatching jedi.evaluate.compiled.load_module) removed and It Just Works™. This is so awesome. Thank you for the release!

Looking at the code changes, it's hard to miss a new plugin system - are you planning on making it public, possibly with an easier plugging support (e.g. setuptools entrypoints group)? I find it incredibly useful if I can now put my own stuff (like various objects counting to infer probabilities for each completion) in a plugin.

[davidhalter]

Yeah, there's a plugin system that you may hijack if you want, but the API is just not stable at the moment and may change at any time.

I generally intend to release a plugin API at some point, but I'm not really sure, yet what people would use it for. I don't really understand your issues. What would you want to use it for?