Allow specialized plugins to help Jedi understand metaclasses (and other things)

[patrys]

Take Django for example. It uses a lot of metaclasses to support declarative syntax. Would it befeasible for Jedi to let a specialized Django plugin re-write a Model AST node before parsing it to simulate what its metaclass would do?

[davidhalter]

Hmm I would actually say that it should be a Jedi plugin (a library called [jedi-django]) or even namespace packages. However, it's going to be difficult to design an API.

[immerrr]

I actually kept a similar idea boiling in my head for quite a while now. The idea was to cleanly separate things that Jedi can confidently reason about and the rest. For example, jedi can trace the path of a function argument within a function, but it cannot with certainty say where did the argument come from in the first place. That would be an "entrypoint".

The simplest way to think about "entrypoints" was to take a module — since Jedi as of now operates on per-module basis — and try to find all the places that could change about that module, that would be, from the top of my head:

• the name of the module in question, which depends on import path (although it can be considered as a module-accessible variable called __name__)
• imported modules and objects from those modules, which depend on import path and hooks
• arguments of publicly accessible functions (local functions should be OK)
• values of module-accessible variables (external code might inject variables or change existing ones)

Once we identify an "entrypoint", there should be a way for a plugin to help Jedi core to resolve that entrypoint using heuristics or other a priori knowledge. For example, docstring parser that tries to guess types of the variables would become a plugin. There also could be a plugin that would scan a certain directory and subdirectories looking for usages of a function and returning found argument types. Or a plugin that traces the execution of a certain script and returns arguments found that way.

In that scenario, a metaclass might simply become another entrypoint to be handled by plugins.

[davidhalter]

@immerrr Have you seen the evaluate/stdlib.py module? Would that be a good place for entry points? Upon function/class execution? Or would it be better if name resolution already would trigger it?

[immerrr]

Built-in (as in written in C, not necessarily part of the standard library) objects and modules (being objects themselves) are indeed prime examples of entry points. Built-in functions, well, they are interesting beasts.

I thought of entry points as objects available in the analyzed module, but created/modified outside of Jedi's reach, in a space you can think of as "terra incognita" and the entry points would be objects that "enter" the analyzed module from that "terra incognita". So a built-in function most likely being a part of a built-in module or a method on a built-in object, can be an entry point, too. What's interesting is that the function is also a short instantiation of that "terra incognita". That means, the returned values become entry points, and so do after-exit values of mutable function arguments. In rare cases, those built-in functions might also alter module-level objects.

Now that I have written this down, I wonder if tracking separate entry points is actually a good idea...

[immerrr]

It might be better to cover with plugins these patches of "terra incognita", such as:

• pre-module-load stage (roughly, covers sys.path, __name__ and builtins)
• imports which can be considered as invocations of a built-in function (or Jedi could implement introspection of PEP302 import hooks provided during pre-module-load stage and then the hooks themselves would become the "terra incognita")
• post-module-load stage (covers external modules changing values of publicly accessible variables and calling publicly accessible functions)
• built-in functions

metaclasses could also be a special case of terra incognita that changes the scope as a whole.

[davidhalter]

I wonder if tracking separate entry points is actually a good idea...

What do you mean by that?

---

My questions about evaluate/stdlib.py is not if that's enough. It's if the API is enough. By that I mean:

• A Jedi plugin somehow registers itself to Jedi (namespace packages? Is that even possible in a nested form within Jedi?)
• Jedi imports these packages and checks each plugin for entry points
• An entry point would be an executed or evaluated function. Or do you need the plugin to be called already when name resolution is happening?

There is no need special need for built-in functions IMO. I don't get what you mean with pre-module and post-module.

[immerrr]

A Jedi plugin somehow registers itself to Jedi (namespace packages? Is that even possible in a nested form within Jedi?)
Jedi imports these packages and checks each plugin for entry points

I know it is possible through setuptools entry points, but never tried that myself. So you could devise a jedi.plugin.v1 interface (they call it entry point group) and have 3rdparty authors implement that interface in their plugins. If something is missing in the API you just bump the version to jedi.plugin.v2 and hope that the authors of the plugins catch up (and they can provide multiple interfaces for backward compatibility). :)

As for assessing the API itself, things going on during Jedi type lookups are still quite confusing to me, so I don't think my opinion on it is worth anything right now.

[davidhalter]

Ok. Thanks for the information. I think such a system (with versioning as well) would make a lot of sense!

As for assessing the API itself, things going on during Jedi type lookups are still quite confusing to me, so I don't think my opinion on it is worth anything right now.

What is exactly confusing? :)

[asmeurer]

Question: how much of this could be managed by using Python 3.5 type hinting stub files https://www.python.org/dev/peps/pep-0484/#stub-files (assuming Jedi could read them)?

[davidhalter]

Good question. Thank you for bringing this up, since I forgot about those pyi files again. I think there's a reasonable amount of files that could be read like that.

However, I don't think they would help with Django's meta classes, since Django really does type conversions.

[dan-passaro]

The last comment is about 7 months old. Just curious: has Jedi gained any mechanisms to help it understand metaclasses (such as Django models) or is this discussion still up-to-date?

[davidhalter]

The discussion is very much up-to-date and will probably be for at least another year.

[cpdean]

for what it's worth, another project also has dealing with static analysis of meta classes on their roadmap: python/mypy#1240

[sassanh]

Is it still up to date?

[davidhalter]

Yes.

[cpdean]

I'm interested having this as a feature, what would the plugin API require? are there other types of plugins you want to see in Jedi @davidhalter ?

[davidhalter]

Well the problem is basically that it requires a LOT of different ways of injecting stuff. I don't Jedi is even closely ready for this. I think opening Jedi only makes sense once a lot of the type inference stuff is documented and you can inherit from that stuff. Otherwise it just won't work, I think.

[cpdean]

regarding the inference, could jedi use code that something like mypy uses to trace what the types of things are?

Maybe not directly, but they've got type inference written in python so I'm hoping there's some potential there.

[davidhalter]

Probably not, but I've never really looked into it.

[fjsj]

@cpdean I believe inferencing via mypy is possible, but it doesn't have a public API for this yet. Check: python/mypy#2097

[tony]

Jedi may also fail for metaclasses like this:

https://github.com/django/django/blob/2.0/django/contrib/admin/options.py#L98

Try a GoToAssignment of https://github.com/django/django/blob/2.0/django/contrib/admin/options.py#L315

form = self._get_form_for_get_fields(request, obj)

[davidhalter]

metaclass support exists now internally. I have used it for enum.EnumMeta. However it's not "public", so you guys have to wait probably quite a bit until this is public. However since Django is used a lot I'm willing to maintain that inside Jedi.

If anyone volunteers, I'm happy to help out with questions about Django metaclasses.

BTW: I haven't had a lot of questions regarding non-Django metaclasses. So I suppose support for metaclasses outside of Django is not even that important.

[ANtlord]

If anyone volunteers, I'm happy to help out with questions about Django metaclasses.

Maybe I could help you. What are the questions you are talking about?

[davidhalter]

Well, it's just someone has to do the work and I would help out if problems arise.

Also if you're interested in helping out, please look at the code of jedi.plugins.stdlib.get_metaclass_filters and implement something similar to the case of EnumMeta for django.db.models.base.ModelBase.

[ANtlord]

Could you provide me a conception? The fields the method provides is a set of fields for Enum instance if metaclass name is EnumMeta and module is enum. What do filters and what do EnumInstance? I see that it's some sort of context and it inherits the general classes. I don't understand how it provides fields for a Enum instance.

Tell me if I'm out of the question too far :)

[davidhalter]

Contexts are basically Python objects: functions, classes, etc. You can call stuff like context.py__doc__ to get the doc string for a Python object or context.execute() to call a method.

Filters are the way how Jedi infers variables. The easiest example is probably classes. The first filter for a class would be the self name filter (because names defined as self.x = have higher priority than a class variable) after that you would go through the mro and have all the class filters. If you want completions for a context, go through all the filter and instead of filtering one name you just get all the names for a context's filters.

So get_metaclass_filters allows you to insert additional filters for meta classes that have higher priority. Therefore if there is a DictFilter({'foo': EnumInstance(...) in get_metaclass_filters for a specific metaclass, it will make it possible to access context.foo if context inherited from that metaclass.

And EnumInstance itself is just there, because contrary to Django's metaclasses, enum has returns that depend on the assignment:

from enum import Enum

class C(Enum):
    foo = 1

from django.db.models import Model, CharField

class Person(Model):
    name = CharField()

>>> C.foo
<C.foo: 1>      # <--- This is the enum instance
>>> Person.name
<django.db.models.query_utils.DeferredAttribute object at 0x7f833991c9b0>

For Django it's probably first ok to ignore the DeferredAttribute issue and just return str() for Person.name as well as Person().name.

Let me know if you have further questions.

[ANtlord]

Ok, I've got the picture. One more question. How can I test my get_metaclass_filters? I can't find any invocation of jedi.plugins.stdlib.get_metaclass_filters

[davidhalter]

It's decorated by the plugin manager, see jedi.evaluate.context.klass.

Don't worry about where you implement this. You don't need to add another "plugin". This stuff is not documented at all, so just do it in stdlib.

Also if you have further questions, please open a specific issue about Django classes. People are probably watching this issue and reading our kind of offtopic discussion :).

[davidhalter]

This likely won't happen in the near/far future. There is a plugin structure, but it's not public. The reasons to not make it public are:

• There are Python stubs now to get almost all cases working
• The need for plugins is probably really small. Now that stubs exist, you can mostly use those. There's cases where Jedi still sucks. The two most popular ones are probably Pytest Add support for pytest fixtures #791 and Django Django classes plugin #1401. There's open issues for both of those (so that will be part of Jedi).
• Avoiding documentation of all Jedi internals (which is a lot of work, that I don't like doing, at least not unpaid).
• Writing a plugin with the internal API is hard. I don't want to constantly give advice how to do it.
• Internal API would need to be stable to work with the current plugins, refactoring will be way harder.

If you really need plugins for your software, feel free to contact me and open a new issue for your specific case. I would seriously be interested in case studies that are not covered. I don't really know about anything else than Django and Pytest, where we are working on better support.