Support metapogramming-style magic (Metatypes?)

[rtpg]

I'm sorry, the following will be vague and could be totally out of scope:

In Django if you do

class C(Model):
    name = CharField(max_length=10)

In this situation, in the final run, you have:

• C.name raises AttributeError
• the type of C().name be string

This situation is a bit complicated, but really common for some of the more magical parts of python that use metaclasses.

My understanding is that the current philosophy on typing relies on a declarative view of things. But wouldn't it be neat if users could hook their own code into the type-building process?

For example, here:

class ModelMetaType(MetaType):
    def contribute_to_type(self, typ, field, field_name):
          # if we're working with a Django Field
          if isinstance(field, type(Field)):
               # add the python type to the instance type of the class
               # but do not modify the class type
               typ.instance_type.fields[field_name] = field.python_type
          else:
               # simply add to the class type
               typ.fields[field_name] = field

class Field:
   pass

class CharField(Field):
   python_type = string

class Model(metatype=ModelMetaType):
    pass

class C(Model):
    a = 123
    b = CharField(max_length=4)

# roughly
type(C).fields == {'a': int }
type(C()).fields == {'b': string}

I think building something like this could potentially be game changing for the future of typing in Python. Totally unsafe, but are the advantages worth the safety loss?

Pros:

• Many type system requests could be polyfilled through this contribute_to_type mechanism. Things would no longer always need to go through "PEP -> mypy/PyCharm " to be tried out.
• Barrier to entry for implementing new features through this is lower than working on a solver (maybe)
• Pretty general interface into the workings of a type checker

Cons:

• arbitrary code execution during the type checking process. This sort of mechanism rules out most "side effect-free" or "import free" ways of implementing the PEP
• There might be too many edge cases breaking safety for this to be worth it?

For example, if you have

class C(object):
    if six.PY2:
       a = 1
    else:
       b = 1

then it's not sure what contribute_to_type will execute on. Though this problem is also present in checkers generally, so maybe this isn't a change of situation.

There's a bit of trickiness regarding circular imports as well. If you execute contribute_to_type but the field type is not available yet, how do you resolve that?

You could imagine something like:

if isinstance(field, NotYetAvailableType):
     yield field # would return once the field becomes fully available

but that won't fully solve circular dependencies.

---

based off of my understanding of how the common checkers work, this would be a change that is internal to the "determining type from class definition code"-part of the code: the type inference/checking code itself shouldn't need to be changed.

[gvanrossum]

Looks like this is (yet another) a perfect use case for python/mypy#1240 which exhorts us to design and implement a plugin system for mypy. I think you're on to something with your suggestion on how this could be implemented. Would you care to think some more about the simplest implementation of your idea?

[rtpg]

I'll write something up! I have a good idea on how we could tackle this, but I'm going to look a little bit into mypy internals to decide on how certain details could be solved (the trickiest issue I can see is around circular dependencies).

[JukkaL]

It would be good to have a more concrete proposal about how this could be used in a type checker.

[rtpg]

Proposal for metatypes

By settings a __metatype__ property on a class, you could indicate to type checkers an object that could handle post-processing of the class's type definition.

Whenever a tool like mypy parses a class definition, it will build up an object to store that type information, containing things like the class's methods and fields, as well as fields that appear on its instance. After loading the class, if a __metatype__ is declared on the class (or in one of its superclasses), the type checker will pass the class type information to this metatype for post-processing (by calling the metatypes __new_type__ method).

An example of what the type information could be shaped like:

class TypeInfo:
      fields: Dict[str, TypeInfo] = {}
      type_object: type

 class ClassTypeInfo(TypeInfo):
     # instance_fields stores the type information for an instance of this class
     # for example, fields that are added in the __init__
      instance_fields: Dict[str, TypeInfo] = {}
      # the class itself, loaded into memory
      cls: type

As an example:

class Field:
   pass

class StringField(Field):
    field_type = string
class NumberField(Field):
    field_type = int

class ModelMetatype: # metatype
    def __new_type__(typ:TypeInfo):
        for name, value in typ.fields.items():
            # if we're looking at an ORM field
            if issubclass(value.type_object, Field):
                # remove the field from the class definition
                del typ.fields[name]
                # for instances of this class, place a field of the "field type" instead
                replacement_type = type_info_for(typ.cls.field_type)
                typ.instance_fields[name] = replacement_type

class Model:
   ...
   __metatype__ = ModelMetatype

class Student(Model):
    name = StringField()
    tuition = NumberField()
    other_field = 3

# type error, because it was removed by the metatype
Student.name
# this will properly type check, because of the instance type modification
Student().name = "Judith" 
# this field has no issues, because it was untouched by the metatype
Student.other_field 

As a starting point, we would say that the type is run through the metatype once all forward references in types have been resolved. Though, because of circular references, we cannot be certain that all metatypes have been processed at that point, so it would be the responsibilities of the writers to understand that type information might not be completely initialized by that point.

The big change that would be required here is actually loading the code during type checking. This is in order to provide the class to the metatype to provide useful information. For example, in the ORM case, to find the type of the field that is wanted. I don't know how feasible this is in some cases, but I believe it can be useful in a lot of "code generation" situations.

Another small example would be a class loading its class definition from a file (useful in dealing with things like WSDL):

class GeneratedClass(FromWSDLFile):
  definition_file="spec.xml"

There we could imagine the metatype loading the file into memory and parsing it using the same tool that the FromWSDLFile class itself would use. This would be a powerful check that mere mortals could implement, but is usually only provided by IDEs.

[ilevkivskyi]

Another thing that was mentioned in #399 is that some of such patterns may be described using descriptors (even if the actual runtime mechanism is more complex). This is for example what is done in sqlalchemy-stubs.

[srittau]

I don't think that static type checking could ever support arbitrary code in a generic way. By now we have type checkers written in Python, JavaScript, C, and Java. Of course, individual type checkers could support extensions in various ways. I am closing this here though.