Can't pickle generic types

[pitrou]

Briefly:

>>> import pickle, typing
>>> tp = typing.List[int]
>>> pickle.loads(pickle.dumps(tp))
Traceback (most recent call last):
  File "<ipython-input-5-ae365d48e86d>", line 1, in <module>
    pickle.loads(pickle.dumps(tp))
PicklingError: Can't pickle typing.List[int]: it's not the same object as typing.List

This can be an issue with certain advanced serialization libraries such as dill or cloudpickle, which are often used by distributed computing frameworks.

[ilevkivskyi]

This is a known issue, unfortunately it is not easy to fix, see discussion in #306. In short, X.__qualname__ == X[int].__qualname__ and pickle doesn't like this since it looks up classes by names. PEP 560 will fix this in Python 3.7, not sure however what to do with the older versions.

[pitrou]

Stupid question, but why isn't List[int] an instance of List?

[ilevkivskyi]

Currently, List[int] is a subclass of List, all subscripted generics are currently actual class objects. One of the motivation of PEP 560 was to make them just normal objects that are aliases (proxies) for the original class object.

[pitrou]

I guess my question was: why the whole design of having types be classes rather than instances? :-)

[ilevkivskyi]

This is a very old (relative to me) decision, I don't know for sure, but I think one of the main motivation was that things like this should work:

class C(List[int], Generic[T]):
    ...

[pitrou]

Regardless of the class issue, it seems it's difficult to find a reliable way to rebuild a generic type from its attributes. The problem is, both GenericMeta.__new__ and GenericMeta.__getitem__ do destructive conversion and fixup on inputs, and you can't derive the original inputs from them, because __extra__, __bases__ and friends only give you the converted form. The internal design seems actually hostile to any kind of instance rebuilding.

[pitrou]

This is the __reduce__-like function I came up with for generic types:
https://gist.github.com/pitrou/a14a012871e852fdfaddbd8141d59611

Unfortunately, for some reason it succeeds on Generator[int, float, str] but fails on Generator[int, None, None] (see the comment about extra being stuffed into bases).

[ilevkivskyi]

Just a note that with PEP 560 in place the problems boils down to TypeVars and therefore is closely related to #512. The solution would be probably the same: consider type variables as immutable and pickle them like classes (they are also considered immutable by copy etc., and pickled as just f{__module__}.{__qualname__}).