typing.NewType isn't supported

[rhofour]

• cattrs version: 1.0.0
• Python version: 3.8.5
• Operating System: Fedora 32

Description

NewType seems to break structuring / destructuring.

Here's a minimal test case:

import unittest
from typing import NewType
import attr
import cattr

ConfigId = NewType('ConfigId', int)

@attr.s(auto_attribs=True)
class ClassWithNewTypeMember:
    configId: ConfigId

class TestAttrs(unittest.TestCase):
    def test_newType(self):

        event = ClassWithNewTypeMember(configId = ConfigId(0))
        unstructured = cattr.unstructure(event)
        structured = cattr.structure(unstructured, ClassWithNewTypeMember)

I would expect this test to pass, but instead I get the following:
E ValueError: Unsupported type: <function NewType.<locals>.new_type at 0x7fa0349398b0>. Register a structure hook for it.

The workaround is to manually register a structure hook like so:
cattr.register_structure_hook(ConfigId, lambda d, _: ConfigId(d))

A destructuring hook isn't needed.

[letalvoj]

According to the documentation https://docs.python.org/3/library/typing.html#newtype

The static type checker will treat the new type as if it were a subclass of the original type. This is useful in helping catch logical errors:

def get_user_name(user_id: UserId) -> str:
    ...

# typechecks
user_a = get_user_name(UserId(42351))

# does not typecheck; an int is not a UserId
user_b = get_user_name(-1)

Hence, as ConfigId is more specific than int, cattrs should not know how to derive an encoder for it, right?

[Tinche]

We probably just need to teach cattrs to treat newtypes as their underlying types. I agree this would be useful.

[raabf]

The following can be also a workaround since it eliminates the NewType alias at runtime (although I personally prefer also to register a un-/structure hook for ID):

from typing import NewType, TYPE_CHECKING
ID = NewType('ID', str)

if not TYPE_CHECKING:
    ID = ID.__supertype__

[uskudnik]

(Commenting here since #173 is closed)

@Tinche Unfortunately manually registering a hook for NewTypes doesn't work any more, based on my tracking down the culprit is the same (traceback below).

It seems to boil down to failing isinstance(ConfigId, type):

In [6]: isinstance(int, type)
Out[6]: True

In [7]: isinstance(ConfigId, type)
Out[7]: False

from typing import NewType
import cattr
 
ConfigId = NewType("ConfigId", int)
cattr.register_structure_hook(ConfigId, lambda d, _: ConfigId(d))

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-3bdf89a2fc7e> in <module>
      3 
      4 ConfigId = NewType("ConfigId", int)
----> 5 cattr.register_structure_hook(ConfigId, lambda d, _: ConfigId(d))

/usr/local/lib/python3.10/site-packages/cattr/converters.py in register_structure_hook(self, cl, func)
    267             self._structure_func.clear_cache()
    268         else:
--> 269             self._structure_func.register_cls_list([(cl, func)])
    270 
    271     def register_structure_hook_func(

/usr/local/lib/python3.10/site-packages/cattr/dispatch.py in register_cls_list(self, cls_and_handler, direct)
     55                 self._direct_dispatch[cls] = handler
     56             else:
---> 57                 self._single_dispatch.register(cls, handler)
     58                 self.clear_direct()
     59         self.dispatch.cache_clear()

/usr/local/lib/python3.10/functools.py in register(cls, func)
    854         else:
    855             if func is not None:
--> 856                 raise TypeError(
    857                     f"Invalid first argument to `register()`. "
    858                     f"{cls!r} is not a class."

TypeError: Invalid first argument to `register()`. __main__.ConfigId is not a class.

[uskudnik]

Okay, might've found a solution by using register_structure_hook_func

import cattr
from typing import NewType

cattr.register_structure_hook_func(
    lambda t: t.__class__ is NewType, lambda d, t: t.__call__(d)
)

🤞 nothing 💥 too much 🙂

[raabf]

Before python 3.10, typing.NewType is a function which returns a function, hence isinstance does not work. In python 3.10 the symbol typing.NewType became an own class where you can do normal isinstance{obj, typing.NewType) checks (or as you noticed t.__class__ is NewType which does the same).

Something like isinstance(ConfigId, type) should not work on python 3.10 because ConfigId is then an instance of the typing.NewType class instead of an own generated function.

Heve a look at this is_new_type() function for a complete test regardless of python version.

[henryiii]

I'm looking for a way to register a custom converter via a type; that is, I'd like to do a little post processing on fields marked with URL, but I don't need a custom type for the URL in the classes, just str is fine. I could define a field for each URL, but doing it via types is simpler and cleaner, and I don't really need a converter for setting an instance (since they are frozen, no real difference). It seems like NewType would have been a nice way to do it.

From what I can tell above (which is literally asking for the opposite of what I want to do - though I'm fine if it also works by default but is customizable), what I want was supported before 3.10 (though possibly by accident and not fully), but now it's broken?

[Tinche]

NewTypes are in, and will be released with the 22.2.0 release!