Skip to content

Commit

Permalink
Stop TypedDictAnalyzer from leaking synthetic types
Browse files Browse the repository at this point in the history
Fixes #10007

Mypy currently crashes when you try:

1. Creating a class-based TypedDict containing a malformed type hint
2. Asking it to compute fine-grained dependencies, either via
   running dmypy or by setting the `--cache-fine-grained` flag.

Here is the exact sequence of events that leads to this crash:

1. Since the type annotation is malformed, semanal initially gives
   the type annotation a type of `RawExpressionType`.

2. TypedDictAnalyzer (correctly) determines determines that the
   type of the malformed type annotation should be treated as
   just `Any` in:
   https://github.com/python/mypy/blob/f5ce4ee6ca7e2d8bb1cde8a2b49865f53bbacff5/mypy/semanal_typeddict.py#L289

3. TypedDictAnalyzer forgets to modify `stmt.type` like we normally
   do after calling `self.anal_type` in normal classes:
   https://github.com/python/mypy/blob/f5ce4ee6ca7e2d8bb1cde8a2b49865f53bbacff5/mypy/semanal.py#L3022

4. Mypy _does_ use the `Any` type when constructing the TypeInfo
   for the TypedDict. This is why mypy will not crash under most
   conditions: the correct type is being used in most places.

5. Setting `--cache-fine-grained` will make mypy perform one final
   pass against the AST to compute fine-grained dependencies. As
   a part of this process, it traverses the AssigmentStatement's `type`
   field using TypeTriggersVisitor.

6. TypeTriggersVisitor is _not_ a SyntheticTypeVisitor. So, the
   visitor trips an assert when we try traversing into the
   RawExpressionType.

Interestingly, this same crash does not occur for NamedTuples
despite the fact that NamedTupleAnalyzer also does not set `stmt.type`:
https://github.com/python/mypy/blob/f5ce4ee6ca7e2d8bb1cde8a2b49865f53bbacff5/mypy/semanal_namedtuple.py#L177

It turns out this is because semanal.py will end up calling the
`analyze_class_body_common(...)` function after NamedTupleAnalyzer
runs, but _not_ after TypedDictAnalyzer runs:

- https://github.com/python/mypy/blob/f5ce4ee6ca7e2d8bb1cde8a2b49865f53bbacff5/mypy/semanal.py#L1510
- https://github.com/python/mypy/blob/f5ce4ee6ca7e2d8bb1cde8a2b49865f53bbacff5/mypy/semanal.py#L1479

I'm not sure why this is: ideally, the two analyzers ought to have
as similar semantics as possible. But refactoring this felt potentially
disruptive, so I went for the narrower route of just patching TypedDictAnalyzer.
  • Loading branch information
Michael0x2a committed Oct 23, 2022
1 parent cf705d7 commit ef4d492
Show file tree
Hide file tree
Showing 3 changed files with 37 additions and 1 deletion.
6 changes: 6 additions & 0 deletions mypy/semanal_typeddict.py
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,12 @@ def analyze_typeddict_classdef_fields(
)
if analyzed is None:
return None, [], set() # Need to defer
# TypedDictAnalyzer sets the AssignmentStmt type here, but
# NamedTupleAnalyzer doesn't and instead has semanal.py set it
# by calling analyze_class_body_common after.
#
# TODO: Resolve this inconsistency?
stmt.type = analyzed
types.append(analyzed)
# ...despite possible minor failures that allow further analyzis.
if stmt.type is None or hasattr(stmt, "new_syntax") and not stmt.new_syntax:
Expand Down
30 changes: 30 additions & 0 deletions test-data/unit/check-semanal-error.test
Original file line number Diff line number Diff line change
Expand Up @@ -152,3 +152,33 @@ class C:
x: P[int] = C()
[builtins fixtures/tuple.pyi]
[out]

[case testSemanalDoesNotLeakSyntheticTypes]
# flags: --cache-fine-grained
from typing import Generic, NamedTuple, TypedDict, TypeVar
from dataclasses import dataclass

T = TypeVar('T')
class Wrap(Generic[T]): pass

invalid_1: 1 + 2 # E: Invalid type comment or annotation
invalid_2: Wrap[1 + 2] # E: Invalid type comment or annotation

class A:
invalid_1: 1 + 2 # E: Invalid type comment or annotation
invalid_2: Wrap[1 + 2] # E: Invalid type comment or annotation

class B(NamedTuple):
invalid_1: 1 + 2 # E: Invalid type comment or annotation
invalid_2: Wrap[1 + 2] # E: Invalid type comment or annotation

class C(TypedDict):
invalid_1: 1 + 2 # E: Invalid type comment or annotation
invalid_2: Wrap[1 + 2] # E: Invalid type comment or annotation

@dataclass
class D:
invalid_1: 1 + 2 # E: Invalid type comment or annotation
invalid_2: Wrap[1 + 2] # E: Invalid type comment or annotation
[builtins fixtures/dict.pyi]
[typing fixtures/typing-typeddict.pyi]
2 changes: 1 addition & 1 deletion test-data/unit/semanal-typeddict.test
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,4 @@ MypyFile:1(
NameExpr(x)
TempNode:4(
Any)
str?)))
builtins.str)))

0 comments on commit ef4d492

Please sign in to comment.