Debugging a segmentation fault #13431

jace · 2022-08-16T07:42:19Z

Bug Report

Some code I was working on last week is causing mypy to segfault. I tried retracing my steps to find the change that broke mypy, without luck.

Config:

Platform: macOS 12.4 on M1
Arch: arm64
Python: 3.9.13
mypy: 0.971
mypy plugins: none (segfault happens with all plugins disabled and is not linked to any imported modules)

mypy --tb and mypy --pdb are segfaulting without any info. gdb is not available on arm64. I get a bit more info using faulthandler, but all it indicates is the file:

$ python -X faulthandler -m mypy -v .
…
LOG:  Processing SCC singleton (coaster.views.classview) as inherently stale with stale deps (__future__ asyncio builtins coaster coaster.auth coaster.sqlalchemy coaster.typing coaster.utils coaster.views.misc flask flask.blueprints flask.typing functools sqlalchemy.orm.attributes sqlalchemy.orm.descriptor_props sqlalchemy.orm.mapper sqlalchemy.orm.properties sqlalchemy.orm.query types typing typing_extensions werkzeug.local werkzeug.routing werkzeug.wrappers)
Fatal Python error: Segmentation fault

Current thread 0x0000000102f10580 (most recent call first):
  File "/Users/jace/.virtualenvs/hasgeek/lib/python3.9/site-packages/mypy/__main__.py", line 12 in console_entry
  File "/Users/jace/.virtualenvs/hasgeek/lib/python3.9/site-packages/mypy/__main__.py", line 34 in <module>
  File "/opt/homebrew/Cellar/[email protected]/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87 in _run_code
  File "/opt/homebrew/Cellar/[email protected]/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197 in _run_module_as_main
[1]    44130 segmentation fault  python -X faulthandler -m mypy -v .

The file in question is located at https://github.com/hasgeek/coaster/blob/369ce391f11850f10fc0ab42a52c196188436a50/coaster/views/classview.py

To Reproduce

git clone https://github.com/hasgeek/coaster.git
cd coaster
git checkout 369ce391f11850f10fc0ab42a52c196188436a50
mypy coaster/views/classview.py

The text was updated successfully, but these errors were encountered:

hauntsaninja · 2022-08-18T21:40:51Z

Thanks, I needed to disable the "sqlmypy" plugin in order to be able to repro.

When running with uncompiled mypy (using latest master), I get the following stacktrace:

coaster/views/classview.py:193: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.980+dev.23ee1e7aff357e656e3102435ad0fe3b5074571e
Traceback (most recent call last):
  File "/Users/shantanu/.virtualenvs/mypy-qyes/bin/mypy", line 33, in <module>
    sys.exit(load_entry_point('mypy', 'console_scripts', 'mypy')())
  File "/Users/shantanu/dev/mypy/mypy/__main__.py", line 15, in console_entry
    main()
  File "/Users/shantanu/dev/mypy/mypy/main.py", line 95, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/Users/shantanu/dev/mypy/mypy/main.py", line 174, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 187, in build
    result = _build(
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 271, in _build
    graph = dispatch(sources, manager, stdout)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2874, in dispatch
    process_graph(graph, manager)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3258, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3359, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2302, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 465, in check_first_pass
    self.accept(d)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1127, in accept
    return visitor.visit_class_def(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 2004, in visit_class_def
    self.accept(defn.defs)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1202, in accept
    return visitor.visit_block(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 2384, in visit_block
    self.accept(s)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 811, in accept
    return visitor.visit_func_def(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 935, in visit_func_def
    self._visit_func_def(defn)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 939, in _visit_func_def
    self.check_func_item(defn, name=defn.name)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 1005, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 1189, in check_func_def
    self.accept(item.body)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1202, in accept
    return visitor.visit_block(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 2384, in visit_block
    self.accept(s)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 571, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1386, in accept
    return visitor.visit_return_stmt(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 3861, in visit_return_stmt
    self.check_return_stmt(s)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 3895, in check_return_stmt
    self.expr_checker.accept(
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 4615, in accept
    typ = node.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 1822, in accept
    return visitor.visit_call_expr(self)
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 385, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 505, in visit_call_expr_inner
    ret_type = self.check_call_expr_with_callee_type(
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 1155, in check_call_expr_with_callee_type
    ret_type, callee_type = self.check_call(
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 1290, in check_call
    return self.check_call(
  File "/Users/shantanu/dev/mypy/mypy/checkexpr.py", line 1257, in check_call
    call_function = analyze_member_access(
  File "/Users/shantanu/dev/mypy/mypy/checkmember.py", line 187, in analyze_member_access
    result = _analyze_member_access(name, typ, mx, override_info)
  File "/Users/shantanu/dev/mypy/mypy/checkmember.py", line 206, in _analyze_member_access
    return analyze_instance_member_access(name, typ, mx, override_info)
  File "/Users/shantanu/dev/mypy/mypy/checkmember.py", line 304, in analyze_instance_member_access
    signature = freshen_function_type_vars(signature)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 101, in freshen_function_type_vars
    fresh = cast(CallableType, expand_type(callee, tvmap)).copy_modified(variables=tvs)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1743, in accept
    return visitor.visit_callable_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 254, in visit_callable_type
    arg_types=self.expand_types(t.arg_types),
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 345, in expand_types
    a.append(t.accept(self))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 2509, in accept
    return visitor.visit_union_type(self)
    
  ... 32000 lines elided ...

  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 325, in visit_union_type
    return make_simplified_union(self.expand_types(t.items), t.line, t.column)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 454, in make_simplified_union
    simplified_set: Sequence[Type] = _remove_redundant_union_items(items, keep_erased)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 517, in _remove_redundant_union_items
    if is_redundant_literal_instance(proper_item, proper_tj) and is_proper_subtype(
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 208, in is_proper_subtype
    return _is_subtype(left, right, subtype_context, proper_subtype=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 306, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1256, in accept
    return visitor.visit_instance(self)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 557, in visit_instance
    call = find_member("__call__", left, left, is_operator=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 996, in find_member
    return find_node_type(method, itype, subtype)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 1088, in find_node_type
    signature = bind_self(
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in bind_self
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in <listcomp>
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 318, in expand
    return expand_type(target, {id: to_apply[all_ids.index(id)] for id in ids})
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 2509, in accept
    return visitor.visit_union_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 325, in visit_union_type
    return make_simplified_union(self.expand_types(t.items), t.line, t.column)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 454, in make_simplified_union
    simplified_set: Sequence[Type] = _remove_redundant_union_items(items, keep_erased)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 517, in _remove_redundant_union_items
    if is_redundant_literal_instance(proper_item, proper_tj) and is_proper_subtype(
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 208, in is_proper_subtype
    return _is_subtype(left, right, subtype_context, proper_subtype=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 306, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1256, in accept
    return visitor.visit_instance(self)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 557, in visit_instance
    call = find_member("__call__", left, left, is_operator=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 996, in find_member
    return find_node_type(method, itype, subtype)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 1088, in find_node_type
    signature = bind_self(
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in bind_self
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in <listcomp>
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 318, in expand
    return expand_type(target, {id: to_apply[all_ids.index(id)] for id in ids})
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 2509, in accept
    return visitor.visit_union_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 325, in visit_union_type
    return make_simplified_union(self.expand_types(t.items), t.line, t.column)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 454, in make_simplified_union
    simplified_set: Sequence[Type] = _remove_redundant_union_items(items, keep_erased)
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 517, in _remove_redundant_union_items
    if is_redundant_literal_instance(proper_item, proper_tj) and is_proper_subtype(
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 208, in is_proper_subtype
    return _is_subtype(left, right, subtype_context, proper_subtype=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 306, in _is_subtype
    return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1256, in accept
    return visitor.visit_instance(self)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 557, in visit_instance
    call = find_member("__call__", left, left, is_operator=True)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 996, in find_member
    return find_node_type(method, itype, subtype)
  File "/Users/shantanu/dev/mypy/mypy/subtypes.py", line 1088, in find_node_type
    signature = bind_self(
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in bind_self
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 320, in <listcomp>
    arg_types = [expand(x) for x in func.arg_types[1:]]
  File "/Users/shantanu/dev/mypy/mypy/typeops.py", line 318, in expand
    return expand_type(target, {id: to_apply[all_ids.index(id)] for id in ids})
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 45, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 2509, in accept
    return visitor.visit_union_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 325, in visit_union_type
    return make_simplified_union(self.expand_types(t.items), t.line, t.column)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 345, in expand_types
    a.append(t.accept(self))
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1743, in accept
    return visitor.visit_callable_type(self)
  File "/Users/shantanu/dev/mypy/mypy/expandtype.py", line 253, in visit_callable_type
    return t.copy_modified(
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1678, in copy_modified
    return CallableType(
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1617, in __init__
    super().__init__(line, column)
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 1348, in __init__
    super().__init__(line, column)
  File "/Users/shantanu/dev/mypy/mypy/types.py", line 207, in __init__
    super().__init__(line, column)
RecursionError: maximum recursion depth exceeded while calling a Python object
coaster/views/classview.py:193: : note: use --pdb to drop into pdb

ilevkivskyi · 2022-08-24T01:13:25Z

Probably another example of issue caused by expand_type() calling make_simplified_union(). I was recently thinking about this but still don't have a good solution.

jace · 2022-09-05T06:48:21Z

Is there a particular type declaration in my code that's causing this? Type checking is broken downstream for me because of this one file. I have to test my client apps with the import removed from env (using a pre-commit container), which sucks because there's no type checking against this library's exports.

hauntsaninja · 2022-09-05T22:06:43Z

You can avoid the crash by doing:

diff --git a/coaster/views/classview.py b/coaster/views/classview.py
index 34e7e47..13e4e5c 100644
--- a/coaster/views/classview.py
+++ b/coaster/views/classview.py
@@ -183,6 +183,7 @@ class ViewHandler(  # pylint: disable=too-many-instance-attributes
         self.requires_roles = requires_roles or {}
         self.endpoints = set()
 
+    @t.no_type_check
     def reroute(self: ViewHandlerType, f: t.Callable) -> ViewHandlerType:
         """Replace a view handler in a subclass while keeping its URL route rules."""
         # Use type(self) instead of ViewHandler so this works for (future) subclasses
@@ -206,6 +207,7 @@ class ViewHandler(  # pylint: disable=too-many-instance-attributes
         r.endpoints = set()
         return r
 
+    @t.no_type_check
     def __call__(
         self: ViewHandlerType, decorated: t.Union[t.Callable, ViewHandlerType]
     ) -> ViewHandlerType:

Didn't yet look into minimising it.

You can also get mypy to avoid processing a single module by doing something like the following in your mypy config:

[mypy-coaster.views.classview]
follow_imports = skip

ilevkivskyi · 2022-09-05T23:30:10Z

A simple repro test case is

[case testCallableUnionCB]
from typing import Union, Callable, TypeVar

TA = TypeVar("TA", bound=A)
class A:
    def __call__(self: TA, other: Union[Callable, TA]) -> TA: ...
a: A
a()

Actually, looking at this, the bug is that we never push to assumptions stack for callback subtyping, while we should. I am not sure what is the best way to do this without a visible performance impact, maybe we can just push them locally in few places where __call__ is special-cased for subtyping/inference (also there may be an additional complication that we may need to "anonimize" generic callables before hashing them, not sure if this is needed).

erictraut · 2023-08-13T18:10:44Z

I'm not able to repro a crash with the latest version of mypy (1.5). I suspect it was fixed somewhere along the way.

hauntsaninja · 2023-08-13T19:37:32Z

Fixed in #14178, but I think worth adding Ivan's test case

See python#13431

See #13431 Authored by ilevkivskyi

jace added the bug mypy got something wrong label Aug 16, 2022

AlexWaygood added crash and removed bug mypy got something wrong labels Aug 16, 2022

hauntsaninja added the priority-0-high label Aug 18, 2022

hauntsaninja added a commit to hauntsaninja/mypy that referenced this issue Aug 13, 2023

Add regression test for expand type -> simplified union crash

6fc1415

See python#13431

hauntsaninja mentioned this issue Aug 13, 2023

Add regression test for expand type -> simplified union crash #15864

Merged

hauntsaninja closed this as completed Aug 13, 2023

hauntsaninja added a commit that referenced this issue Aug 13, 2023

Add regression test for expand type -> simplified union crash (#15864)

98881d2

See #13431 Authored by ilevkivskyi

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Debugging a segmentation fault #13431

Debugging a segmentation fault #13431

jace commented Aug 16, 2022

hauntsaninja commented Aug 18, 2022

ilevkivskyi commented Aug 24, 2022 •

edited

Loading

jace commented Sep 5, 2022

hauntsaninja commented Sep 5, 2022

ilevkivskyi commented Sep 5, 2022

erictraut commented Aug 13, 2023

hauntsaninja commented Aug 13, 2023

Debugging a segmentation fault #13431

Debugging a segmentation fault #13431

Comments

jace commented Aug 16, 2022

hauntsaninja commented Aug 18, 2022

ilevkivskyi commented Aug 24, 2022 • edited Loading

jace commented Sep 5, 2022

hauntsaninja commented Sep 5, 2022

ilevkivskyi commented Sep 5, 2022

erictraut commented Aug 13, 2023

hauntsaninja commented Aug 13, 2023

ilevkivskyi commented Aug 24, 2022 •

edited

Loading