Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

functools: Add typing for decorated function arguments in cache API [take 2] #10523

Closed
wants to merge 8 commits into from

Conversation

jakkdl
Copy link

@jakkdl jakkdl commented Jul 31, 2023

Giving #9768 another attempt, adding WMOkiishis suggested changes to see if that helps with mypy primer output.

Also rebased on top of main/

@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/coretypes/serializable.py:82: error: Too few arguments  [call-arg]
+ mitmproxy/coretypes/serializable.py:90: error: Missing positional argument "cls" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/coretypes/serializable.py:98: error: Too few arguments  [call-arg]
+ mitmproxy/proxy/mode_specs.py:164: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/connection.py:181: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/connection.py:181: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[<nothing>]"  [arg-type]
+ mitmproxy/utils/asyncio_utils.py:56: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "Any | str"; expected "tuple[Any, ...] | None"  [arg-type]
+ mitmproxy/tools/console/common.py:811: error: Argument "error_message" to "__call__" of "_lru_cache_wrapper" has incompatible type "str | None"; expected "str"  [arg-type]
+ mitmproxy/tools/console/common.py:848: error: Incompatible types in assignment (expression has type "_lru_cache_wrapper[[NamedArg(RenderMode, 'render_mode'), NamedArg(bool, 'focused'), NamedArg(str, 'marked'), NamedArg(str | None, 'is_replay'), NamedArg(str, 'request_method'), NamedArg(str, 'request_scheme'), NamedArg(str, 'request_host'), NamedArg(str, 'request_path'), NamedArg(str, 'request_url'), NamedArg(str, 'request_http_version'), NamedArg(float, 'request_timestamp'), NamedArg(bool, 'request_is_push_promise'), NamedArg(bool, 'intercepted'), NamedArg(int | None, 'response_code'), NamedArg(str | None, 'response_reason'), NamedArg(int | None, 'response_content_length'), NamedArg(str | None, 'response_content_type'), NamedArg(float | None, 'duration'), NamedArg(str | None, 'error_message')], Any]", variable has type "_lru_cache_wrapper[[NamedArg(RenderMode, 'render_mode'), NamedArg(bool, 'focused'), NamedArg(str, 'marked'), NamedArg(bool, 'is_replay'), NamedArg(str, 'request_method'), NamedArg(str, 'request_scheme'), NamedArg(str, 'request_host'), NamedArg(str, 'request_path'), NamedArg(str, 'request_url'), NamedArg(str, 'request_http_version'), NamedArg(float, 'request_timestamp'), NamedArg(bool, 'request_is_push_promise'), NamedArg(bool, 'intercepted'), NamedArg(int | None, 'response_code'), NamedArg(str | None, 'response_reason'), NamedArg(int | None, 'response_content_length'), NamedArg(str | None, 'response_content_type'), NamedArg(float | None, 'duration'), NamedArg(str | None, 'error_message')], Any]")  [assignment]
+ mitmproxy/tools/console/common.py:853: error: Argument "is_replay" to "__call__" of "_lru_cache_wrapper" has incompatible type "str | None"; expected "bool"  [arg-type]
+ mitmproxy/command.py:285: error: Too few arguments  [call-arg]
+ mitmproxy/command.py:285: error: Argument 1 has incompatible type "str"; expected "CommandManager"  [arg-type]
+ mitmproxy/test/tflow.py:134: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/test/tflow.py:134: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[<nothing>]"  [arg-type]
+ mitmproxy/test/tflow.py:235: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/test/tflow.py:235: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[<nothing>]"  [arg-type]
+ mitmproxy/master.py:115: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/tools/console/commander/commander.py:77: error: Too few arguments  [call-arg]
+ mitmproxy/tools/console/commander/commander.py:77: error: Argument 1 has incompatible type "str"; expected "CommandManager"  [arg-type]
+ mitmproxy/tools/console/commander/commander.py:109: error: Too few arguments  [call-arg]
+ mitmproxy/tools/console/commander/commander.py:110: error: Argument 1 has incompatible type "str"; expected "CommandManager"  [arg-type]
+ mitmproxy/proxy/server.py:556: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/proxy/server.py:556: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[<nothing>]"  [arg-type]
+ mitmproxy/proxy/mode_servers.py:113: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/proxy/mode_servers.py:113: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[<nothing>]"  [arg-type]
+ mitmproxy/addons/clientplayback.py:97: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/addons/proxyserver.py:102: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/addons/proxyserver.py:102: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[<nothing>]"  [arg-type]
+ mitmproxy/addons/proxyserver.py:245: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/addons/proxyserver.py:282: error: Missing positional argument "spec" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ mitmproxy/tools/console/keymap.py:87: error: "Callable[[Keymap, Keymap, str], Binding | None]" has no attribute "cache_clear"  [attr-defined]
+ mitmproxy/tools/console/flowview.py:68: error: "Callable[[FlowDetails, FlowDetails, Any, Any, Any], Any]" has no attribute "cache_clear"  [attr-defined]
+ mitmproxy/tools/console/flowview.py:303: error: Too few arguments  [call-arg]
+ mitmproxy/tools/console/flowview.py:304: error: Argument 2 has incompatible type "int"; expected "FlowDetails"  [arg-type]
+ mitmproxy/tools/console/flowlist.py:59: error: "Callable[[FlowListWalker, FlowListWalker, int], tuple[FlowItem | None, int | None]]" has no attribute "cache_clear"  [attr-defined]
+ mitmproxy/tools/console/flowlist.py:78: error: Too few arguments  [call-arg]
+ mitmproxy/tools/console/flowlist.py:81: error: Too few arguments  [call-arg]
+ mitmproxy/tools/console/quickhelp.py:175: error: Too few arguments  [call-arg]
+ mitmproxy/tools/console/quickhelp.py:175: error: Argument 1 has incompatible type "str"; expected "Keymap"  [arg-type]

pip (https://github.com/pypa/pip)
+ src/pip/_internal/metadata/base.py:406: error: Too few arguments  [call-arg]
+ src/pip/_internal/models/link.py:245: error: Missing positional argument "url" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ src/pip/_internal/models/link.py:245: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[LinkHash]"  [arg-type]
+ src/pip/_internal/req/req_install.py:528: error: Too few arguments  [call-arg]
+ src/pip/_internal/req/req_install.py:554: error: Too few arguments  [call-arg]
+ src/pip/_internal/index/package_finder.py:890: error: Too few arguments  [call-arg]
+ src/pip/_internal/index/package_finder.py:890: error: Argument 1 has incompatible type "str"; expected "PackageFinder"  [arg-type]
+ src/pip/_internal/index/package_finder.py:908: error: Too few arguments  [call-arg]
+ src/pip/_internal/index/package_finder.py:909: error: Argument 1 has incompatible type "str | None"; expected "PackageFinder"  [arg-type]
+ src/pip/_internal/self_outdated_check.py:177: error: Too few arguments  [call-arg]
+ src/pip/_internal/self_outdated_check.py:177: error: Argument 1 has incompatible type "str"; expected "PackageFinder"  [arg-type]
+ src/pip/_internal/wheel_builder.py:73: error: Too few arguments  [call-arg]
+ src/pip/_internal/distributions/sdist.py:114: error: Too few arguments  [call-arg]
+ src/pip/_internal/resolution/resolvelib/factory.py:279: error: Too few arguments  [call-arg]
+ src/pip/_internal/resolution/resolvelib/factory.py:604: error: Too few arguments  [call-arg]
+ src/pip/_internal/resolution/resolvelib/factory.py:604: error: Argument 1 has incompatible type "NormalizedName"; expected "PackageFinder"  [arg-type]
+ src/pip/_internal/commands/list.py:239: error: Too few arguments  [call-arg]
+ src/pip/_internal/commands/list.py:239: error: Argument 1 has incompatible type "NormalizedName"; expected "PackageFinder"  [arg-type]
+ src/pip/_internal/commands/index.py:119: error: Too few arguments  [call-arg]

manticore (https://github.com/trailofbits/manticore)
+ manticore/core/smtlib/solver.py:505: error: Signature of "can_be_true" incompatible with supertype "Solver"  [override]
+ manticore/core/smtlib/solver.py:505: note:      Superclass:
+ manticore/core/smtlib/solver.py:505: note:          def can_be_true(self, constraints: Any, expression: Any = ...) -> bool
+ manticore/core/smtlib/solver.py:505: note:      Subclass:
+ manticore/core/smtlib/solver.py:505: note:          _lru_cache_wrapper[[SMTLIBSolver, ConstraintSet, Union[bool, Bool]], bool]
+ manticore/core/smtlib/solver.py:605: error: Signature of "get_all_values" incompatible with supertype "Solver"  [override]
+ manticore/core/smtlib/solver.py:605: note:      Superclass:
+ manticore/core/smtlib/solver.py:605: note:          def get_all_values(self, constraints: Any, x: Any, maxcnt: Any = ..., silent: Any = ...) -> Any
+ manticore/core/smtlib/solver.py:605: note:      Subclass:
+ manticore/core/smtlib/solver.py:605: note:          _lru_cache_wrapper[[SMTLIBSolver, ConstraintSet, Any, Optional[int], bool], Any]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/dtypes/cast.py:589: error: Unused "type: ignore" comment  [unused-ignore]
+ pandas/core/groupby/ops.py:395: error: Too few arguments  [call-arg]
+ pandas/core/groupby/ops.py:395: error: Argument 1 has incompatible type "str"; expected "WrappedCythonOp"  [arg-type]
+ pandas/core/groupby/ops.py:395: error: Argument 2 has incompatible type "str"; expected "type[WrappedCythonOp]"  [arg-type]
+ pandas/core/groupby/ops.py:395: error: Argument 4 has incompatible type "bool"; expected "str"  [arg-type]

aiohttp (https://github.com/aio-libs/aiohttp)
+ aiohttp/connector.py:932:24: error: Too few arguments  [call-arg]
+ aiohttp/connector.py:932:47: error: Argument 1 has incompatible type "bool"; expected "TCPConnector"  [arg-type]
+ aiohttp/connector.py:938:24: error: Too few arguments  [call-arg]
+ aiohttp/connector.py:938:47: error: Argument 1 has incompatible type "bool"; expected "TCPConnector"  [arg-type]
+ aiohttp/connector.py:939:20: error: Too few arguments  [call-arg]
+ aiohttp/connector.py:939:43: error: Argument 1 has incompatible type "bool"; expected "TCPConnector"  [arg-type]

rich (https://github.com/Textualize/rich)
+ rich/theme.py:25: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/theme.py:25: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Style]"  [arg-type]
+ rich/theme.py:54: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/theme.py:54: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Style]"  [arg-type]
+ rich/style.py:147: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/style.py:147: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Color]"  [arg-type]
+ rich/style.py:368: error: Too few arguments  [call-arg]
+ rich/style.py:368: error: Argument 1 has incompatible type "ColorSystem"; expected "Color"  [arg-type]
+ rich/style.py:371: error: Too few arguments  [call-arg]
+ rich/style.py:371: error: Argument 1 has incompatible type "ColorSystem"; expected "Color"  [arg-type]
+ rich/style.py:391: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/style.py:391: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Style]"  [arg-type]
+ rich/style.py:527: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/style.py:527: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Color]"  [arg-type]
+ rich/style.py:554: error: Missing positional argument "color" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/style.py:554: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Color]"  [arg-type]
+ rich/style.py:758: error: Too few arguments  [call-arg]
+ rich/style.py:758: error: Argument 1 has incompatible type "Style | None"; expected "Style"  [arg-type]
+ rich/segment.py:169: error: Too few arguments  [call-arg]
+ rich/segment.py:169: error: Argument 2 has incompatible type "int"; expected "type[Segment]"  [arg-type]
+ rich/markup.py:160: error: Missing positional argument "style" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/markup.py:160: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Style]"  [arg-type]
+ rich/markup.py:217: error: Missing positional argument "style" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/markup.py:217: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Style]"  [arg-type]
+ rich/jupyter.py:72: error: Too few arguments  [call-arg]
+ rich/jupyter.py:72: error: Argument 1 has incompatible type "TerminalTheme"; expected "Style"  [arg-type]
+ rich/console.py:1476: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/console.py:1476: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Style]"  [arg-type]
+ rich/console.py:2210: error: Too few arguments  [call-arg]
+ rich/console.py:2210: error: Argument 1 has incompatible type "TerminalTheme"; expected "Style"  [arg-type]
+ rich/console.py:2222: error: Too few arguments  [call-arg]
+ rich/console.py:2222: error: Argument 1 has incompatible type "TerminalTheme"; expected "Style"  [arg-type]
+ rich/color.py:552: error: Too few arguments  [call-arg]
+ rich/color.py:552: error: Argument 1 has incompatible type "ColorTriplet"; expected "Palette"  [arg-type]
+ rich/color.py:565: error: Too few arguments  [call-arg]
+ rich/color.py:565: error: Argument 1 has incompatible type "ColorTriplet"; expected "Palette"  [arg-type]
+ rich/ansi.py:175: error: Missing positional argument "style_definition" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ rich/ansi.py:175: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "str"; expected "type[Style]"  [arg-type]
+ rich/progress_bar.py:144: error: Too few arguments  [call-arg]
+ rich/progress_bar.py:145: error: Argument 1 has incompatible type "Style"; expected "ProgressBar"  [arg-type]
+ rich/progress_bar.py:145: error: Argument 2 has incompatible type "Style"; expected "ProgressBar"  [arg-type]
+ rich/progress_bar.py:145: error: Argument 3 has incompatible type "str | None"; expected "Style"  [arg-type]
+ rich/progress_bar.py:145: error: Argument 4 has incompatible type "bool"; expected "Style"  [arg-type]

dacite (https://github.com/konradhalas/dacite)
+ dacite/core.py:51: error: Argument "localns" to "get_type_hints" has incompatible type "FrozenDict | None"; expected "dict[str, Any] | None"  [arg-type]

pylint (https://github.com/pycqa/pylint)
+ pylint/checkers/utils.py:2238: error: "_lru_cache_wrapper" expects 2 type arguments, but 1 given  [type-arg]
+ pylint/message/message_definition_store.py:79: error: Too few arguments  [call-arg]
+ pylint/message/message_definition_store.py:79: error: Argument 1 has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/message/message_definition_store.py:88: error: Too few arguments  [call-arg]
+ pylint/message/message_definition_store.py:89: error: Argument 1 has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/lint/message_state_handler.py:123: error: Too few arguments  [call-arg]
+ pylint/lint/message_state_handler.py:123: error: Argument 1 has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/lint/message_state_handler.py:151: error: Too few arguments  [call-arg]
+ pylint/lint/message_state_handler.py:151: error: Argument 1 has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/lint/pylinter.py:1261: error: Too few arguments  [call-arg]
+ pylint/lint/pylinter.py:1261: error: Argument 1 has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/lint/pylinter.py:1288: error: Too few arguments  [call-arg]
+ pylint/lint/pylinter.py:1288: error: Argument 1 has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]
+ pylint/checkers/format.py:446: error: Too few arguments  [call-arg]
+ pylint/checkers/format.py:447: error: Argument 1 has incompatible type "str"; expected "MessageDefinitionStore"  [arg-type]

mypy (https://github.com/python/mypy)
+ mypy/find_sources.py:160: error: Too few arguments  [call-arg]
+ mypy/find_sources.py:160: error: Argument 1 has incompatible type "str"; expected "SourceFinder"  [arg-type]
+ mypy/find_sources.py:202: error: Too few arguments  [call-arg]
+ mypy/find_sources.py:202: error: Argument 1 has incompatible type "str"; expected "SourceFinder"  [arg-type]

isort (https://github.com/pycqa/isort)
+ isort/deprecated/finders.py:296: error: Too few arguments  [call-arg]
+ isort/deprecated/finders.py:296: error: Argument 1 has incompatible type "str"; expected "RequirementsFinder"  [arg-type]
+ isort/deprecated/finders.py:327: error: Too few arguments  [call-arg]
+ isort/deprecated/finders.py:327: error: Argument 1 has incompatible type "str"; expected "RequirementsFinder"  [arg-type]

psycopg (https://github.com/psycopg/psycopg)
+ psycopg/psycopg/rows.py:145: error: Argument 2 to "__call__" of "_lru_cache_wrapper" has incompatible type "*Generator[bytes | None, None, None]"; expected "bytes"  [arg-type]

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/datafusion/__init__.py:334: error: Missing positional argument "cls" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ ibis/backends/polars/__init__.py:347: error: Missing positional argument "cls" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ ibis/backends/pandas/__init__.py:228: error: Missing positional argument "cls" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]
+ ibis/backends/base/sql/__init__.py:360: error: Missing positional argument "cls" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]

jax (https://github.com/google/jax)
+ jax/_src/sharding_impls.py:74: error: Signature of "devices_indices_map" incompatible with supertype "Sharding"  [override]
+ jax/_src/sharding_impls.py:74: note:      Superclass:
+ jax/_src/sharding_impls.py:74: note:          def devices_indices_map(self, global_shape: tuple[int, ...]) -> Mapping[Device?, tuple[slice, ...] | None]
+ jax/_src/sharding_impls.py:74: note:      Subclass:
+ jax/_src/sharding_impls.py:74: note:          _lru_cache_wrapper[[XLACompatibleSharding, tuple[int, ...]], Mapping[Device?, tuple[slice, ...]]]
+ jax/_src/sharding_impls.py:85: error: Signature of "shard_shape" incompatible with supertype "Sharding"  [override]
+ jax/_src/sharding_impls.py:85: note:      Superclass:
+ jax/_src/sharding_impls.py:85: note:          def shard_shape(self, global_shape: tuple[int, ...]) -> tuple[int, ...]
+ jax/_src/sharding_impls.py:85: note:      Subclass:
+ jax/_src/sharding_impls.py:85: note:          _lru_cache_wrapper[[XLACompatibleSharding, tuple[int, ...]], tuple[int, ...]]
+ jax/_src/sharding_impls.py:344: error: Signature of "_to_xla_hlo_sharding" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding_impls.py:344: note:      Superclass:
+ jax/_src/sharding_impls.py:344: note:          def _to_xla_hlo_sharding(self, num_dimensions: int) -> Any
+ jax/_src/sharding_impls.py:344: note:      Subclass:
+ jax/_src/sharding_impls.py:344: note:          _lru_cache_wrapper[[NamedSharding, int, SPMDAxisContext | ShardingContext | None], Any]
+ jax/_src/sharding_impls.py:521: error: Signature of "devices_indices_map" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding_impls.py:521: note:      Superclass:
+ jax/_src/sharding_impls.py:521: note:          _lru_cache_wrapper[[XLACompatibleSharding, tuple[int, ...]], Mapping[Device?, tuple[slice, ...]]]
+ jax/_src/sharding_impls.py:521: note:      Subclass:
+ jax/_src/sharding_impls.py:521: note:          _lru_cache_wrapper[[PmapSharding, tuple[int, ...]], Mapping[Device?, tuple[slice, ...]]]
+ jax/_src/sharding_impls.py:521: error: Signature of "devices_indices_map" incompatible with supertype "Sharding"  [override]
+ jax/_src/sharding_impls.py:521: note:      Superclass:
+ jax/_src/sharding_impls.py:521: note:          def devices_indices_map(self, global_shape: tuple[int, ...]) -> Mapping[Device?, tuple[slice, ...] | None]
+ jax/_src/sharding_impls.py:521: note:      Subclass:
+ jax/_src/sharding_impls.py:521: note:          _lru_cache_wrapper[[PmapSharding, tuple[int, ...]], Mapping[Device?, tuple[slice, ...]]]
+ jax/_src/sharding_impls.py:522: error: Too few arguments  [call-arg]
+ jax/_src/sharding_impls.py:522: error: Argument 1 has incompatible type "tuple[int, ...]"; expected "PmapSharding"  [arg-type]
+ jax/_src/sharding_impls.py:545: error: Signature of "shard_shape" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding_impls.py:545: note:      Superclass:
+ jax/_src/sharding_impls.py:545: note:          _lru_cache_wrapper[[XLACompatibleSharding, tuple[int, ...]], tuple[int, ...]]
+ jax/_src/sharding_impls.py:545: note:      Subclass:
+ jax/_src/sharding_impls.py:545: note:          _lru_cache_wrapper[[PmapSharding, tuple[int, ...]], tuple[int, ...]]
+ jax/_src/sharding_impls.py:545: error: Signature of "shard_shape" incompatible with supertype "Sharding"  [override]
+ jax/_src/sharding_impls.py:545: note:      Superclass:
+ jax/_src/sharding_impls.py:545: note:          def shard_shape(self, global_shape: tuple[int, ...]) -> tuple[int, ...]
+ jax/_src/sharding_impls.py:545: note:      Subclass:
+ jax/_src/sharding_impls.py:545: note:          _lru_cache_wrapper[[PmapSharding, tuple[int, ...]], tuple[int, ...]]
+ jax/_src/sharding_impls.py:690: error: Signature of "_to_xla_hlo_sharding" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding_impls.py:690: note:      Superclass:
+ jax/_src/sharding_impls.py:690: note:          def _to_xla_hlo_sharding(self, num_dimensions: int) -> Any
+ jax/_src/sharding_impls.py:690: note:      Subclass:
+ jax/_src/sharding_impls.py:690: note:          _lru_cache_wrapper[[PositionalSharding, int], Any]
+ jax/_src/sharding_impls.py:826: error: Signature of "devices_indices_map" incompatible with supertype "XLACompatibleSharding"  [override]
+ jax/_src/sharding_impls.py:826: note:      Superclass:
+ jax/_src/sharding_impls.py:826: note:          _lru_cache_wrapper[[XLACompatibleSharding, tuple[int, ...]], Mapping[Device?, tuple[slice, ...]]]
+ jax/_src/sharding_impls.py:826: note:      Subclass:
+ jax/_src/sharding_impls.py:826: note:          _lru_cache_wrapper[[GSPMDSharding, tuple[int, ...]], Mapping[Device?, tuple[slice, ...]]]
+ jax/_src/sharding_impls.py:826: error: Signature of "devices_indices_map" incompatible with supertype "Sharding"  [override]
+ jax/_src/sharding_impls.py:826: note:      Superclass:
+ jax/_src/sharding_impls.py:826: note:          def devices_indices_map(self, global_shape: tuple[int, ...]) -> Mapping[Device?, tuple[slice, ...] | None]
+ jax/_src/sharding_impls.py:826: note:      Subclass:
+ jax/_src/sharding_impls.py:826: note:          _lru_cache_wrapper[[GSPMDSharding, tuple[int, ...]], Mapping[Device?, tuple[slice, ...]]]
+ jax/_src/sharding_impls.py:827: error: Too few arguments  [call-arg]
+ jax/_src/sharding_impls.py:827: error: Argument 1 has incompatible type "tuple[int, ...]"; expected "GSPMDSharding"  [arg-type]
+ jax/_src/interpreters/mlir.py:575: error: Too few arguments  [call-arg]
+ jax/_src/interpreters/mlir.py:575: error: Argument 1 has incompatible type "Any | tuple[int | DArray | Tracer | Var | DBIdx | InDBIdx | OutDBIdx, ...]"; expected "XLACompatibleSharding"  [arg-type]
+ jax/_src/interpreters/pxla.py:1088: error: Too few arguments  [call-arg]
+ jax/experimental/multihost_utils.py:250: error: Too few arguments  [call-arg]
+ jax/experimental/multihost_utils.py:357: error: Too few arguments  [call-arg]
+ jax/experimental/jax2tf/shape_poly.py:1218: error: Too few arguments  [call-arg]
+ jax/experimental/jax2tf/shape_poly.py:1218: error: Argument 1 has incompatible type "int | Any"; expected "CachingShapeEvaluator"  [arg-type]
+ jax/experimental/jax2tf/shape_poly.py:1240: error: Too few arguments  [call-arg]
+ jax/experimental/jax2tf/shape_poly.py:1240: error: Argument 1 has incompatible type "int | Any"; expected "CachingShapeEvaluator"  [arg-type]
+ jax/experimental/jax2tf/shape_poly.py:1292: error: Too few arguments  [call-arg]
+ jax/experimental/jax2tf/shape_poly.py:1292: error: Argument 1 has incompatible type "Any | int"; expected "CachingShapeEvaluator"  [arg-type]
+ jax/experimental/jax2tf/shape_poly.py:1466: error: Too few arguments  [call-arg]
+ jax/experimental/array_serialization/serialization.py:53: error: Too few arguments  [call-arg]
+ jax/experimental/array_serialization/serialization.py:53: error: Argument 1 has incompatible type "tuple[int, ...]"; expected "XLACompatibleSharding"  [arg-type]
+ jax/experimental/array_serialization/serialization.py:258: error: Too few arguments  [call-arg]
+ jax/experimental/array_serialization/serialization.py:258: error: Argument 1 has incompatible type "tuple[Any, ...]"; expected "XLACompatibleSharding"  [arg-type]
+ jax/experimental/jax2tf/jax_export.py:943: error: Too few arguments  [call-arg]

materialize (https://github.com/MaterializeInc/materialize)
+ misc/python/materialize/mzbuild.py:634: error: Too few arguments  [call-arg]
+ misc/python/materialize/mzbuild.py:802: error: Too few arguments  [call-arg]
+ misc/python/materialize/cli/mzimage.py:39: error: Too few arguments  [call-arg]
+ misc/python/materialize/cli/mzimage.py:47: error: Too few arguments  [call-arg]

paroxython (https://github.com/laowantong/paroxython)
+ paroxython/assess_costs.py:85: error: Incompatible types in assignment (expression has type "_lru_cache_wrapper[[int, int], float]", variable has type "Callable[[LearningCostAssessor, int, int], float]")  [assignment]
+ paroxython/assess_costs.py:85: note: "_lru_cache_wrapper[[int, int], float].__call__" has type "Callable[[Arg(int, 'start'), Arg(int, 'stop')], float]"
+ paroxython/assess_costs.py:89: error: "Callable[[LearningCostAssessor, LearningCostAssessor, TaxonName], float]" has no attribute "cache_clear"  [attr-defined]
+ paroxython/assess_costs.py:128: error: Too few arguments  [call-arg]
+ paroxython/assess_costs.py:128: error: Argument 1 has incompatible type "TaxonName"; expected "LearningCostAssessor"  [arg-type]
+ paroxython/recommend_programs.py:296: error: Too few arguments  [call-arg]
+ paroxython/recommend_programs.py:296: error: Argument 1 has incompatible type "TaxonName"; expected "LearningCostAssessor"  [arg-type]
+ paroxython/map_taxonomy.py:243: error: Too few arguments  [call-arg]
+ paroxython/map_taxonomy.py:243: error: Argument 1 has incompatible type "LabelName"; expected "Taxonomy"  [arg-type]

jinja (https://github.com/pallets/jinja)
+ src/jinja2/environment.py:1186: error: Unused "type: ignore" comment  [unused-ignore]
+ src/jinja2/environment.py:1201: error: Unused "type: ignore" comment  [unused-ignore]

Daverball

This comment was marked as resolved.

@Daverball
Copy link
Contributor

Daverball commented Aug 5, 2023

Considering how hard you have to work with ParamSpec and Concatenate just to get to something that's close to being correct for some cases, but still not all of them, I'm inclined to believe that these just aren't the right tools for it, and it would be better suited to an Intersection and would be yet another use case where Intersection would really shine and keep things relatively simple by comparison.

So in this case rather than trying to return a different type from lru_cache you just return an Intersection between the callable that's been passed in and a Protocol which contains the additional properties that were added to the function. i.e. something like this:

_F = TypeVar("_F", bound=Callable[..., Any])

class _CacheMethods(Protocol):
    def cache_info(self) -> _CacheInfo: ...
    def cache_clear(self) -> None: ...
    if sys.version_info >= (3, 9):
        def cache_parameters(self) -> _CacheParameters: ...

if sys.version_info >= (3, 8):
    @overload
    def lru_cache(maxsize: int | None = 128, typed: bool = False) -> Callable[[_F], Intersection[_F, _CacheMethods]]: ...
    @overload
    def lru_cache(maxsize: _F, typed: bool = False) -> Intersection[_F, _CacheMethods]: ...

else:
    def lru_cache(maxsize: int | None = 128, typed: bool = False) -> Callable[[_F], Intersection[_F, _CacheMethods]]: ...

But maybe even this wouldn't work correctly, so not sure.

@jakkdl
Copy link
Author

jakkdl commented Aug 6, 2023

Considering how hard you have to work with ParamSpec and Concatenate just to get to something that's close to being correct for some cases, but still not all of them, I'm inclined to believe that these just aren't the right tools for it, and it would be better suited to an Intersection and would be yet another use case where Intersection would really shine and keep things relatively simple by comparison.

Yeah after digging thoroughly into this I have more or less given up for the moment. It might be possible to improve the stubs a little bit merely using Concatenate, but in general the tools seems insufficient for the job. Intersection sounds kinda perfect, but with the PEP only being WIP I'm assuming it's not coming with 3.12

@JelleZijlstra
Copy link
Member

with the PEP only being WIP I'm assuming it's not coming with 3.12

Just FYI, assuming that PEP goes through, for typeshed usage the CPython release cycle doesn't really matter. The hypothetical timeline would go something like this:

  • The PEP is created and gets a number
  • typing_extensions.Intersection is added
  • Type checkers add support for intersections
  • The PEP is accepted
  • Typeshed starts using typing_extensions.Intersection
  • CPython implements the & operator for types (in 3.13 if this happens before ~May 2024, in 3.14 if after)

@srittau
Copy link
Collaborator

srittau commented Aug 7, 2023

@jakkdl Thanks for your work on this so far. Should we close the PR for the time being?

@jakkdl
Copy link
Author

jakkdl commented Aug 7, 2023

@jakkdl Thanks for your work on this so far. Should we close the PR for the time being?

yeah, might as well :)

@tamird
Copy link
Contributor

tamird commented Feb 23, 2024

For the uninitiated: which PEP are you discussing above?

@jakkdl
Copy link
Author

jakkdl commented Feb 23, 2024

For the uninitiated: which PEP are you discussing above?

python/typing#213 (comment) iirc

@Daverball
Copy link
Contributor

For the uninitiated: which PEP are you discussing above?

python/typing#213 (comment) iirc

Yes, exactly, the linked repository contains further information.

OrderedIntersection seems like it will get a PEP draft first however, courtesy of @mikeshardmind, read more about it here with the original pitch: CarliJoy/intersection_examples#38

@mikeshardmind
Copy link
Contributor

👀 Let me add this issue to the list of motivating cases, thanks for the mention!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants