Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make the built-in slice type generic in terms of its start, stop, and step #10174

Closed
wants to merge 3 commits into from
Closed

Make the built-in slice type generic in terms of its start, stop, and step #10174

wants to merge 3 commits into from

Conversation

daniil-berg
Copy link

Motivation

The idea for builtins.slice to be generic goes back as far as 2015 (see python/typing#159) and the discussion was ported to issue #8647 more recently. See the latter issue as well as the related discussion in python/mypy#2410 for some great points about why this may be useful.

In short, many libraries that provide some sequence-like class offer a slicing mechanism with non-integer parameters, such as for example a time series object containing datetime items that can be sliced by starting and/or stopping with a specific datetime and potentially restricting the slice step to a specific timedelta.

Implementation

As it is right now, it is fairly straightforward. builtins.slice is made generic and parameterized by three type variables without an upper bound. The __init__ signatures and the read-only start, stop, and step properties are adjusted accordingly.

Limitations

Missing type arguments

Since slice allows initialization with just one argument and such a call only binds the stop type argument, initializing it as e.g. slice(5) will leave the start and step types undefined, which in practice will probably have the type of the slice be inferred as slice[Any, int, Any]. But that will still cause mypy --stict so complain that the assignment needs to be clarified with a specific type annotation like e.g. s: slice[int, int, int] = slice(5).

Likewise, initialization with just two arguments is also possible and would leave the step type undefined.

No shorthand notation

With this naive implementation, the slice type can only ever be used with exactly three type arguments. Some people have suggested that it would be very convenient to have shorthand notations for implicitly setting one or two of the three type arguments.

slice[S, T] would be a reasonable shorthand for slice[S, T, Any].

slice[T] would be a reasonable shorthand either for slice[Any, T, Any] or slice[T, T, Any]. I tend to favor the latter because it makes sense to assume the start type to match the stop type, if only the stop type is specified.

In any case, no shorthand is currently supported.

Future improvements

The type variable defaults proposed in PEP 696 would greatly improve a generic slice type by partially solving the limitations mentioned above. Specifically, being able to use another type variable as a default picks up this exact issue.

I still thought it best to provide one imperfect solution now and incorporate those improvements later on. I am willing to pick up any future work directly related to this change, if this is deemed desirable at all.

Miscellaneous

I went ahead and modified every existing slice annotation in stdlib/stubs to slice[Any, Any, Any] for now (except the one in pyinstaller because there was an explicit comment indicating slice[None, None, None] may be more appropriate). I am pretty sure this is not optimal because at least in a few cases a more restrictive annotation (e.g. slice[int, int, int]) may be appropriate/obvious. But I thought I would put forward the proposal first and fine-tune it later, if the maintainers think it is worthwhile.

Running the CI tests I only got errors from pytype in all places, where I passed type arguments to slice:

pytype.pytd.visitors.ContainerError: Class builtins.slice is not a container

I am not sure why pytype is ignoring the typeshed stubs. But the rest of the tests seemed to pass normally.

I did not run the Mypy primer script locally. I am curious to see the results of the workflow triggered by this PR. I expect quite a few type-arg errors for obvious reasons.

@github-actions
Copy link
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

kornia (https://github.com/kornia/kornia)
+ kornia/geometry/subpix/nms.py:100: error: Need type annotation for "center"  [var-annotated]
+ kornia/geometry/subpix/nms.py:101: error: Need type annotation for "left"  [var-annotated]
+ kornia/geometry/subpix/nms.py:102: error: Need type annotation for "right"  [var-annotated]
+ kornia/contrib/models/sam/architecture/mask_decoder.py:90: error: Need type annotation for "mask_slice"  [var-annotated]
+ kornia/contrib/models/sam/architecture/mask_decoder.py:92: error: Argument 2 to "slice" has incompatible type "int"; expected "None"  [arg-type]
+ kornia/geometry/vector.py:25: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/vector.py:103: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/quaternion.py:75: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/filters/sobel.py:83: error: Need type annotation for "center"  [var-annotated]
+ kornia/filters/sobel.py:84: error: Need type annotation for "left"  [var-annotated]
+ kornia/filters/sobel.py:85: error: Need type annotation for "right"  [var-annotated]
+ kornia/geometry/boxes.py:219: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/boxes.py:224: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/boxes.py:736: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/boxes.py:741: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/liegroup/so3.py:52: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/liegroup/so2.py:59: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/liegroup/se3.py:80: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/liegroup/se2.py:81: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/keypoints.py:50: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/keypoints.py:54: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/keypoints.py:244: error: Missing type parameters for generic type "slice"  [type-arg]
+ kornia/geometry/keypoints.py:248: error: Missing type parameters for generic type "slice"  [type-arg]

manticore (https://github.com/trailofbits/manticore)
- tests/wasm/json2mc.py:103: note:     def __getitem__(self, slice, /) -> List[Any]
+ tests/wasm/json2mc.py:103: note:     def __getitem__(self, slice[Any, Any, Any], /) -> List[Any]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/tests/indexing/common.py:30: error: Need type annotation for "new_axes"  [var-annotated]
+ pandas/core/array_algos/transforms.py:34: error: Need type annotation for "axis_indexer"  [var-annotated]
+ pandas/core/array_algos/transforms.py:36: error: Argument 2 to "slice" has incompatible type "int"; expected "None"  [arg-type]
+ pandas/core/array_algos/take.py:531: error: Need type annotation for "outindexer"  [var-annotated]
+ pandas/core/algorithms.py:1430: error: Need type annotation for "na_indexer"  [var-annotated]
+ pandas/core/algorithms.py:1431: error: Argument 1 to "slice" has incompatible type "None"; expected "int"  [arg-type]
+ pandas/core/algorithms.py:1431: error: Argument 2 to "slice" has incompatible type "int"; expected "None"  [arg-type]
+ pandas/core/algorithms.py:1441: error: Need type annotation for "_res_indexer"  [var-annotated]
+ pandas/core/algorithms.py:1442: error: Argument 1 to "slice" has incompatible type "int"; expected "None"  [arg-type]
+ pandas/core/algorithms.py:1442: error: Argument 2 to "slice" has incompatible type "None"; expected "int"  [arg-type]
+ pandas/core/algorithms.py:1442: error: Argument 2 to "slice" has incompatible type "int"; expected "None"  [arg-type]
+ pandas/core/algorithms.py:1445: error: Need type annotation for "_lag_indexer"  [var-annotated]
+ pandas/core/algorithms.py:1446: error: Argument 1 to "slice" has incompatible type "None"; expected "int"  [arg-type]
+ pandas/core/algorithms.py:1446: error: Argument 2 to "slice" has incompatible type "int"; expected "None"  [arg-type]
+ pandas/core/arrays/sparse/accessor.py:283: error: Need type annotation for "sl"  [var-annotated]
+ pandas/core/indexing.py:90: error: Need type annotation for "_NS"  [var-annotated]
+ pandas/core/indexing.py:1324: error: Need type annotation for "indexer"  [var-annotated]
+ pandas/core/indexing.py:2469: error: Need type annotation for "new_key"  [var-annotated]
+ pandas/core/generic.py:4113: error: Need type annotation for "_indexer"  [var-annotated]
+ pandas/core/generic.py:10563: error: Need type annotation for "slicer"  [var-annotated]
+ pandas/core/window/rolling.py:1161: error: Need type annotation for "lead_indexer"  [var-annotated]
+ pandas/core/internals/managers.py:896: error: Need type annotation for "slc"  [var-annotated]
+ pandas/core/indexes/datetimelike.py:568: error: Need type annotation for "lslice"  [var-annotated]
+ pandas/io/formats/csvs.py:307: error: Need type annotation for "slicer"  [var-annotated]

sympy (https://github.com/sympy/sympy)
+ sympy/functions/combinatorial/numbers.py:1688: error: Need type annotation for "_M"  [var-annotated]

pytest (https://github.com/pytest-dev/pytest)
+ src/_pytest/_code/source.py:53: error: Missing type parameters for generic type "slice"  [type-arg]
+ src/_pytest/_code/source.py:56: error: Missing type parameters for generic type "slice"  [type-arg]
+ src/_pytest/_code/code.py:389: error: Missing type parameters for generic type "slice"  [type-arg]
+ src/_pytest/_code/code.py:393: error: Missing type parameters for generic type "slice"  [type-arg]

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/utilities/annotations.py:48: note:          def __getitem__(self, slice, /) -> Tuple[<nothing>, ...]
+ src/prefect/utilities/annotations.py:48: note:          def __getitem__(self, slice[Any, Any, Any], /) -> Tuple[<nothing>, ...]
- src/prefect/utilities/annotations.py:48: note:          def __getitem__(self, slice, /) -> Sequence[<nothing>]
+ src/prefect/utilities/annotations.py:48: note:          def __getitem__(self, slice[Any, Any, Any], /) -> Sequence[<nothing>]

steam.py (https://github.com/Gobot1234/steam.py)
- steam/clan.py:332: error: Invalid index type "str" for "Union[Tag, NavigableString]"; expected type "Union[SupportsIndex, slice]"  [index]
+ steam/clan.py:332: error: Invalid index type "str" for "Union[Tag, NavigableString]"; expected type "Union[SupportsIndex, slice[Any, Any, Any]]"  [index]
- steam/clan.py:337: error: Invalid index type "str" for "Union[Tag, NavigableString]"; expected type "Union[SupportsIndex, slice]"  [index]
+ steam/clan.py:337: error: Invalid index type "str" for "Union[Tag, NavigableString]"; expected type "Union[SupportsIndex, slice[Any, Any, Any]]"  [index]

operator (https://github.com/canonical/operator)
- ops/model.py:2801: note:     def __getitem__(self, slice, /) -> List[JsonObject]
+ ops/model.py:2801: note:     def __getitem__(self, slice[Any, Any, Any], /) -> List[JsonObject]
- ops/model.py:2801: note:     def __getitem__(self, slice, /) -> Tuple[JsonObject, ...]
+ ops/model.py:2801: note:     def __getitem__(self, slice[Any, Any, Any], /) -> Tuple[JsonObject, ...]
- ops/model.py:2801: error: Invalid index type "str" for "Union[str, None, int, float, Dict[str, JsonObject], List[JsonObject], Tuple[JsonObject, ...]]"; expected type "Union[SupportsIndex, slice]"  [index]
+ ops/model.py:2801: error: Invalid index type "str" for "Union[str, None, int, float, Dict[str, JsonObject], List[JsonObject], Tuple[JsonObject, ...]]"; expected type "Union[SupportsIndex, slice[Any, Any, Any]]"  [index]
- ops/model.py:2802: note:     def __getitem__(self, slice, /) -> List[JsonObject]
+ ops/model.py:2802: note:     def __getitem__(self, slice[Any, Any, Any], /) -> List[JsonObject]
- ops/model.py:2802: note:     def __getitem__(self, slice, /) -> Tuple[JsonObject, ...]
+ ops/model.py:2802: note:     def __getitem__(self, slice[Any, Any, Any], /) -> Tuple[JsonObject, ...]
- ops/model.py:2802: error: Invalid index type "str" for "Union[str, Any, None, int, float, Dict[str, JsonObject], List[JsonObject], Tuple[JsonObject, ...]]"; expected type "Union[SupportsIndex, slice]"  [index]
+ ops/model.py:2802: error: Invalid index type "str" for "Union[str, Any, None, int, float, Dict[str, JsonObject], List[JsonObject], Tuple[JsonObject, ...]]"; expected type "Union[SupportsIndex, slice[Any, Any, Any]]"  [index]
- ops/model.py:2803: note:     def __getitem__(self, slice, /) -> List[JsonObject]
+ ops/model.py:2803: note:     def __getitem__(self, slice[Any, Any, Any], /) -> List[JsonObject]
- ops/model.py:2803: note:     def __getitem__(self, slice, /) -> Tuple[JsonObject, ...]
+ ops/model.py:2803: note:     def __getitem__(self, slice[Any, Any, Any], /) -> Tuple[JsonObject, ...]
- ops/model.py:2803: error: Invalid index type "str" for "Union[str, Any, None, int, float, Dict[str, JsonObject], List[JsonObject], Tuple[JsonObject, ...]]"; expected type "Union[SupportsIndex, slice]"  [index]
+ ops/model.py:2803: error: Invalid index type "str" for "Union[str, Any, None, int, float, Dict[str, JsonObject], List[JsonObject], Tuple[JsonObject, ...]]"; expected type "Union[SupportsIndex, slice[Any, Any, Any]]"  [index]
- ops/framework.py:1252: note:          def __getitem__(self, slice, /) -> MutableSequence[_StorableType]
+ ops/framework.py:1252: note:          def __getitem__(self, slice[Any, Any, Any], /) -> MutableSequence[_StorableType]
- ops/framework.py:1252: note:          def __getitem__(self, slice, /) -> Sequence[_StorableType]
+ ops/framework.py:1252: note:          def __getitem__(self, slice[Any, Any, Any], /) -> Sequence[_StorableType]
- ops/framework.py:1255: note:          def __setitem__(self, slice, Iterable[_StorableType], /) -> None
+ ops/framework.py:1255: note:          def __setitem__(self, slice[Any, Any, Any], Iterable[_StorableType], /) -> None
- ops/framework.py:1259: note:          def __delitem__(self, slice, /) -> None
+ ops/framework.py:1259: note:          def __delitem__(self, slice[Any, Any, Any], /) -> None

more-itertools (https://github.com/more-itertools/more-itertools)
+ more_itertools/more.pyi:43: error: Missing type parameters for generic type "slice"  [type-arg]
+ more_itertools/more.pyi:74: error: Missing type parameters for generic type "slice"  [type-arg]
+ more_itertools/more.pyi:367: error: Missing type parameters for generic type "slice"  [type-arg]
+ more_itertools/more.pyi:404: error: Missing type parameters for generic type "slice"  [type-arg]
+ more_itertools/more.pyi:427: error: Missing type parameters for generic type "slice"  [type-arg]

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ tests/test_windowing.py:357: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:172: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:173: error: Expression is of type "Any", not "Index"  [assert-type]
+ tests/test_resampler.py:178: error: Expression is of type "Any", not "Tuple[Hashable, Series[Any]]"  [assert-type]
+ tests/test_resampler.py:182: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:183: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:184: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:185: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:186: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:187: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:188: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:189: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:190: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:191: error: Expression is of type "Any", not "DataFrame"  [assert-type]
+ tests/test_resampler.py:192: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:196: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:197: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:199: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:205: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:206: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:210: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:211: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:215: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:216: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:217: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:221: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:222: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:223: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:224: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:225: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:229: error: Expression is of type "Any", not "Union[DataFrame, Series[Any]]"  [assert-type]
+ tests/test_resampler.py:230: error: Expression is of type "Any", not "Union[DataFrame, Series[Any]]"  [assert-type]
+ tests/test_resampler.py:231: error: Expression is of type "Any", not "Union[DataFrame, Series[Any]]"  [assert-type]
+ tests/test_resampler.py:233: error: Expression is of type "Any", not "Union[DataFrame, Series[Any]]"  [assert-type]
+ tests/test_resampler.py:237: error: Expression is of type "Any", not "Union[DataFrame, Series[Any]]"  [assert-type]
+ tests/test_resampler.py:241: error: Expression is of type "Any", not "Union[DataFrame, Series[Any]]"  [assert-type]
+ tests/test_resampler.py:251: error: Expression is of type "Any", not "Union[DataFrame, Series[Any]]"  [assert-type]
+ tests/test_resampler.py:255: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:259: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:260: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_resampler.py:264: error: Expression is of type "Any", not "None"  [assert-type]
+ tests/test_resampler.py:276: error: Expression is of type "Any", not "Union[Union[str, bytes, date, datetime, timedelta, Any, Any, bool, int, float, Timestamp, Timedelta], complex]"  [assert-type]
+ tests/test_resampler.py:288: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_plotting.py:549: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "Series[Any]")  [assignment]
- tests/test_plotting.py:569: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "Series[Any]")  [assignment]
+ tests/test_frame.py:173: error: Expression is of type "Any", not "DataFrame"  [assert-type]
+ tests/test_frame.py:174: error: Expression is of type "Any", not "DataFrame"  [assert-type]
- tests/test_frame.py:1424: error: Incompatible types in assignment (expression has type "Series[Any]", variable has type "DataFrame")  [assignment]
+ tests/test_frame.py:1647: error: Expression is of type "Any", not "DataFrame"  [assert-type]
+ tests/test_frame.py:1648: error: Expression is of type "Any", not "DataFrame"  [assert-type]
+ tests/test_frame.py:1650: error: Expression is of type "Any", not "DataFrame"  [assert-type]
+ tests/test_frame.py:1655: error: Expression is of type "Any", not "DataFrame"  [assert-type]
- tests/test_frame.py:1660: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1660: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1797: error: Incompatible types in assignment (expression has type "Series[Any]", variable has type "DataFrame")  [assignment]
- tests/test_frame.py:1936: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1936: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1937: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1937: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1938: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1938: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1939: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1939: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1940: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1940: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1941: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1941: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1942: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1942: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1943: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1943: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1944: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1944: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1945: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1945: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1946: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1946: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
- tests/test_frame.py:1953: error: Expression is of type "DataFrame", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:1953: error: Expression is of type "Any", not "Series[Any]"  [assert-type]
+ tests/test_frame.py:2182: error: Expression is of type "Any", not "DataFrame"  [assert-type]
+ tests/test_frame.py:2183: error: Expression is of type "Any", not "DataFrame"  [assert-type]
- tests/test_frame.py:2275: error: Expression is of type "DataFrame", not "Union[Series[Any], DataFrame]"  [assert-type]
+ tests/test_frame.py:2275: error: Expression is of type "Any", not "Union[Series[Any], DataFrame]"  [assert-type]
- tests/test_frame.py:2629: error: Expression is of type "DataFrame", not "Union[Series[Any], DataFrame]"  [assert-type]
+ tests/test_frame.py:2629: error: Expression is of type "Any", not "Union[Series[Any], DataFrame]"  [assert-type]
- tests/test_series.py:146: error: Expression is of type "Timestamp", not "Series[int]"  [assert-type]
+ tests/test_series.py:146: error: Expression is of type "Any", not "Series[int]"  [assert-type]
- tests/test_series.py:147: error: Expression is of type "Timestamp", not "Series[int]"  [assert-type]
+ tests/test_series.py:147: error: Expression is of type "Any", not "Series[int]"  [assert-type]

ibis (https://github.com/ibis-project/ibis)
- ibis/selectors.py:475: error: Item "slice" of "Union[slice, Iterable[Union[int, str]]]" has no attribute "__iter__" (not iterable)  [union-attr]
+ ibis/selectors.py:475: error: Item "slice[Any, Any, Any]" of "Union[slice[Any, Any, Any], Iterable[Union[int, str]]]" has no attribute "__iter__" (not iterable)  [union-attr]
- ibis/selectors.py:478: error: Item "Iterable[Union[int, str]]" of "Union[slice, Iterable[Union[int, str]]]" has no attribute "start"  [union-attr]
+ ibis/selectors.py:478: error: Item "Iterable[Union[int, str]]" of "Union[slice[Any, Any, Any], Iterable[Union[int, str]]]" has no attribute "start"  [union-attr]
- ibis/selectors.py:479: error: Item "Iterable[Union[int, str]]" of "Union[slice, Iterable[Union[int, str]]]" has no attribute "stop"  [union-attr]
+ ibis/selectors.py:479: error: Item "Iterable[Union[int, str]]" of "Union[slice[Any, Any, Any], Iterable[Union[int, str]]]" has no attribute "stop"  [union-attr]
- ibis/selectors.py:480: error: Item "Iterable[Union[int, str]]" of "Union[slice, Iterable[Union[int, str]]]" has no attribute "step"  [union-attr]
+ ibis/selectors.py:480: error: Item "Iterable[Union[int, str]]" of "Union[slice[Any, Any, Any], Iterable[Union[int, str]]]" has no attribute "step"  [union-attr]

jax (https://github.com/google/jax)
+ jax/_src/core.py:1706: error: Need type annotation for "slices"  [var-annotated]
+ jax/_src/core.py:1706: error: Argument 1 to "slice" has incompatible type "int"; expected "None"  [arg-type]
+ jax/_src/numpy/fft.py:87: error: Argument 1 to "map" has incompatible type "Type[slice[Any, Any, Any]]"; expected "Callable[[int], slice[_T1, _T2, _T3]]"  [arg-type]
+ jax/_src/prng.py:666: error: Need type annotation for "trailing_inds"  [var-annotated]
+ jax/_src/numpy/lax_numpy.py:353: error: Need type annotation for "out_order"  [var-annotated]
+ jax/_src/numpy/lax_numpy.py:650: error: Need type annotation for "slice1"  [var-annotated]
+ jax/_src/numpy/lax_numpy.py:651: error: Need type annotation for "slice2"  [var-annotated]
+ jax/_src/numpy/lax_numpy.py:653: error: Argument 2 to "slice" has incompatible type "int"; expected "None"  [arg-type]
+ jax/_src/image/scale.py:264: error: Need type annotation for "indices"  [var-annotated]
+ jax/_src/random.py:560: error: Need type annotation for "slices"  [var-annotated]
+ jax/_src/scipy/signal.py:216: error: Need type annotation for "sl"  [var-annotated]
+ jax/experimental/sparse/bcoo.py:1925: error: Argument 1 to "slice" has incompatible type "None"; expected "int"  [arg-type]
+ jax/experimental/sparse/bcoo.py:1926: error: Argument 1 to "slice" has incompatible type "None"; expected "int"  [arg-type]
+ jax/experimental/sparse/bcoo.py:1927: error: Argument 1 to "slice" has incompatible type "None"; expected "int"  [arg-type]
+ jax/experimental/sparse/bcoo.py:1928: error: Argument 1 to "slice" has incompatible type "None"; expected "int"  [arg-type]

rich (https://github.com/Textualize/rich)
+ rich/containers.py:83: error: Missing type parameters for generic type "slice"  [type-arg]
+ rich/containers.py:86: error: Missing type parameters for generic type "slice"  [type-arg]
+ rich/text.py:181: error: Missing type parameters for generic type "slice"  [type-arg]

psycopg (https://github.com/psycopg/psycopg)
+ psycopg/psycopg/types/multirange.py:97: error: Missing type parameters for generic type "slice"  [type-arg]
+ psycopg/psycopg/types/multirange.py:100: error: Missing type parameters for generic type "slice"  [type-arg]
+ psycopg/psycopg/types/multirange.py:114: error: Missing type parameters for generic type "slice"  [type-arg]
+ psycopg/psycopg/types/multirange.py:119: error: Missing type parameters for generic type "slice"  [type-arg]
+ psycopg/psycopg/types/multirange.py:131: error: Missing type parameters for generic type "slice"  [type-arg]

schema_salad (https://github.com/common-workflow-language/schema_salad)
+ schema_salad/tests/test_misc.py:3: note: In module imported here:
+ mypy-stubs/rdflib/graph.pyi: note: In member "__getitem__" of class "Graph":
+ mypy-stubs/rdflib/graph.pyi:54:21: error: Missing type parameters for generic type "slice"  [type-arg]
+ mypy-stubs/rdflib/graph.pyi:15: note: In module imported here,
+ schema_salad/tests/test_misc.py:3: note: ... from here:
+ mypy-stubs/rdflib/query.pyi: note: In function "__getitem__":
+ mypy-stubs/rdflib/query.pyi:17:32: error: Missing type parameters for generic type "slice"  [type-arg]

bokeh (https://github.com/bokeh/bokeh)
+ src/bokeh/core/serialization.py: note: In member "_encode_slice" of class "Serializer":
+ src/bokeh/core/serialization.py:372:34: error: Missing type parameters for generic type "slice"  [type-arg]
+ src/bokeh/core/serialization.py: note: At top level:
+ src/bokeh/core/serialization.py: note: In member "_decode_slice" of class "Deserializer":
+ src/bokeh/core/serialization.py:608:47: error: Missing type parameters for generic type "slice"  [type-arg]

nionutils (https://github.com/nion-software/nionutils)
+ nion/utils/Geometry.py:635: error: Missing type parameters for generic type "slice"  [type-arg]

aiohttp (https://github.com/aio-libs/aiohttp)
+ aiohttp/web_routedef.py:168: error: Missing type parameters for generic type "slice"  [type-arg]
+ aiohttp/web_request.py:578: error: Missing type parameters for generic type "slice"  [type-arg]

jinja (https://github.com/pallets/jinja)
+ src/jinja2/nodes.py:885: error: Missing type parameters for generic type "slice"  [type-arg]

discord.py (https://github.com/Rapptz/discord.py)
- discord/http.py:226: note:     def __setitem__(self, slice, Iterable[Embed], /) -> None
+ discord/http.py:226: note:     def __setitem__(self, slice[Any, Any, Any], Iterable[Embed], /) -> None

cwltool (https://github.com/common-workflow-language/cwltool)
+ cwltool/command_line_tool.py:13: note: In module imported here:
+ mypy-stubs/urllib/parse.pyi: note: In function "__getitem__":
+ mypy-stubs/urllib/parse.pyi:53:30: error: Missing type parameters for generic type "slice"  [type-arg]
+ cwltool/builder.py:28: note: ... from here:
+ mypy-stubs/rdflib/graph.pyi: note: In member "__getitem__" of class "Graph":
+ mypy-stubs/rdflib/graph.pyi:54:21: error: Missing type parameters for generic type "slice"  [type-arg]
+ cwltool/cwlrdf.py:6: note: In module imported here:
+ mypy-stubs/rdflib/query.pyi: note: In function "__getitem__":
+ mypy-stubs/rdflib/query.pyi:17:32: error: Missing type parameters for generic type "slice"  [type-arg]

Tanjun (https://github.com/FasterSpeeding/Tanjun)
+ tanjun/annotations.py:2409: error: Missing type parameters for generic type "slice"  [type-arg]

Expression (https://github.com/cognitedata/Expression)
- expression/collections/array.py:483: note:          def __setitem__(self, slice, Iterable[_TSource], /) -> None
+ expression/collections/array.py:483: note:          def __setitem__(self, slice[Any, Any, Any], Iterable[_TSource], /) -> None
- expression/collections/array.py:489: note:          def __delitem__(self, slice, /) -> None
+ expression/collections/array.py:489: note:          def __delitem__(self, slice[Any, Any, Any], /) -> None

speedrun.com_global_scoreboard_webapp (https://github.com/Avasam/speedrun.com_global_scoreboard_webapp)
- backend/api/tournament_scheduler_api.py:327: note:     def __getitem__(self, slice, /) -> List[Any]
+ backend/api/tournament_scheduler_api.py:327: note:     def __getitem__(self, slice[Any, Any, Any], /) -> List[Any]
- backend/api/tournament_scheduler_api.py:331: note:     def __getitem__(self, slice, /) -> List[Any]
+ backend/api/tournament_scheduler_api.py:331: note:     def __getitem__(self, slice[Any, Any, Any], /) -> List[Any]
- backend/api/tournament_scheduler_api.py:335: note:     def __getitem__(self, slice, /) -> List[Any]
+ backend/api/tournament_scheduler_api.py:335: note:     def __getitem__(self, slice[Any, Any, Any], /) -> List[Any]

@srittau
Copy link
Collaborator

srittau commented May 12, 2023

While I agree that the slice type should be generic, doing it before we have type var defaults is just too disruptive, as the primer hits show.

@AlexWaygood
Copy link
Member

I agree with @srittau. Without PEP 696, this would just be too much of a compatibility break, unfortunately. It sounds like we might have to wait for a year or so before we get a decision from the Steering Council on PEP 696, so I'm closing this PR for now. But thank you for it -- we can reconsider once a decision on PEP 696 has been made.

Running the CI tests I only got errors from pytype in all places, where I passed type arguments to slice:

pytype.pytd.visitors.ContainerError: Class builtins.slice is not a container

I am not sure why pytype is ignoring the typeshed stubs.

Pytype has its own version of builtins.pyi and typing.pyi; it doesn't pay attention to what typeshed has for these modules. This often causes a bit of confusion in our CI :)

@srittau
Copy link
Collaborator

srittau commented May 12, 2023

@AlexWaygood Should we mark this PR as deferred and leave it open?

@AlexWaygood
Copy link
Member

@AlexWaygood Should we mark this PR as deferred and leave it open?

I'm not really a fan of leaving PRs open for extended periods of time. They tend to be hard to keep up to date (merge conflicts tend to stack up), so it's generally easier just to recreate them when the time comes. And it sounds like the Steering Council is resolute in not taking a decision on PEP 696 for quite a while (unfortunately). We already have #8647 open to remind us to look into this once PEP 696 is (hopefully) accepted.

@daniil-berg
Copy link
Author

Fair enough. Thank you for taking the time to respond and elaborate. I guess it makes sense not to rely on people keeping their PR branches up to date for extended periods of time.

For what it's worth, I agree with your position on the order of PEP 695 vs. PEP 696 and wish the SC had come to a different decision.

With a bit of luck I'll be able to keep this up to date and ready for when PEP 696 lands. Then maybe you can reopen this in the future.

@AlexWaygood
Copy link
Member

With a bit of luck I'll be able to keep this up to date and ready for when PEP 696 lands. Then maybe you can reopen this in the future.

@daniil-berg absolutely. Very happy to reconsider when that time comes :)

@jorwoods
Copy link

PEP 696 has been accepted and will be in Python 3.13.

@daniil-berg
Copy link
Author

Hi @AlexWaygood and @srittau. Since PEP 696 has been accepted, I overhauled the branch to incorporate type parameter defaults for the start, stop, and step parameters. I just rebased the branch on top of the current main. The commit message basically spells out what I changed.

The type parmeters default to Any. I introduced the IndexSlice type alias for slice[SupportsIndex | None, SupportsIndex | None, SupportsIndex | None] for convenience because I use that type a lot throughout the stubs.

Basically, wherever there is an overload for a method that accepts either a SupportsIndex or a slice argument, I replaced slice with IndexSlice. There were a few instances throughout the stubs, where similarly overloaded methods accepted int or slice. (Example: SupportsGetItemBuffer) Since I wanted my changes to be minimally invasive, I took that at face value and replaced slice with slice[int | None, int | None, int | None] instead of my new IndexSlice. But I have a feeling that those int annotations were an oversight, so I can change that, if you like.

There was also some weird stuff going on in the AST modules, where slice was being shadowed. I did not touch those.

The slice.__new__ method returns either slice[None, _T, None] or slice[_T1, _T2, _T3] depending on the overload signature.

I humbly ask for this PR to be re-opened and my proposal to be reconsidered.

@srittau
Copy link
Collaborator

srittau commented Sep 13, 2024

Unfortunately, the PR can't be reopened due to the rebase. Please open a new PR. Thanks for working on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@daniil-berg @srittau @AlexWaygood @jorwoods