Typing support for dtypes

[shoyer]

Per #7370 (comment), we could specify dtype as part of ndarray types (including type inference) by using NumPy's scalar types, e.g., np.ndarray[Any, np.float64] or np.ndarray[np.float64, Any] (where Any indicates the shape).

Is this a good way to indicate dtypes? How would we handle dtypes that don't have a well specified scalar type? For example:

• datetime64 and timedelta64 do not have precision as part of the type
• np.void is used for all arbitrary structured dtypes, but doesn't indicate the field names or subtypes (side note: np.struct would be a much better name)
• string dtypes have their size as part of the type

One simple answer is to not worry about more detailed type information for now, and worry about more precise specification for scalar types later, when support in the typing module becomes available (e.g., for integer types or custom struct-like data structures like TypedDict).

[rmcgibbo]

For complex dtypes like datetime64, np.void, and the string dtypes, I propose that we delay dealing with them until this package is much more fully developed.

Very little functionality is lost in the meantime if numpy functions that return arrays of dtype('S1') are labeled as returning (or are inferred to return) results of type ndarray[Any, str]. More specificity can be added later.

[ryanpeach]

This is really all I need, what is the issue? Anything I can do?

[person142]

Circling back around to this: another issue is that you can mutate an array's dtype in-place, e.g.

>>> x = np.ones((2,), dtype=np.int64)
>>> x
array([1, 1])
>>> x.dtype = np.bool_
>>> x
array([ True, False, False, False, False, False, False, False,  True,
       False, False, False, False, False, False, False])

So if you tried to add types:

x: np.ndarray[np.int64] = np.ones((2,), dtype=np.int64)
x.dtype = np.bool_  # Uh oh, the type is incorrect now.

I imagine that most of the time people don't do that (and use views instead), which we could handle quite nicely in some cases with the right overload:

T = TypeVar('T', bound=dtype)

class ndarray(...):
    ...
    @overload
    def view(self, dtype: T = ...) -> ndarray[T]: ...

A practical solution might be to say "if you are going to set dtype, you are on your own", and ask that in such cases a user do

x: np.ndarray[Any] = np.ones((2,), dtype=np.int64)
x.dtype = np.bool_  # Still ok

Thoughts?

[shoyer]

I don't think we should support modifying array dtype in place. There's really not good reason to do this rather than making a new view.

Likewise, I don't think we should support mutating shape, though if I recall that may be the only way to do reshaping while ensuring that an error would be raised if a copy would be needed.

[person142]

I don't think we should support modifying array dtype in place

Should we go so far as removing the setter from the types?

https://github.com/numpy/numpy-stubs/blob/master/numpy-stubs/__init__.pyi#L300

[person142]

PR banning the setter: numpy/numpy-stubs#47

[person142]

Updating this issue with my current plan to support dtypes; see also the discussion with @seberg in numpy/numpy-stubs#48.

• Stubs have been moved into NumPy itself, and API: Create Preliminary DTypeMeta class and np.dtype subclasses #15508 has been merged so we have dtype subclasses on master

• We write a mypy plugin using get_dynamic_class_hook:

  https://mypy.readthedocs.io/en/latest/extending_mypy.html#current-list-of-plugin-hooks

  to support creating the dynamic dtype classes at type-checking time

• We can then make ndarray generic over dtype, and not just the scalar hierarchy

[BvB93]

The PR for introducing dtype support is up: #17719.

[davetapley]

Sanity check, should something like datetime64['us', UTC]) type check okay?
I get Expected no type arguments for class "datetime64" 🤔