Add stubs for the pyserial package

[hamdanal]

Stubs for https://pypi.org/project/pyserial/. This PR pyserial/pyserial#591 adds basic stubs for the library but it has been open for more than a year with no comment from the maintainers of pyserial. The author of the PR says they lost interest in it and will not follow up.

[hamdanal]

I don't know how to fix the stubtest errors

[hauntsaninja]

Looks like what's going on is that stubtest is trying to import serial.__main__, but that is raising SystemExit.

I can fix stubtest to catch this, but that means you'll probably need to turn stubtest off for this to get merged, at least until the fix is present in a released version of mypy. You can do this by setting skip = true in the tool.stubtest section of METADATA.toml

For the record, this is the result when I run it locally with the patch:

serial.__main__ failed to import, SystemExit: 2
serial.rfc2217.Serial.read is inconsistent, stub argument "__size" should be positional or keyword (remove leading double underscore)
serial.rs485.RS485.write is inconsistent, stub argument "data" differs from runtime argument "b"
serial.serialcli failed to import, ModuleNotFoundError: No module named 'System'
serial.serialjava failed to import, ImportError: No Java Communications API implementation found
serial.serialwin32 failed to import, ImportError: cannot import name 'WinDLL' from 'ctypes' (/Users/shantanu/.pyenv/versions/3.9.9/lib/python3.9/ctypes/__init__.py)
serial.tools.hexlify_codec.Codec.decode is inconsistent, stub argument "input" differs from runtime argument "data"
serial.tools.hexlify_codec.Codec.encode is inconsistent, stub argument "input" differs from runtime argument "data"
serial.tools.hexlify_codec.IncrementalDecoder.decode is inconsistent, stub argument "input" differs from runtime argument "data"
serial.tools.hexlify_codec.IncrementalEncoder.encode is inconsistent, stub argument "input" differs from runtime argument "data"
serial.tools.list_ports_windows failed to import, ImportError: cannot import name 'WinDLL' from 'ctypes' (/Users/shantanu/.pyenv/versions/3.9.9/lib/python3.9/ctypes/__init__.py)
serial.tools.miniterm.Console.__init__ is inconsistent, runtime does not have argument "miniterm"
serial.tools.miniterm.Console.sigint is not present at runtime
serial.tools.miniterm.ConsoleBase.__init__ is inconsistent, runtime does not have argument "miniterm"
serial.tools.miniterm.main is inconsistent, runtime does not have argument "serial_instance"
serial.urlhandler.protocol_cp2110 failed to import, ModuleNotFoundError: No module named 'hid'
serial.urlhandler.protocol_loop.Serial.read is inconsistent, stub argument "__size" should be positional or keyword (remove leading double underscore)
serial.urlhandler.protocol_rfc2217.Serial.read is inconsistent, stub argument "__size" should be positional or keyword (remove leading double underscore)
serial.urlhandler.protocol_socket.Serial.read is inconsistent, stub argument "__size" should be positional or keyword (remove leading double underscore)
serial.urlhandler.protocol_spy.FormatLog is not present at runtime
serial.urlhandler.protocol_spy.FormatLogHex is not present at runtime
serial.urlhandler.protocol_spy.Serial.write is inconsistent, stub argument "data" differs from runtime argument "tx"
serial.win32 failed to import, ImportError: cannot import name 'WinDLL' from 'ctypes' (/Users/shantanu/.pyenv/versions/3.9.9/lib/python3.9/ctypes/__init__.py)

[hamdanal]

Thank you for investigating @hauntsaninja. I applied the mypy patch locally to fix the stubtest errors. Now stubtest passes at least on linux on my machine.

You can do this by setting skip = true in the tool.stubtest section of METADATA.toml

This will skip all stubtest right? Is there a way to tell stubtest to just skip the problematic file?

[AlexWaygood]

Is there a way to tell stubtest to just skip the problematic file?

Not for now, no. Part of the design of stubtest is that it will attempt to import every submodule of a package at runtime in order to compare it to the stub. Since this is the only way of getting CI to pass on this PR (at least until mypy 1.0 is released), I took the liberty of adding skip = true to the METADATA.toml file for you :)

[hamdanal]

I took a look at the mypy primer diff of home assistant and here are some notes/questions:

• homeassistant/components/modem_callerid/config_flow.py:76: error: Argument 5 to "human_readable_device_name" has incompatible type "Optional[int]"; expected "Optional[str]" [arg-type]
• homeassistant/components/modem_callerid/config_flow.py:77: error: Argument 6 to "human_readable_device_name" has incompatible type "Optional[int]"; expected "Optional[str]" [arg-type]

These are genuine typing mistakes caught by mypy.

• homeassistant/components/usb/init.py:309: error: Argument 1 to "async_add_executor_job" of "HomeAssistant" has incompatible type "Callable[[bool], List[SysFS]]"; expected "Callable[..., List[ListPortInfo]]" [arg-type]
• homeassistant/components/crownstone/config_flow.py:62: error: Argument 1 to "list_ports_as_str" has incompatible type "List[SysFS]"; expected "List[ListPortInfo]" [arg-type]
• homeassistant/components/crownstone/config_flow.py:64: error: Argument 1 to "list_ports_as_str" has incompatible type "List[SysFS]"; expected "List[ListPortInfo]" [arg-type]

These are caused by list variance. SysFS is a sublass of ListPortInfo so they might have to use Sequence to properly type their functions.

• homeassistant/components/usb/utils.py:13: error: Argument 1 to "hex" has incompatible type "Optional[int]"; expected "Union[int, SupportsIndex]" [arg-type]
• homeassistant/components/usb/utils.py:14: error: Argument 1 to "hex" has incompatible type "Optional[int]"; expected "Union[int, SupportsIndex]" [arg-type]

These are flagged because ListPortInfo.vid and ListPortInfo.pid are assumed to never be None which is only true for USB devices according to https://github.com/pyserial/pyserial/blob/master/serial/tools/list_ports_common.py#L43-L45. Now I never used pyserial with a real serial port since almost all devices nowadays are USB devices and computers haven't had serial ports for years. I would imagine that almost all pyserial users only use USB devices as well. My question for you since you deal with this correctness vs practicality all the time when it comes to typing is, what would you consider in this case:

1. Annotate ListPortInfo.vid and ListPortInfo.pid as int and satisfy the use-cases of the majority of the users
2. Keep them as int | None so that they represent the runtime behavior.

[hauntsaninja]

Thanks for the analysis!

If you genuinely believe that almost everyone will use this for USB devices, I'm fine with using int.
A slightly more conservative option in cases like this is to use int | Any; this will flag obvious typing errors like ListPortInfo.pid + "str" but will also propagate Any through user's code.

I'm happy with whatever you choose, but do add a comment :-)

[hamdanal]

If you genuinely believe that almost everyone will use this for USB devices, I'm fine with using int.
A slightly more conservative option in cases like this is to use int | Any; this will flag obvious typing errors like ListPortInfo.pid + "str" but will also propagate Any through user's code.

Thanks for the feedback.
I would be surprised if people are using pyserial with devices other than USB but of course I can't be sure. I like you suggestion of Union-ing with Any, I'll choose it. I didn't know about this feature so thanks for teaching me something new (TIL) -- is it documented somewhere?

[hauntsaninja]

Thanks for the PR! I spot checked several things and they all looked good to me. Will merge tomorrow :-)

[hauntsaninja]

I'm not aware of any good documentation of the union with Any trick. I'm also not sure all type checkers handle it the exact same way (although I think both mypy and pyright do).

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

core (https://github.com/home-assistant/core)
+ homeassistant/components/usb/__init__.py:309: error: Argument 1 to "async_add_executor_job" of "HomeAssistant" has incompatible type "Callable[[bool], List[SysFS]]"; expected "Callable[..., List[ListPortInfo]]"  [arg-type]
+ homeassistant/components/modem_callerid/config_flow.py:76: error: Argument 5 to "human_readable_device_name" has incompatible type "Union[int, Any]"; expected "Optional[str]"  [arg-type]
+ homeassistant/components/modem_callerid/config_flow.py:77: error: Argument 6 to "human_readable_device_name" has incompatible type "Union[int, Any]"; expected "Optional[str]"  [arg-type]
+ homeassistant/components/crownstone/config_flow.py:62: error: Argument 1 to "list_ports_as_str" has incompatible type "List[SysFS]"; expected "List[ListPortInfo]"  [arg-type]
+ homeassistant/components/crownstone/config_flow.py:62: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ homeassistant/components/crownstone/config_flow.py:62: note: Consider using "Sequence" instead, which is covariant
+ homeassistant/components/crownstone/config_flow.py:64: error: Argument 1 to "list_ports_as_str" has incompatible type "List[SysFS]"; expected "List[ListPortInfo]"  [arg-type]
+ homeassistant/components/crownstone/config_flow.py:64: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ homeassistant/components/crownstone/config_flow.py:64: note: Consider using "Sequence" instead, which is covariant

[hauntsaninja]

Thank you again!