Improve typing for BS4 element.Tag's get and get_attribute_list.

[kemus]

Currently,

• get() will always have None as a possible return type, even if the default is set.
• get_attribute_list() will never have None in the possible types inside the list it returns, even if the default allows for that.
• (minor issue) get() and get_attribute_list()'s typing doesn't allow for default to have a type that isn't str or list[str], even though that's technically allowed by the implementation.

For reference, here's what the implementation of get and get_attribute_list looks like:

    def get(self, key, default=None):
        """Returns the value of the 'key' attribute for the tag, or
        the value given for 'default' if it doesn't have that
        attribute."""
        return self.attrs.get(key, default)

    def get_attribute_list(self, key, default=None):
        """The same as get(), but always returns a list.

        :param key: The attribute to look for.
        :param default: Use this value if the attribute is not present
            on this PageElement.
        :return: A list of values, probably containing only a single
            value.
        """
        value = self.get(key, default)
        if not isinstance(value, list):
            value = [value]
        return value

This change improves the type handling for these methods by using the same idea as stdlib/builtins.pyi uses for dict.get().

This is what dict.get()'s typing looks like:

  @overload
  def get(self, key: _KT) -> _VT | None: ...
  @overload
  def get(self, key: _KT, default: _VT) -> _VT: ...
  @overload
  def get(self, key: _KT, default: _T) -> _VT | _T: ...

Since Tag.get takes a str key and returns a str or list[str] if the attribute exists, we have types similar to a dict[str, str | list[str]]:

  @overload
  def get(self, key: str) -> str | list | None: ...
  @overload
  def get(self, key: str, default: str | list) -> str | list: ...
  @overload
  def get(self, key: str, default: _T) -> str | list | _T: ...

Then because str | list == str | list | str | list, we don't actually need the second overload:

  @overload
  def get(self, key: str) -> str | list | None: ...
  @overload
  def get(self, key: str, default: _T) -> str | list | _T: ...

---

We can also do something similar for get_attribute_list():

  @overload
  def get_attribute_list(self, key: str) -> list[str | None]: ...
  @overload
  def get_attribute_list(self, key: str, default: _T) -> list[str | _T]: ...

Except this isn't quite right -- if the key isn't found and the default is a list[_T], the implementation returns it instead of a list[list[_T]], so we need to unwrap it:

  @overload
  def get_attribute_list(self, key: str) -> list[str | None]: ...
  @overload
  def get_attribute_list(self, key: str, default: list[_T]) -> list[str | _T]: ...
  @overload
  def get_attribute_list(self, key: str, default: _T) -> list[str | _T]: ...

• I've seen Document why having a union return type is often a problem mypy#1693, however I don't think it applies in this case because we're already returning a union including None in the get() method
• Using _typeshed.MaybeNone would incorrectly hide cases where the user really should be checking for None.

---

Here's an example of the types that mypy infers for a few cases before and after this change:

from __future__ import annotations

from bs4 import Tag

tag = Tag()
k = "class"
# str | list[str] | None -> str | list[str] | None (unchanged)
value_1 = tag.get(k)
# str | list[str] | None -> str | list[str] | None (unchanged)
value_2 = tag.get(k, default=None)
# str | list[str] | None -> str | list[str]
value_3 = tag.get(k, default="a")
# str | list[str] | None -> str | list[str]
value_4 = tag.get(k, default=["a"])
# type error on the default arg -> str | list[str] | int
value_5 = tag.get(k, default=1)
# type error on the default arg -> str | list[str] | list[int]
value_6 = tag.get(k, default=[1])

# list[str] -> list[str | None]
values_1 = tag.get_attribute_list(k)
# list[str] -> list[str | None]
values_2 = tag.get_attribute_list(k, default=None)
# list[str] -> list[str] (unchanged)
values_3 = tag.get_attribute_list(k, default="a")
# list[str] -> list[str] (unchanged)
values_4 = tag.get_attribute_list(k, default=["a"])
# type error on the default arg -> list[str | int]
values_5 = tag.get_attribute_list(k, default=1)
# type error on the default arg -> list[str | int]
values_6 = tag.get_attribute_list(k, default=[1])

[kemus]

If you want, I could also change it so we still only allow default to be str or list[str], I'm not necessarily attached to allowing any default value since I can't think of any cases where that's useful.

[kemus]

Thanks for the suggestions @srittau -- I should've seen that I could combine those.

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉