override http.HTTPMessage methods to only use str for the header type

[Avasam]

Assuming HTTPMessage should only work in str headers, which I'm not certain about. But either way would close #2333 .

Signature of the overridden methods based on #11095 .
An alternative to overriding is to make email.message.Message generic. Which would cause close to 100 changes in stdlib. Unless we defer until PEP 696 support.

[srittau]

I think the PR looks good overall. Limiting HTTPMessage to just str seems ok, especially given the primer output.

[srittau]

str | None should be correct here. A header is not guaranteed to exist, so not being prepared for a None value is a strong indication of a bug.

[Avasam]

Since it's a copy, the reasoning is the same as in https://github.com/python/typeshed/pull/11095/files#diff-ced02b01b42b244e9b7bc2e12ef7a215bde89a29072c99ade903f58fb83a3bc7R66-R67

Same as get with failobj=None, but with the expectation that it won't return None in most scenarios
This is important for protocols using __getitem__, like SupportsKeysAndGetItem

And #11095 (comment)

[...] I don't think the false-positives a None union here would introduce is something we want. Especially when .get exists for that purpose.

[Avasam]

That being said, the Any trick could be usable here (and in the referenced PR). I should try a primer run with it.

[srittau]

I'm not sure I agree with that reasoning. While get() exists, __getattr__ returns it as well. This is arguably a misfeature in the implementation (as it should just raise), but still that's something that a user should check against, because it absolutely can return None, even in unexpected scenarios.

[Avasam]

#11095 (comment) At the very least using the Any trick should avoid the bad false-positive with __getitem__ based protocols like SupportsKeysAndGetItem.
Ref #11094

Suggested change

	def __getitem__(self, name: str) -> str: ... # AnyOf[str, None]
	def __getitem__(self, name: str) -> str | Any: ... # See comment on `Message.__getitem__`

[srittau]

I'm still not happy about this. This should be an error in my opinion:

def parse_headers(headers: SupportsGetItem[str]):
    header_value = headers["Content-Type"]
    full_header = f"Content-Type: {header_value}"

parse_headers(HTTPMessage(...))

Users of HTTPMessage (and email.message.Message) must be equipped to get None values from __getitem__.

[srittau]

Ideally, we'd be able to have an overload like this, but I don't think it's possible:

@overload
def get_all(self, name: str, failobj: list[<empty>]) -> list[str]: ...

[Avasam]

I'm not sure what such an overload would be trying to do. Consumer code is the one passing the failobj and immediately receiving the return type. get_all does nothing with failobj.
If it only supports a list[_HeaderType] it would be its responsibility to pass a value of the appropriate type to failobj, which a typechecker checks for.

def foo(message: HttpMessage) -> list[str] | SentinelType:
	return message.get_all("foo", sentinel_value)  # Ok
	
def foo(message: HttpMessage) -> list[str]:
	return message.get_all("foo", [])  # Fails pyright `reportUnknownArgumentType` but by no fault of the definition
	
def foo(message: HttpMessage) -> list[str]:
	return message.get_all("foo", list[str]())  # ok
	
def foo(message: HttpMessage) -> list[str]:
	return message.get_all("foo", sentinel_value)  # Fails type-checking

If we still really wanna restrict it, we can bind _T to list[_HeaderType]

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

spack (https://github.com/spack/spack)
+ lib/spack/spack/oci/oci.py:142: error: Argument 1 to "endpoint" of "ImageReference" has incompatible type "Optional[str]"; expected "str"  [arg-type]