gh-91533: Move much of typing.rst to a new 'Guide to typing features' document

[AlexWaygood]

The first step towards #91533.

Everything from the beginning of the "Type aliases" section to the end of the "Nominal vs structural subtyping" section has been cut and pasted from typing.rst into the new document. To make this PR easy to review, no other changes have been made to those sections (they've just been moved from one document to the other).

The consensus in #91533 was that we'd prefer to have HOWTOs hosted at https://typing.readthedocs.io/en/latest/ rather than at docs.python.org. However, we've now had over a year with next to no progress made on that front. I personally have found the idea of starting work on https://typing.readthedocs.io/en/latest/ pretty overwhelming, as you'd be starting basically from scratch -- there's very little documentation there at all currently. Working within the CPython repo, I can work incrementally, which makes it far more likely that improvements will be made. And I think we probably all agree that making improvements to the documentation is a more important thing to focus on than arguing about where exactly specific bits of the documentation should be hosted.

• Issue: Proposal: Convert the typing-module PEPs into a series of HOWTOs #91533

---

📚 Documentation preview 📚: https://cpython-previews--105827.org.readthedocs.build/

[JelleZijlstra]

My main worry with this approach is that it means the docs remain tied to the Python version: we'd add this only to the 3.11+ docs, while most people don't use 3.11+ yet. I think it's generally more helpful to have one documentation that covers the type system across all Python versions, which is what typing.readthedocs.io could provide.

[AlexWaygood]

My main worry with this approach is that it means the docs remain tied to the Python version: we'd add this only to the 3.11+ docs, while most people don't use 3.11+ yet.

I disagree that this is true in a meaningful way. I might be wrong about this, but my gut feeling is that most users looking up the typing docs will just go straight to https://docs.python.org/3/library/typing.html, whether they're using Python 3.6 or Python 3.11. https://docs.python.org/3/library/typing.html currently defaults to the Python 3.11 documentation (and will do until Python 3.12 is released), so most users should be able to benefit from the latest user guides as soon as they're added to the 3.11 docs. It's true that we'll want to advertise how to use the latest features in this new document, but I think it should be possible to clearly signal which features are new in which Python versions, and how you can do things if you're still using an older Python version (or want to be backwards-compatible with an older Python version).

I think it's generally more helpful to have one documentation that covers the type system across all Python versions, which is what typing.readthedocs.io could provide.

It's true that it could provide it, but the fact of the matter is that it isn't providing it now, and won't do any time soon, I don't think. #91533 has been sitting open for over a year, and I've tried to motivate myself to start adding documentation to typing.readthedocs.io, but have failed. Nobody else has stepped up either. Speaking personally as to why I've found it difficult:

• The task is too enormous. We already have lots of documentation at docs.python.org, but very little at typing.readthedocs.io.
• The task feels much harder to do in an incremental way. I can only link to stuff at typing.readthedocs.io from docs.python.org if it already exists at typing.readthedocs.io. That means that if I want to move information from docs.python.org to typing.readthedocs.io, I have to make two PRs: one adding the documentation at typing.readthedocs.io (where it will duplicate information that already exists at docs.python.org), and then a second PR removing the information from docs.python.org and linking to typing.readthedocs.io. The two PRs will be potentially reviewed by different sets of people, and it will be hard to see why I'm making the first change without doing the second change at the same time. Working within the CPython repository, I can make the two changes at the same time, which feels much more like how it should be done.

In general, I'm also not sure why we should take a different strategy for typing than we do with any other CPython module. enum has seen rapid feature development in recent Python versions, and logging has also had new features added in recent Python versions, but both of those have HOWTOs in the CPython docs, and it seems to work fine?

I'm not wedded to this being the place that this documentation lives forever -- if we get to a point where we feel like we have a relatively complete corpus of documentation in docs.python.org and still want to move it across to typing.readthedocs.io at that point, I'm open to it. But I really feel that doing it this way is going to make it easier for me and others to iteratively improve the documentation at this point in time.

[hauntsaninja]

no progress made on that front [...] I've tried to motivate myself to start adding documentation to typing.readthedocs.io, but have failed. Nobody else has stepped up either [...] The task is too enormous. We already have lots of documentation at docs.python.org, but very little at typing.readthedocs.io.

For what it's worth, I thought there was rough consensus on importing mypy's docs into typing.readthedocs.io, so with that in mind I've made several dozen improvements to mypy's docs in the last year. There's maybe 4x as much documentation in mypy's docs as there is in typing.rst.

If that is indeed the case, I can make a PR that imports them into typing.readthedocs.io. (I didn't do this in April last year because I wanted to make some improvements first and nothing really nudged me to do it since... but I forgot this weekend is a long weekend, so I have some time :-) )

I do think the Python version argument holds some merit, especially in terms of what the focus should be when documenting "legacy" ways to do things (e.g. how much should the Python 3.12 docs caveat how you should define generics?). I also feel like there's more freedom at typing.readthedocs.io.... I think there's a limit to how accessible a tutorial can be if it doesn't cover questions new users have like "here's how to install pyright" or "consider passing --strict to mypy" and I think there'd be hesitance to put this kind of thing straight into CPython's documentation (lmk if this assumption is unfounded!)

[AlexWaygood]

For what it's worth, I thought there was rough consensus on importing mypy's docs into typing.readthedocs.io

There was consensus, but I changed my mind due to the apparent lack of progress (...and because I realised how much I disliked the idea of constantly hopping between repositories in order to make incremental improvements)

so with that in mind I've made several dozen improvements to mypy's docs in the last year. There's maybe 4x as much documentation in mypy's docs as there is in typing.rst.

I've been following that work, and think it's been hugely valuable! I fully agree that the mypy docs are the single most complete corpus of typing documentation at the moment.

I stand by what I said above, though — from my perspective, it didn't appear as though there had been any progress in actually importing those docs from the mypy docs to typing.readthedocs.io. If you're planning on doing that in the next few days, I'm happy to hold off here and assess the lay of the land afterwards :)

I do think the Python version argument holds some merit, especially in terms of what the focus should be when documenting "legacy" ways to do things (e.g. how much should the Python 3.12 docs caveat how you should define generics?).

I personally thought Jelle did a really great job with this when he rewrote the typing docs following PEP-695 — the PEP-695 examples come first in typing.rst, but there's still quite a bit of space dedicated towards the backwards-compatible ways of defining generics and type aliases

I think there's a limit to how accessible a tutorial can be if it doesn't cover questions new users have like "here's how to install pyright" or "consider passing --strict to mypy" and I think there'd be hesitance to put this kind of thing straight into CPython's documentation (lmk if this assumption is unfounded!)

Maybe I'm focusing too much on specifics here, but would we really want to include detailed typechecker-specific installation instructions and configuration options at typing.readhedocs.io? I thought the whole point of moving the docs out of the mypy repo was so that we could have typechecker-agnostic documentation. Wouldn't the better strategy be to point users to the docs for their typechecker of choice when it comes to installation instructions and configuration options? It feels like that would be a strategy that would be much less likely to go out of date

[JelleZijlstra]

A few responses:

I disagree that this is true in a meaningful way. I might be wrong about this, but my gut feeling is that most users looking up the typing docs will just go straight to https://docs.python.org/3/library/typing.html, whether they're using Python 3.6 or Python 3.11. https://docs.python.org/3/library/typing.html currently defaults to the Python 3.11 documentation (and will do until Python 3.12 is released), so most users should be able to benefit from the latest user guides as soon as they're added to the 3.11 docs.

At work we use 3.9, and in my experience my coworkers absolutely look at the 3.9 docs (and therefore sometimes think they can't use e.g. Self).

It's true that it could provide it, but the fact of the matter is that it isn't providing it now, and won't do any time soon, I don't think. #91533 has been sitting open for over a year, and I've tried to motivate myself to start adding documentation to typing.readthedocs.io, but have failed.

That's absolutely true (or it was when you wrote it, since Shantanu has added some material to typing.rtd since).

• The task feels much harder to do in an incremental way. I can only link to stuff at typing.readthedocs.io from docs.python.org if it already exists at typing.readthedocs.io. That means that if I want to move information from docs.python.org to typing.readthedocs.io, I have to make two PRs: one adding the documentation at typing.readthedocs.io (where it will duplicate information that already exists at docs.python.org), and then a second PR removing the information from docs.python.org and linking to typing.readthedocs.io. The two PRs will be potentially reviewed by different sets of people, and it will be hard to see why I'm making the first change without doing the second change at the same time. Working within the CPython repository, I can make the two changes at the same time, which feels much more like how it should be done.

They're pretty likely to be the same set of people :)

And I don't think it necessarily has to be a move. Adding things to typing.rtd doesn't require removing near-equivalent material from CPython immediately.

In general, I'm also not sure why we should take a different strategy for typing than we do with any other CPython module. enum has seen rapid feature development in recent Python versions, and logging has also had new features added in recent Python versions, but both of those have HOWTOs in the CPython docs, and it seems to work fine?

The main difference is that typing has a well-established mechanism (typing_extensions) that means that for many (though not all) features, the CPython release cycle shouldn't really matter to end users. That's not true for enum and logging.

[AlexWaygood]

It's clear that the consensus is against me here, so I won't push this any further :)

There's still a bunch of material I'd like to add to the CPython typing docs, which I don't feel would fit in well to the corpus of documentation at typing.readthedocs.io. But we can cross those bridges when we come to them.

It was never my intention to belittle anybody else's work in this space. I think it's clear that I did so nonetheless. I'm sorry for that, and will try to do better in the future.

[JelleZijlstra]

It was never my intention to belittle anybody else's work in this space. I think it's clear that I did so nonetheless. I'm sorry for that, and will try to do better in the future.

I don't think you did! All I see is a honest and civil disagreement on the right approach to use.

[hauntsaninja]

I don't think you did either!

From my perspective: the task was big, mypy docs were a reasonable consensus starting point, but I / we dropped the ball on actually copying them over. This PR got that ball rolling again, so big success! Also note I'm genuinely very happy for us to rewrite / remove any of the mypy docs we copy over — they're a good starting point for a large chunk of a large task, but by no means are perfect.