Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_typeshed.Self doesn't work in third-party packages with mypy yet #5751

Closed
JelleZijlstra opened this issue Jul 9, 2021 · 7 comments
Closed

Comments

@JelleZijlstra
Copy link
Member

I noticed while investigating #5750 that I got Any from a method associated as returning _typeshed.Self. I tracked down that it is because to mypy 0.910, _typeshed.Self doesn't exist yet because it's bundling an older version of the stdlib. The problem went away when I manually added Self to the installed stub file for _typeshed.

I see two options:

  • Ignore the problem and wait for the next mypy release to update typeshed. Until then, people will just see Any types from anything we annotate with Self in third-party packages.
  • Avoid using Self in third-party packages for now.
@srittau
Copy link
Collaborator

srittau commented Jul 9, 2021

I'd say we wait for the next mypy release. We should probably also force mypy to use its bundled typeshed when checking third-party libs instead of typeshed's stdlib from HEAD.

@JukkaL
Copy link
Contributor

JukkaL commented Jul 10, 2021

More generally, if we start using new features in third-party library stubs, older mypy releases will silently start seeing Any types once users upgrade to most recent stubs (or they may get errors). This affects all tools that don't bundle the entire typeshed.

I think that this is a fairly bad -- silently losing type checking precision is a real problem. Here are some ideas about how we could perhaps make this better:

  1. Publish a schedule for making changes to _typeshed and the adoption of new typing features in third-party library stubs. Announce new features well ahead of time to give tool maintainers time to prepare for them.
  2. Only start using new features in typeshed 6 or 12 months after popular tools support them, to give users time to upgrade (and time for the new features to become stable).
  3. Recommend that users should pin all stub packages if they also pin the type checker version, or otherwise they risk problems with newer releases of stubs that use recent typing features unsupported by the tool.
  4. Synchronize features that can be used in stubs to Python releases. Mark the minimum required "typing level" in stub metadata. For example, if a stub uses Python 3.10 typing features, and a tool doesn't support 3.10 fully yet, the tool can generate an error about the installed stubs and suggest installing an earlier version of the type package.

This doesn't affect stdlib stubs, since the assumption is that tools bundle them (in part because of this versioning problem).

@JukkaL
Copy link
Contributor

JukkaL commented Jul 10, 2021

Yet another idea that could work for some tools is to publish point releases with updated _typeshed and backported support for new features used in stubs. This is clearly not always feasible, since backporting complex features may be too difficult and can break semver versioning schemes. For example, I'm thinking of releasing mypy 0.911 with some fixes to built-in generics and new union type syntax so that stubs that use these features can still be used, without having to upgrade to a new full feature release.

@hauntsaninja
Copy link
Collaborator

hauntsaninja commented Jul 10, 2021

also force mypy to use its bundled typeshed when checking third-party libs instead of typeshed's stdlib from HEAD

I'm pretty sure we used to do this and ran into some problematic false negatives as a result. I seem to recall at least something involving the protobuf stubs, maybe someone else remembers better.

@srittau
Copy link
Collaborator

srittau commented Jul 11, 2021

Asking users to pin specific types packages is a good idea anyway. Otherwise builds and CI can easily break, since type changes can be fairly intrusive. That said, we should not ship stubs that are incompatible with the latest released version of one of the major type checkers, which is something we can easily test automatically. But waiting an arbitrary amount of time after that is hard to manage with our limited maintainer resources and slows down typeshed's velocity. It would create quite a bit confusion about which features can be used and which can't.

@SnoopJ
Copy link

SnoopJ commented Oct 12, 2021

I agree with @JelleZijlstra that #6156's problem with _typeshed.IdentityFunction appears to be the same issue, dropping this note so the issue shows up if someone else searches the typeshed issues for that annotation.

@hauntsaninja
Copy link
Collaborator

This is a tricky problem, but I'm not sure there's anything too directly actionable for typeshed in this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants