Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP561: Clarify stub-only namespace package behavior #2083

Merged
merged 5 commits into from
Oct 6, 2021
Merged

PEP561: Clarify stub-only namespace package behavior #2083

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
79f69c5
Clarify stub-only namespace package behavior in PEP561
nipunn1313 Sep 20, 2021
538d8ee
Clarify single file modules inside namespace packages
nipunn1313 Sep 20, 2021
9267be7
Apply suggestions from code review
nipunn1313 Sep 26, 2021
7141d80
Add note to changelog about handling of single file modules
nipunn1313 Sep 26, 2021
b3accc1
Clarify partial packages for namespace packages
nipunn1313 Oct 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 45 additions & 4 deletions pep-0561.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,10 @@ For namespace packages (see PEP 420), the ``py.typed`` file should be in the
submodules of the namespace, to avoid conflicts and for clarity.

This PEP does not support distributing typing information as part of
module-only distributions. The code should be refactored into a package-based
distribution and indicate that the package supports typing as described
module-only distributions or single-file modules within namespace packages.

The single-file module should be refactored into a package
and indicate that the package supports typing as described
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This clarification was inspired by googleapis/python-firestore#447 - around the module google/cloud/firestore.py which exists within a package distribution in a way that can't be py.typed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think is quite right. In the above issue, the py.typed file belongs inside the google/cloud folder, as google is the namespace package (unless I am mistaken), so the py.typed file lives in each sub-package of the namespace package.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

google.cloud is also a namespace package (sorry for confusion)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a nested namespace package

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you should still list the sub-package of the outermost package as typed (even if partial).

Copy link
Contributor Author

@nipunn1313 nipunn1313 Sep 26, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to make sure we're on the same page - I don't think the preexisting PEP unambiguously specifies this case (single-file modules in possibly nested namespace packages) - we are discussing/reasoning about what makes the most sense to make as the specification here.

The sub-namespace-package (google.cloud in this case) will be populated by several different pip distributions (https://pypi.org/search/?q=google-cloud-). I don't think it works - or makes sense - for each of those pip distributions to place a py.typed partial in google.cloud.

As a concrete example - for something like google/cloud/firestore.py which is fully typed - It doesn't make sense to have a google/cloud/py.typed indicating full typed-ness - since google/cloud could have other entries from other distributions. Rather we must require it to be google/cloud/firestore/__init__.py with google/cloud/firestore/py.typed so that typecheckers can know that firestore is fully typed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that seems reasonable.

Copy link

@henribru henribru Oct 12, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nipunn1313 I ran into something related to this while working on typing for https://pypi.org/project/googleapis-common-protos/ (using the excellent mypy-protobuf, thanks for your work!) and was wondering if you could confirm if my understanding of the updated spec is correct.

Like most Google distributions this distribution contains a namespace google package, which contains multiple subpackages. Here's the tree showing one particular subpackage:

google
├── __init__.py
├── rpc
│   ├── code_pb2.py
│   ├── code.proto
│   ├── context
│   │   ├── attribute_context_pb2.py
│   │   ├── attribute_context.proto
│   │   └── __init__.py
│   ├── error_details_pb2.py
│   ├── error_details.proto
│   ├── __init__.py
│   ├── README.md
│   ├── status_pb2.py
│   └── status.proto

Here both google and rpc are namespace packages (using declare_namespace/extend_path), while context is a regular package. As you can see google.rpc actually contains some modules. Presumably some other Google distribution could put other modules in this namespace. If I understand this discussion correctly, there's no way to distribute typing information for code_pb2.py, error_details_pb2.py and status_pb2.py and for this to work Google would have to refactor these into packages? Is the correct approach when making a stub-only package for this to simply leave them out, like this?

google-stubs
├── rpc
│   └── context
│       ├── attribute_context_pb2.pyi
│       ├── __init__.pyi
|       └── py.typed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually wrote this spec modification with a case in mind - single-file-modules where types are included in the original package (googleapis-common-protos in your eg) - because in that case - a py.typed is required.

For a stub-only package - By the spec clarification - all namespace packages within stub-packages are to be considered partially typed by type checkers.

Type checkers should treat namespace packages within stub-packages as
incomplete since multiple distributions may populate them.
Regular packages within namespace packages in stub-package distributions
are considered complete unless a ``py.typed`` with ``partial\n`` is included.

Thus I think you can opt to do

google-stubs
├── rpc
│   └── context
│       ├── attribute_context_pb2.pyi
│       ├── __init__.pyi
|       └── py.typed

You may also add stubs for the remaining files (eg status_pb2.pyi), but be sure to omit the __init__.pyi for namespace packages. Typechecker should consider google-stubs and google-stubs/rpc as incomplete because they are namespace packages.

I haven't tested whether mypy/pyright obey this spec clarification yet - but I believe pyright recently made a change to make this work. I believe in my past testing mypy needed the --namespace-packages flag to treat these properly, but I didn't test this specific case.

Copy link

@henribru henribru Oct 12, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the very quick clarification! That's my bad for not properly distinguishing between typed "real" packages and stub-only packages. I guess the restriction on single-file modules is Google's problem if they ever decide to type this in the package itself. I'll just do this for my stub-only package then:

google-stubs
├── rpc
│   ├── code_pb2.pyi
│   ├── context
│   │   ├── attribute_context_pb2.pyi
│   │   ├── __init__.pyi
│   │   └── py.typed
│   ├── error_details_pb2.pyi
│   └── status_pb2.pyi

above.

Stub-only Packages
Expand Down Expand Up @@ -156,6 +158,28 @@ in pip 9.0, if you update ``flyingcircus-stubs``, it will update
``--upgrade-strategy=only-if-needed`` flag. In pip 10.0 this is the default
behavior.

For namespace packages (see PEP 420), stub-only packages should
use the ``-stubs`` suffix on only the root namespace package.
All stub-only namespace packages should omit ``__init__.pyi`` files. ``py.typed``
marker files are not necessary for stub-only packages, but similarly
to packages with inline types, if used, they should be in submodules of the namespace to
avoid conflicts and for clarity.

For example, if the ``pentagon`` and ``hexagon`` are separate distributions
installing within the namespace package ``shapes.polygons``
The corresponding types-only distributions should produce packages
laid out as follows::

shapes-stubs
└── polygons
└── pentagon
└── __init__.pyi

shapes-stubs
└── polygons
└── hexagon
   └── __init__.pyi


Type Checker Module Resolution Order
------------------------------------
Expand All @@ -180,6 +204,11 @@ resolve modules containing type information:
5. Typeshed (if used) - Provides the stdlib types and several third party
libraries.

If typecheckers identify a stub-only namespace package without the desired module
in step 3, they should continue to step 4/5. Typecheckers should identify namespace packages
by the absence of ``__init__.pyi``. This allows different subpackages to
independently opt for inline vs stub-only.

Type checkers that check a different Python version than the version they run
on MUST find the type information in the ``site-packages``/``dist-packages``
of that Python version. This can be queried e.g.
Expand All @@ -204,8 +233,15 @@ typeshed folder and type checking the combined directory structure. Thus type
checkers MUST maintain the normal resolution order of checking ``*.pyi`` before
``*.py`` files.

If a stub package is partial it MUST include ``partial\n`` in a top level
``py.typed`` file.
If a stub package distribution is partial it MUST include ``partial\n`` in a
``py.typed`` file. For stub-packages distributing within a namespace
package (see PEP 420), the ``py.typed`` file should be in the
submodules of the namespace.

Type checkers should treat namespace packages within stub-packages as
incomplete since multiple distributions may populate them.
Regular packages within namespace packages in stub-package distributions
are considered complete unless a ``py.typed`` with ``partial\n`` is included.


Implementation
Expand Down Expand Up @@ -236,6 +272,11 @@ Vlasovskikh, Nathaniel Smith, and Guido van Rossum.
Version History
===============

* 2021-09-20

* Clarify expectations and typechecker behavior for stub-only namespace packages
* Clarify handling of single-file modules within namespace packages.

* 2018-07-09

* Add links to sample stub-only packages
Expand Down