Testing third-party stubs in isolated environments

[srittau]

At the moment, whenever there is change to typeshed, we test all stdlib and third-party packages in the same testing environment, using the --custom-typeshed-dir so that all packages can see each other. There are several practical problems with that:

• We can't test the requires fields for correctness; all packages are always available.
• We can't have packages that depend on external packages, see for example Allow non-types dependencies #5768, Remove cryptography (depends on #5952) #5618, paramiko: Change dependencies on types-cryptography, allow using upstream type hints #5847, and the discussion in Allow non-types dependencies #5769 (all related to cryptography). (Except if we'd just install all non-types dependencies for all package into our venv.)
• It doesn't scale if we increase the number of third-party packages in typeshed.
• Ideas like per-package configuration is not possible at the moment, see for example mypy tests: Allow per distribution strictness options #1526.

My suggestion: Only run the tests for the third-party stubs that have actually changed, and run each test in a separate environment. This environment only has the requirements from METADATA.toml installed. This fixes the problems above and also means:

• The stubs will not be able to use features from stdlib that are not yet in the latest type checker distributions.
• Changes that depend on changes in another third-party stub need to wait until that change has been released. (Happens every three hours, so it shouldn't be too much of a problem.)
• Running the tests for all stubs will be slower, since each package needs its own venv installation.

[Akuli]

A possible downside: If the tests of a third-party stub break for some reason, e.g. because a new version of the corresponding non-stub package is released, the problem will remain unnoticed until someone makes a PR for that specific stub package. I remember most pull requests showing red CI because of some Pillow thing, but can't find it from the PR history now.

[srittau]

It would be useful to run full tests at a regular interval, say once per day or week.

[hauntsaninja]

Note the stubtest third party code already does this venv creation. We could steal that code / maybe use some fancy Github Actions CI caching to cache venvs across workflows. Switching to testing only changed third party stubs is probably a better idea than just doing everything and sharding, which is the approach test_stubtest currently takes.

Like you say, I'd want to think through how we manage different stdlib and types-* versions; to that goal, here are some previous issues we've had on those lines: #4815 #5786 #5751

[Akuli]

This seems to be done now.

[srittau]

No, we still need to use separate venvs for each distribution, so that the dependencies don't interfere with each other.

[hmc-cs-mdrissi]

I would be interested in working on this mainly to make it possible to have non-types dependencies.

Would reasonable steps in order be,

1. Adjust each check to only run for changed folders. Start with mypy then continue to pyright then pytype.
2. Create a new venv for each folder checked. Again mypy -> pyright -> pytype
3. Add support for reading metadata.toml and install dependencies for given folder.
4. Add support for per package mypy.ini/pyrightconfig.json/etc files.

Any major tasks I'm missing?

[hauntsaninja]

That sounds reasonable. Some thoughts:

• It might make sense to split each test up into running on the stdlib vs running on individual third party distributions (like we do with stubtest)
• Here's some code for venv creation:

  typeshed/tests/stubtest_third_party.py

  Line 39 in 823592e

  with tempfile.TemporaryDirectory() as tmp:

• We'd want to make sure that other stubs don't interfere, e.g. we should not be picking up cryptography-stubs if someone declares a cryptography dep. The various type checkers probably each handle this differently
• We talked about only having a small, vetted list of deps. If this is the case, I personally wouldn't hate getting started on Allow non-types dependencies #5768 with a global venv. This might be controversial though :-)
• It might be nice to use different keys in pyproject.toml for typeshed deps vs other deps (e.g., could make automation in the future easier / avoid mistakes where we assume that "types-*" packages are trusted). Might need a change in https://github.com/typeshed-internal/stub_uploader
• There's an @tests folder you can reuse for per-distribution config files. We even store some extra requirements.txt's in there

[srittau]

@hmc-cs-mdrissi For now you could try to skip step 1 and we can see how that affects CI runtimes.

[AlexWaygood]

As of #9408, third-party stubs with non-types dependencies are now tested with mypy in isolated venvs. The venvs are setup concurrently using a threadpool, meaning we still test all typeshed stubs packages in every run, but the script remains performant.

(Stubs packages with no non-types dependencies are not tested in a separate venv, but they are tested using the --no-site-packages flag when they are tested, so are still run in an isolated environment.)

Closing as completed! 🥳