tests failing on master (macOS Monterey / py 3.7)

[huguesb]

Bug Report

Tests are failing for the master branch on my macOS laptop.

A few tests in PEP561Suite are flaky for reasons I don't quite understand.

To Reproduce

git pull
git checkout f501cf649d7976077a7196e3548d773d67340a8c
python3.7 -m venv .venv
.venv/bin/pip install -r test-requirements.txt
.venv/bin/pip install -e .
PATH=$(pwd)/.venv/bin:$PATH .venv/bin/pytest -q -k 'PEP561Suite'

Expected Behavior

Tests pass.

Actual Behavior

There's usually exactly one failure per run. On some rare occasions I've seen more than one failure, or no failures.

Here are some sample failures:

_____________________________________________________________________________________________________________ testStubPrecedence _____________________________________________________________________________________________________________
[gw3] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:66:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:152: in test_pep561
    testcase.file, testcase.line, iter_count))
E   AssertionError: Invalid output (/System/Volumes/Data/repos/mypy/test-data/unit/pep561.test, line 66)
------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------
Expected:
  testStubPrecedence.py:5: note: Revealed type is "builtins.list[builtins.str]" (diff)
Actual:
  testStubPrecedence.py:3: error: Module "typedpkg" has no attribute "dne" (diff)
  testStubPrecedence.py:5: note: Revealed type is "builtins.list[builtins.str]" (diff)

Alignment of first line difference:
  E: testStubPrecedence.py:5: note: Revealed type is "builtins.list[builtins....
  A: testStubPrecedence.py:3: error: Module "typedpkg" has no attribute "dne"...
                           ^

______________________________________________________________________________________________ testNamespacePkgWStubsWithNamespacePackagesFlag _______________________________________________________________________________________________
[gw2] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:204:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:152: in test_pep561
    testcase.file, testcase.line, iter_count))
E   AssertionError: Invalid output (/System/Volumes/Data/repos/mypy/test-data/unit/pep561.test, line 204)
------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------
Expected:
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:7: error: Argument 1 to "bf" has incompatible type "int"; expected "bool" (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:8: error: Argument 1 to "bf" has incompatible type "int"; expected "bool" (diff)
Actual:
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:3: error: Skipping analyzing "typedpkg_ns.a.bbb": module is installed, but missing library stubs or py.typed marker (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:3: error: Skipping analyzing "typedpkg_ns.a": module is installed, but missing library stubs or py.typed marker (diff)
  testNamespacePkgWStubsWithNamespacePackagesFlag.py:8: error: Argument 1 to "bf" has incompatible type "int"; expected "bool" (diff)

Alignment of first line difference:
  E: ...spacePackagesFlag.py:7: error: Argument 1 to "bf" has incompatible ty...
  A: ...spacePackagesFlag.py:3: error: Skipping analyzing "typedpkg_ns.a.bbb"...
                             ^

_______________________________________________________________________________________________________ testTypedPkgNamespaceRegImport _______________________________________________________________________________________________________
[gw2] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:150:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:152: in test_pep561
    testcase.file, testcase.line, iter_count))
E   AssertionError: Invalid output (/System/Volumes/Data/repos/mypy/test-data/unit/pep561.test, line 150)
------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------
Expected:
  testTypedPkgNamespaceRegImport.py:4: error: Cannot find implementation or library stub for module named "typedpkg_ns.a.dne" (diff)
  testTypedPkgNamespaceRegImport.py:4: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
  testTypedPkgNamespaceRegImport.py:10: error: Argument 1 has incompatible type "bool"; expected "str" (diff)
  testTypedPkgNamespaceRegImport.py:11: error: Argument 1 has incompatible type "int"; expected "bool" (diff)
Actual:
  testTypedPkgNamespaceRegImport.py:3: error: Cannot find implementation or library stub for module named "typedpkg_ns.a.bbb" (diff)
  testTypedPkgNamespaceRegImport.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
  testTypedPkgNamespaceRegImport.py:3: error: Cannot find implementation or library stub for module named "typedpkg_ns" (diff)
  testTypedPkgNamespaceRegImport.py:3: error: Cannot find implementation or library stub for module named "typedpkg_ns.a" (diff)
  testTypedPkgNamespaceRegImport.py:4: error: Cannot find implementation or library stub for module named "typedpkg_ns.a.dne" (diff)
  testTypedPkgNamespaceRegImport.py:10: error: Argument 1 has incompatible type "bool"; expected "str" (diff)

Alignment of first line difference:
  E: ...kgNamespaceRegImport.py:4: error: Cannot find implementation or libra...
  A: ...kgNamespaceRegImport.py:3: error: Cannot find implementation or libra...
                                ^

_______________________________________________________________________________________________________ testTypedPkgNamespaceRegImport _______________________________________________________________________________________________________
[gw0] darwin -- Python 3.7.13 /System/Volumes/Data/repos/mypy/.venv/bin/python3.7
data: /System/Volumes/Data/repos/mypy/test-data/unit/pep561.test:150:
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:30: in run_case
    test_pep561(test_case)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:113: in test_pep561
    install_package(pkg, python_executable, use_pip, editable)
/System/Volumes/Data/repos/mypy/mypy/test/testpep561.py:86: in install_package
    raise Exception(proc.stdout.decode('utf-8') + proc.stderr.decode('utf-8'))
E   Exception: Looking in indexes: https://registry.affirm-stage.com/artifactory/api/pypi/pypi/simple
E   Processing /System/Volumes/Data/repos/mypy/test-data/packages/typedpkg_ns_a
E     Preparing metadata (setup.py): started
E     Preparing metadata (setup.py): finished with status 'done'
E   Building wheels for collected packages: typedpkg-namespace.alpha
E     Building wheel for typedpkg-namespace.alpha (setup.py): started
E     Building wheel for typedpkg-namespace.alpha (setup.py): finished with status 'done'
E     Created wheel for typedpkg-namespace.alpha: filename=typedpkg_namespace.alpha-1.0.0-py3-none-any.whl size=1588 sha256=21e162c192f4c7f108ec1b970a54d677de03e021e7143da28589e81f308a7505
E     Stored in directory: /private/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/pip-ephem-wheel-cache-niwbfkch/wheels/85/1b/b9/19a1923b949574d3f92d0b2fd98b06cfe4eb7aa0b819bbbcf7
E   ERROR: Exception:
E   Traceback (most recent call last):
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2681, in version
E       return self._version
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
E       raise AttributeError(attr)
E   AttributeError: _version
E
E   During handling of the above exception, another exception occurred:
E
E   Traceback (most recent call last):
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
E       status = run_func(*args)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
E       return func(self, options, args)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 366, in run
E       global_options=[],
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/wheel_builder.py", line 354, in build
E       req.editable and req.permit_editable_wheels,
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/wheel_builder.py", line 227, in _build_one
E       _verify_one(req, wheel_path)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/wheel_builder.py", line 175, in _verify_one
E       dist_verstr = str(dist.version)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_internal/metadata/pkg_resources.py", line 147, in version
E       return parse_version(self._dist.version)
E     File "/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/tmptvamo6c8/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2689, in version
E       raise ValueError(msg, self)
E   ValueError: ("Missing 'Version:' header and/or METADATA file at path: [could not detect]", typedpkg-namespace-alpha [unknown version] (/private/var/folders/kd/x5k73mjs4jn02v7k1c0spz_40000gn/T/pip-ephem-wheel-cache-niwbfkch/wheels/85/1b/b9/19a1923b949574d3f92d0b2fd98b06cfe4eb7aa0b819bbbcf7/typedpkg_namespace.alpha-1.0.0-py3-none-any.whl))

Your Environment

mypy commit f501cf649d7976077a7196e3548d773d67340a8c
Python 3.7.13
macOS Monterey 12.3.1
x86_64

[huguesb]

NB: initial report included a mypyc build failure, which turned out to be caused by using python 3.7.8 built via pyenv prior to the upgrade to Monterey. Using a freshly build 3.7.13 solved that. The flakiness persists however...

[erikkemperman]

For what it's worth, I am seeing the exact same thing on Python 3.10, also on Mac Monterey (12.3.1), not sure if it's relevant but I have the Apple M1 cpu, and so I compile my Pythons for arm64.

[JukkaL]

I suspect that this is caused by some shared file system state, and only reproduces when running tests in parallel. All the tests passed when I ran them sequentially (on macOS). Also CI has been reliable, and it doesn't use much/any parallelism.

[erikkemperman]

I saw this on Linux too, on Python 3.10.4. Confirm that things work reliably and consistently with pytest -q -n0 -k PEP561Suite. However, as far as I can see from a casual glance, it seems that each case of the PEP 561 suite runs within its own venv (and associated unique tempdir) so I don't understand exactly how parallel mode gives rise to FS race conditions here.

[huguesb]

I saw this on Linux too, on Python 3.10.4. Confirm that things work reliably and consistently with pytest -q -n0 -k PEP561Suite. However, as far as I can see from a casual glance, it seems that each case of the PEP 561 suite runs within its own venv (and associated unique tempdir) so I don't understand exactly how parallel mode gives rise to FS race conditions here.

pip would be my first guess based on the stack trace I posted in the original bug report... It's notoriously prone to breakage if invoked concurrently, even on separate venvs, because it may still use shared global state, for instance when downloading packages or building wheels.

[erikkemperman]

I got this suite to pass, reliably, in two ways -- neither of them very pretty. One is to just pass -n0 to this particular suite and avoid race conditions by virtue of not having any concurrency. The other involves using a filesystem-based lock around the installation steps. This would rely on a new entry in test-requirements -- but , for what it's worth, it's one that folks will have already if they use tox.

Some completely informal timings:
pytest -q -n0 -k PEP561Suite took 79s
pytest -q -n4 -k PEP561Suite with filelock took 48s
pytest -q -n8 -k PEP561Suite with filelock took 47s

For reference, the current state of master branch does them rather more quickly -- but requires a bunch of attempts to pass all cases, in particular with more workers:

pytest -q -n4 -k PEP561Suite took 26s (passed the 3rd attempt)
pytest -q -n8 -k PEP561Suite took 19s (passed the 5th attempt)

I'm not sure if this actually affects a lot of people; it might be just something particular to my environment (although I did see it on both Mac and Linux recently). But if anyone thinks that the added running time is worth it, here's what I've done:

sequential: erikkemperman@b68aac5

filelock: erikkemperman@3648f5c

[emmatyping]

One thing I was hoping would work, but seems not to, is passing --cache-dir. It seems pip does not work concurrently even with independent cache directories...

I think a lock on the tests is unfortunately the only way forward then.

[emmatyping]

By the way, the pip issue for this is pypa/pip#2361

[erikkemperman]

Yes, I tried setting —cache-dir as well, and —no-cache, but no luck.

[emmatyping]

@erikkemperman well if you want to make a PR I'd be happy to review it, I think the tests running a bit slower rather than failing is an acceptable tradeoff.

[erikkemperman]

@ethans Done: #12857

[ethans]

Excellent @erikkemperman .
@ethanhs - FYI

[erikkemperman]

Excellent @erikkemperman . @ethanhs - FYI

Ah my apologies for misspelling the name!

[ethans]

Excellent @erikkemperman . @ethanhs - FYI

Ah my apologies for misspelling the name!

My pleasure to take part in great doing!