Crash with pytest-subtests in Windows

[JakobDev]

Describe the bug
I'm trying to use pyfakefs together with pytest-subtests. It works without any problems in Linux, but when I tested it on Windows, it just crashes. I'm not sure if I should report it here or on the pytest-subtests, but error points to pyfakefs, so I report it here.

How To Reproduce
Execute this test in Windows:

import pytest_subtests
import pyfakefs


def test_subtest(subtests: pytest_subtests.SubTests, fs: pyfakefs.fake_filesystem.FakeFilesystem) -> None:
    fs.os = pyfakefs.fake_filesystem.OSType.LINUX

    with subtests.test("Test"):
        print("Hello")

Full Error

========================================================================================================================================================================= test session starts =========================================================================================================================================================================
platform win32 -- Python 3.11.1, pytest-7.4.3, pluggy-1.0.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: C:\Users\username\tmp
plugins: anyio-3.7.1, Faker-18.4.0, pyfakefs-5.3.1, asyncio-0.21.0, benchmark-4.0.0, cov-4.1.0, integration-0.2.3, mock-3.11.1, recording-0.12.2, subtests-0.11.0, xdist-3.3.1, requests-mock-1.11.0
asyncio: mode=Mode.STRICT
collected 1 item

test.py F                                                                                                                                                                                                                                                                                                                                                        [100%]

============================================================================================================================================================================== FAILURES ===============================================================================================================================================================================
____________________________________________________________________________________________________________________________________________________________________________ test_subtest _____________________________________________________________________________________________________________________________________________________________________________

self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x0000017B356D4790>, file_path = '/C:\\Users\\username\\AppData\\Local\\Temp/tmph6mkc_a2', check_read_perm = True, check_owner = True

    def get_object_from_normpath(
        self,
        file_path: AnyPath,
        check_read_perm: bool = True,
        check_owner: bool = False,
    ) -> AnyFile:
        """Search for the specified filesystem object within the fake
        filesystem.

        Args:
            file_path: Specifies target FakeFile object to retrieve, with a
                path that has already been normalized/resolved.
            check_read_perm: If True, raises OSError if a parent directory
                does not have read permission
            check_owner: If True, and check_read_perm is also True,
                only checks read permission if the current user id is
                different from the file object user id

        Returns:
            The FakeFile object corresponding to file_path.

        Raises:
            OSError: if the object is not found.
        """
        path = make_string_path(file_path)
        if path == matching_string(path, self.root.name):
            return self.root
        if path == matching_string(path, self.dev_null.name):
            return self.dev_null

        path = self._original_path(path)
        path_components = self._path_components(path)
        target = self.root
        try:
            for component in path_components:
                if S_ISLNK(target.st_mode):
                    if target.contents:
                        target = cast(FakeDirectory, self.resolve(target.contents))
                if not S_ISDIR(target.st_mode):
                    if not self.is_windows_fs:
                        self.raise_os_error(errno.ENOTDIR, path)
                    self.raise_os_error(errno.ENOENT, path)
>               target = target.get_entry(component)  # type: ignore

C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_filesystem.py:1633:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_file.FakeDirectory object at 0x0000017B357CF3D0>, pathname_name = 'C:\\Users\\username\\AppData\\Local\\Temp'

    def get_entry(self, pathname_name: str) -> AnyFile:
        """Retrieves the specified child file or directory entry.

        Args:
            pathname_name: The basename of the child object to retrieve.

        Returns:
            The fake file or directory object.

        Raises:
            KeyError: if no child exists by the specified name.
        """
        pathname_name = self._normalized_entryname(pathname_name)
>       return self.entries[to_string(pathname_name)]
E       KeyError: 'C:\\Users\\username\\AppData\\Local\\Temp'

C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_file.py:542: KeyError

During handling of the above exception, another exception occurred:

subtests = SubTests(ihook=<_pytest.config.compat.PathAwareHookProxy object at 0x0000017B306B1450>, suspend_capture_ctx=<bound met...='suspended' _in_suspended=False> _capture_fixture=None>>, request=<SubRequest 'subtests' for <Function test_subtest>>), fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x0000017B356D4790>

    def test_subtest(subtests: pytest_subtests.SubTests, fs: pyfakefs.fake_filesystem.FakeFilesystem) -> None:
        fs.os = pyfakefs.fake_filesystem.OSType.LINUX

>       with subtests.test("Test"):

C:\Users\username\tmp\test.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\contextlib.py:137: in __enter__
    return next(self.gen)
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytest_subtests.py:200: in test
    with self._capturing_output() as captured_output, self._capturing_logs() as captured_logs:
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\contextlib.py:137: in __enter__
    return next(self.gen)
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytest_subtests.py:169: in _capturing_output
    fixture._start()
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\_pytest\capture.py:913: in _start
    out=self.captureclass(1),
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\_pytest\capture.py:466: in __init__
    TemporaryFile(buffering=0),
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\tempfile.py:563: in NamedTemporaryFile
    file = _io.open(dir, mode, buffering=buffering,
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_io.py:124: in open
    return fake_open(
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_open.py:97: in __call__
    return self.call(*args, **kwargs)
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_open.py:158: in call
    file_ = opener(file_, self._open_flags_from_open_modes(open_modes))
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\tempfile.py:560: in opener
    fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\tempfile.py:256: in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_os.py:1353: in wrapped
    return f(*args, **kwargs)
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_os.py:295: in open
    self.chmod(path, mode)
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_os.py:1353: in wrapped
    return f(*args, **kwargs)
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_os.py:1017: in chmod
    self.filesystem.chmod(path, mode, follow_symlinks)
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_filesystem.py:738: in chmod
    file_object = self.resolve(
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_filesystem.py:1709: in resolve
    return self.get_object_from_normpath(
C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_filesystem.py:1642: in get_object_from_normpath
    self.raise_os_error(errno.ENOENT, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x0000017B356D4790>, err_no = 2, filename = '/C:\\Users\\username\\AppData\\Local\\Temp/tmph6mkc_a2', winerror = None

    def raise_os_error(
        self,
        err_no: int,
        filename: Optional[AnyString] = None,
        winerror: Optional[int] = None,
    ) -> NoReturn:
        """Raises OSError.
        The error message is constructed from the given error code and shall
        start with the error string issued in the real system.
        Note: this is not true under Windows if winerror is given - in this
        case a localized message specific to winerror will be shown in the
        real file system.

        Args:
            err_no: A numeric error code from the C variable errno.
            filename: The name of the affected file, if any.
            winerror: Windows only - the specific Windows error code.
        """
        message = os.strerror(err_no) + " in the fake filesystem"
        if winerror is not None and sys.platform == "win32" and self.is_windows_fs:
            raise OSError(err_no, message, filename, winerror)
>       raise OSError(err_no, message, filename)
E       FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: '/C:\\Users\\username\\AppData\\Local\\Temp/tmph6mkc_a2'

C:\Users\username\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyfakefs\fake_filesystem.py:429: FileNotFoundError
======================================================================================================================================================================= short test summary info =======================================================================================================================================================================
FAILED test.py::test_subtest - FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: '/C:\\Users\\username\\AppData\\Local\\Temp/tmph6mkc_a2'
========================================================================================================================================================================== 1 failed in 1.16s ========================================================================================================================================================================== 

Your environment
Please run the following in the environment where the problem happened and
paste the output.

python -c "import platform; print(platform.platform())"
Windows-10-10.0.19045-SP0
python -c "import sys; print('Python', sys.version)"
Python 3.11.1 (tags/v3.11.1:a7a450f, Dec  6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)]
python -c "from pyfakefs import __version__; print('pyfakefs', __version__)"
pyfakefs 5.3.1
python -c "import pytest; print('pytest', pytest.__version__)"
pytest 7.4.3

[mrbean-bremen]

Thanks, I will have a look later tonight!
I've not used pytest-subtests so far, so it may well be that there is a problem. At a first glance it looks patching is still active while collecting log files, but it may also be a problem due to emulating Linux under Windows.

[mrbean-bremen]

Ok, this is a problem with skipping modules (in this case the capture module), which does not work as expected, because it still may indirectly call patched modules (at least in the case of emulating Linux under Windows). To fix this properly may take a while, I have to think about this.

I also tried to not skip the capture module, which should result in all capture logs written in the fake filesystem and taken from there, but that has its own problems related to some manipulation of the standard stream that currently do not work. This is something that also needs fixing, I think.

I haven't thought yet of a workaround that could be used meanwhile (other than avoiding to emulate another os, of course), sorry...

[JakobDev]

To fix this properly may take a while, I have to think about this.

No pressure. take your time. Thanks for the quick response. If you need some change in pytest-subtests, here's a link to their Issue tracker.

[mrbean-bremen]

Turns out I was mistaken - this isn't complicated at all.
There may still be other problems, but this concrete problem was caused by a mismatch of os separators in the temp path (that is accessed by the tempfile module containing backslashes under Windows), and the expected separator for Linux. There have been similar problems before that have been fixed, but I have obviously missed this case.

[mrbean-bremen]

@JakobDev: Should be fixed in main now - can you please check if this fixes it for you?

[JakobDev]

I have tested it, It no longer crashes, but it still not works correctly. Take this example:

import pytest_subtests
import pyfakefs
import os


def test_subtest(subtests: pytest_subtests.SubTests, fs: pyfakefs.fake_filesystem.FakeFilesystem) -> None:
    fs.os = pyfakefs.fake_filesystem.OSType.LINUX

    with subtests.test("Test1"):
        assert os.path.abspath("/test.txt") == "/test.txt"

    with subtests.test("Test2"):
        assert os.path.abspath("/test.txt") == "/test.txt"

Test1 passes, but Test2 don't. In Test2 os.path.abspath("/test.txt") returns C:\\test.txt instead of /test, so it looks like it stopped working after the first subtest.

[mrbean-bremen]

Ok, thanks - I will have a look.

[mrbean-bremen]

That is actually another problem that is caused by a recent fix for #904. Patching is now paused at reporting start and resumed before the next test, but with subtests, the resume does not happen, because the respective hook is not called. Maybe I should check for another hook...

[mrbean-bremen]

It was actually an incorrect fix for the other issue that caused that - should have read the pytest documentation better...

Please check if it works for you now in main.

[JakobDev]

Unfortunately the error still exists.

[mrbean-bremen]

Hm, that's strange, I thought I had tested this... well, another round tonight, sorry. Just to be clear: the test above with the two subtests still fails in the same way, right?

[JakobDev]

Yes

Console Log

PS C:\Users\username\tmp> pip install -U git+https://github.com/pytest-dev/pyfakefs.git
Collecting git+https://github.com/pytest-dev/pyfakefs.git
  Cloning https://github.com/pytest-dev/pyfakefs.git to c:\users\username\appdata\local\temp\pip-req-build-sak3jfq0
  Running command git clone --filter=blob:none --quiet https://github.com/pytest-dev/pyfakefs.git 'C:\Users\username\AppData\Local\Temp\pip-req-build-sak3jfq0'
  Resolved https://github.com/pytest-dev/pyfakefs.git to commit cd3dc82c7f950f7276e78377577471fd4b89b7cc
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done

[notice] A new release of pip is available: 23.0 -> 23.3.1
[notice] To update, run: python.exe -m pip install --upgrade pip
PS C:\Users\username\tmp> cat ./test.py
import pytest_subtests
import pyfakefs
import os


def test_subtest(subtests: pytest_subtests.SubTests, fs: pyfakefs.fake_filesystem.FakeFilesystem) -> None:
    fs.os = pyfakefs.fake_filesystem.OSType.LINUX

    with subtests.test("Test1"):
        assert os.path.abspath("/test.txt") == "/test.txt"

    with subtests.test("Test2"):
        assert os.path.abspath("/test.txt") == "/test.txt"
PS C:\Users\username\tmp> pytest .\test.py
========================================================================================================================================================================= test session starts =========================================================================================================================================================================
platform win32 -- Python 3.11.1, pytest-7.4.3, pluggy-1.0.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: C:\Users\username\tmp
plugins: anyio-3.7.1, Faker-18.4.0, pyfakefs-5.4.dev0, asyncio-0.21.0, benchmark-4.0.0, cov-4.1.0, integration-0.2.3, mock-3.11.1, recording-0.12.2, subtests-0.11.0, xdist-3.3.1, requests-mock-1.11.0
asyncio: mode=Mode.STRICT
collected 1 item

test.py ,u.                                                                                                                                                                                                                                                                                                                                                      [100%]

============================================================================================================================================================================== FAILURES ===============================================================================================================================================================================
________________________________________________________________________________________________________________________________________________________________________ test_subtest [Test2] _________________________________________________________________________________________________________________________________________________________________________

subtests = SubTests(ihook=<_pytest.config.compat.PathAwareHookProxy object at 0x0000016C67302190>, suspend_capture_ctx=<bound met...te='started' _in_suspended=False> _capture_fixture=None>>, request=<SubRequest 'subtests' for <Function test_subtest>>), fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x0000016C6C2BB550>

    def test_subtest(subtests: pytest_subtests.SubTests, fs: pyfakefs.fake_filesystem.FakeFilesystem) -> None:
        fs.os = pyfakefs.fake_filesystem.OSType.LINUX

        with subtests.test("Test1"):
            assert os.path.abspath("/test.txt") == "/test.txt"

        with subtests.test("Test2"):
>           assert os.path.abspath("/test.txt") == "/test.txt"
E           AssertionError: assert 'C:\\test.txt' == '/test.txt'
E             - /test.txt
E             ? ^
E             + C:\test.txt
E             ? ^^^

test.py:13: AssertionError
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------------------------------------------------------------------------------------------

======================================================================================================================================================================= short test summary info =======================================================================================================================================================================
[Test2] SUBFAIL test.py::test_subtest - AssertionError: assert 'C:\\test.txt' == '/test.txt'
=========================================================================================================================================================== 1 failed, 1 passed, 1 subtests passed in 0.42s ============================================================================================================================================================
PS C:\Users\username\tmp> 

[mrbean-bremen]

Hm, I really cannot reproduce this. I tried to copy your environment as far as I can see, and I get:

(pyfakefs-3.11) c:\dev\GitHub>type subtest_test.py
import pytest_subtests
import pyfakefs
import os


def test_subtest(subtests: pytest_subtests.SubTests, fs: pyfakefs.fake_filesystem.FakeFilesystem) -> None:
    fs.os = pyfakefs.fake_filesystem.OSType.LINUX

    with subtests.test("Test1"):
        assert os.path.abspath("/test.txt") == "/test.txt"

    with subtests.test("Test2"):
        assert os.path.abspath("/test.txt") == "/test.txt"

(pyfakefs-3.11) c:\dev\GitHub>pytest -v subtest_test.py
============================================================================ test session starts ============================================================================
platform win32 -- Python 3.11.1, pytest-7.4.3, pluggy-1.0.0 -- C:\dev\venv\pyfakefs-3.11\Scripts\python.exe
cachedir: .pytest_cache
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
metadata: {'Python': '3.11.1', 'Platform': 'Windows-10-10.0.19045-SP0', 'Packages': {'pytest': '7.4.3', 'pluggy': '1.0.0'}, 'Plugins': {'anyio': '3.7.1', 'Faker': '18.4.0', 'pyfakefs': '5.4.dev0', 'asyncio': '0.21.0', 'benchmark': '4.0.0', 'cov': '4.1.0', 'integration': '0.2.3', 'metadata': '3.0.0', 'mock': '3.11.1', 'recording': '0.12.2', 'subtests': '0.11.0', 'xdist': '3.3.1', 'requests-mock': '1.11.0'}}
rootdir: c:\dev\GitHub
plugins: anyio-3.7.1, Faker-18.4.0, pyfakefs-5.4.dev0, asyncio-0.21.0, benchmark-4.0.0, cov-4.1.0, integration-0.2.3, metadata-3.0.0, mock-3.11.1, recording-0.12.2, subtests-0.11.0, xdist-3.3.1, requests-mock-1.11.0
asyncio: mode=Mode.STRICT
collected 1 item

subtest_test.py::test_subtest [Test1] SUBPASS                                                                                                                          [100%]
subtest_test.py::test_subtest [Test2] SUBPASS                                                                                                                          [100%]
subtest_test.py::test_subtest PASSED                                                                                                                                   [100%]

=================================================================== 1 passed, 2 subtests passed in 0.46s ====================================================================

To make sure, I also installed pyfakefs from GitHub as you did. SO, at the moment I'm stumped... can you think of anything else in your environment that may influence that?

[JakobDev]

Interesting. I now tried to uninstall pyfakefs before installing it from git and it works. It looks like pip have not installed the package before for some reason. Maybe -U is not correctly working for git. Many thanks for your patience and the fast solution!

[mrbean-bremen]

Ah, that's good to hear! I was out of ideas here...
If it works for you, I will make a new patch release, and I guess we can close this one.

[mrbean-bremen]

The release is out.