Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

building sdist fails with CMake subproject #668

Open
lucascolley opened this issue Sep 16, 2024 · 15 comments
Open

building sdist fails with CMake subproject #668

lucascolley opened this issue Sep 16, 2024 · 15 comments

Comments

@lucascolley
Copy link

lucascolley commented Sep 16, 2024 •
edited
Loading

Traceback:

$ python -m build --no-isolation -x -Csetup-args="-Duse-pythran=false"
...
+ meson setup D:\a\scipy\scipy D:\a\scipy\scipy\.mesonpy-nmuol9tn -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md -Duse-pythran=false --native-file=D:\a\scipy\scipy\.mesonpy-nmuol9tn\meson-python-native-file.ini
...
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 373, in <module>
    main()
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 357, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 326, in build_sdist
    return backend.build_sdist(sdist_directory, config_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\mesonpy\__init__.py", line 1020, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\mesonpy\__init__.py", line 1061, in build_sdist
    with _project(config_settings) as project:
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\contextlib.py", line 144, in __exit__
    next(self.gen)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\mesonpy\__init__.py", line 944, in _project
    with contextlib.ExitStack() as ctx:
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\contextlib.py", line 601, in __exit__
    raise exc_details[1]
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\contextlib.py", line 586, in __exit__
    if cb(*exc_details):
       ^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 943, in __exit__
    self.cleanup()
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 947, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 929, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 787, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 629, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 629, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 629, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 1 more time]
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 638, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 919, in onerror
    cls._rmtree(path, ignore_errors=ignore_errors,
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 929, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 787, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 638, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 919, in onerror
    cls._rmtree(path, ignore_errors=ignore_errors,
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 929, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 787, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 638, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 636, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '.\\.mesonpy-nmuol9tn\\meson-private\\cmake_boost_math\\CMakeFiles\\CMakeScratch'

ERROR Backend subprocess exited when trying to invoke build_sdist

Full log: https://github.com/scipy/scipy/actions/runs/10884814141/job/30200964434?pr=21270

x-ref scipy/scipy#21270
@lucascolley
Copy link
Author

After naive searches for shutil.rmtree here and in mesonbuild/meson, perhaps this is an issue for that repo instead.

@lucascolley
Copy link
Author

@eli-schwartz do you have the permission to transfer this issue to the other repo? (if that looks like the right idea here?)

@dnicolodi
Copy link
Member

dnicolodi commented Sep 16, 2024 •
edited
Loading

The stack trace points to the code in meson-python that cleans up the temporary build directory, thus this does not look to be a meson issue. However, I do not have any idea of what is keeping files in the build directory busy, except something related to meson. Windows is so weird in this regard that I don't even know where to start looking and I don't have easy access to a system where I can test this. Someone would need to debug this.

EDIT: The interesting part of the stack trace has been cut off from the report, but it is available in the CI log. Here is the whole thing:

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 373, in <module>
    main()
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 357, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 326, in build_sdist
    return backend.build_sdist(sdist_directory, config_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\mesonpy\__init__.py", line 1020, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\mesonpy\__init__.py", line 1061, in build_sdist
    with _project(config_settings) as project:
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\contextlib.py", line 144, in __exit__
    next(self.gen)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\mesonpy\__init__.py", line 944, in _project
    with contextlib.ExitStack() as ctx:
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\contextlib.py", line 601, in __exit__
    raise exc_details[1]
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\contextlib.py", line 586, in __exit__
    if cb(*exc_details):
       ^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 943, in __exit__
    self.cleanup()
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 947, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 929, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 787, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 629, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 629, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 629, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 1 more time]
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 638, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 919, in onerror
    cls._rmtree(path, ignore_errors=ignore_errors,
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 929, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 787, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 638, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 919, in onerror
    cls._rmtree(path, ignore_errors=ignore_errors,
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\tempfile.py", line 929, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 787, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 638, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\shutil.py", line 636, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '.\\.mesonpy-nmuol9tn\\meson-private\\cmake_boost_math\\CMakeFiles\\CMakeScratch'

@lucascolley
Copy link
Author

oops, thanks for catching the full traceback! I've edited the top post

@lucascolley
Copy link
Author

I do not have any idea of what is keeping files in the build directory busy, except something related to meson.

If I understand you correctly, it looks like a process related to building the CMake subproject introduced in that PR:

.\\.mesonpy-nmuol9tn\\meson-private\\cmake_boost_math\\CMakeFiles\\CMakeScratch

@dnicolodi
Copy link
Member

Not building but just configuring. Sub-projects are not build, only configured, to create an sdist. I saw the path in the stack trace. But I have no idea what may keep that file busy. The only process executed by meson-python is meson. And meson is a single process Python application, thus parts of it are for sure not lingering around. For configuring a CMake subproject, meson executes cmake, but I'm pretty sure it waits for the child process to exit before continuing. Thus it is either a process started by CMake that stays around, or it is some weird Windows thing where something interposes between the process opening the file and the filesystem.

@eli-schwartz
Copy link
Member

Well, cmake's normal means of operation when configuring a cmake build, is for the cmake equivalent of meson's cc.links() to produce a tiny ninja/make project in ${CMAKE_BINARY_DIR}/CMakeFiles/CMakeScratch/<random_name> and then try building that and checking if the ninja/make project succeeds.

This may be a recursion issue -- but I have no idea what or how cmake waits on its own subprocesses. Perhaps a sub-cmake is still lingering?

CMakeScratch is a definite red flag though because a directory filename as specific as that is absolutely 100% a cmake internal and meson doesn't touch it either. And cmake claims it cleans that up automatically as part of its own configure stage before exiting.

@lucascolley
Copy link
Author

@mwoehlke-kitware I noticed that you touched the following lines. Perhaps you could shed some light here? https://github.com/Kitware/CMake/blob/master/Help/command/try_compile.rst#L120-L125

@mwoehlke-kitware
Copy link

I don't see how this has anything to do with the change cited? I'm not at all a Windows expert, but I do seem to recall running into files "locked" by the OS longer than you'd expect is a thing that can happen when compiling code. Maybe try asking on a more general CMake forum, especially one that gets more Windows users?

@lucascolley
Copy link
Author

thanks for the quick response! will do

@lucascolley
Copy link
Author

@eli-schwartz would you be able to tell me what call meson makes to cmake so that I can report the potential problem on their side?

@eli-schwartz
Copy link
Member

I can look for this information later tonight. But off the top of my head I think we're supposed to log the exact call in the debug log...

@lucascolley
Copy link
Author

I tried to build an sdist locally on windows to access a debug log here but I wasn't able to set up compilers correctly - building SciPy on windows can be a bit of a nightmare.

@lucascolley lucascolley mentioned this issue Nov 15, 2024
Draft
@lucascolley
Copy link
Author

This seems to have resolved itself?! Has anything changed on the meson side in relation to this as far as you know @eli-schwartz ?

@eli-schwartz
Copy link
Member

As far as I'm aware nothing has changed on the meson side. How mysterious...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dnicolodi @mwoehlke-kitware @eli-schwartz @lucascolley