external_project: workaround for MinGW #13886

MihailJP · 2024-11-09T13:16:06Z

On Python 3.12, self.prefix.relative_to(self.prefix.drive) no longer works and raises error like:

ValueError: 'C:/msys64/mingw64' is not in the subpath of 'C:'

Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/mesonmain.py", line 193, in run
    return options.run_func(options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/msetup.py", line 365, in run
    app.generate()
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/msetup.py", line 188, in generate
    return self._generate(env, capture, vslite_ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/msetup.py", line 227, in _generate
    intr.run()
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreter/interpreter.py", line 3063, in run
    super().run()
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 169, in run
    self.evaluate_codeblock(self.ast, start=1)
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 195, in evaluate_codeblock
    raise e
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 187, in evaluate_codeblock
    self.evaluate_statement(cur)
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 219, in evaluate_statement
    return self.evaluate_if(cur)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 309, in evaluate_if
    self.evaluate_codeblock(i.block)
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 195, in evaluate_codeblock
    raise e
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 187, in evaluate_codeblock
    self.evaluate_statement(cur)
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 201, in evaluate_statement
    return self.function_call(cur)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 528, in function_call
    res = func(node, func_args, kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/decorators.py", line 237, in wrapper
    return f(*nargs, **wrapped_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/decorators.py", line 556, in wrapper
    return f(*wrapped_args, **wrapped_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreter/interpreter.py", line 869, in func_subproject
    return self.do_subproject(args[0], kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreter/interpreter.py", line 951, in do_subproject
    raise e
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreter/interpreter.py", line 939, in do_subproject
    return methods_map[method](subp_name, subdir, default_options, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreter/interpreter.py", line 990, in _do_subproject_meson
    subi.run()
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreter/interpreter.py", line 3063, in run
    super().run()
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 169, in run
    self.evaluate_codeblock(self.ast, start=1)
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 195, in evaluate_codeblock
    raise e
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 187, in evaluate_codeblock
    self.evaluate_statement(cur)
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 205, in evaluate_statement
    self.assignment(cur)
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 642, in assignment
    value = self.evaluate_statement(node.value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 207, in evaluate_statement
    return self.method_call(cur)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/interpreterbase.py", line 557, in method_call
    res = obj.method_call(method_name, args, kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreter/interpreterobjects.py", line 885, in method_call
    ret = method(state, args, kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/decorators.py", line 237, in wrapper
    return f(*nargs, **wrapped_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/interpreterbase/decorators.py", line 556, in wrapper
    return f(*wrapped_args, **wrapped_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/modules/external_project.py", line 296, in add_project
    project = ExternalProject(state,
              ^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/site-packages/mesonbuild/modules/external_project.py", line 93, in __init__
    self.prefix = self.prefix.relative_to(self.prefix.drive)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/msys64/mingw64/lib/python3.12/pathlib.py", line 682, in relative_to
    raise ValueError(f"{str(self)!r} is not in the subpath of {str(other)!r}")
ValueError: 'C:/msys64/mingw64' is not in the subpath of 'C:'

D:/Users/parrot/git/ham/subprojects/eggs/meson.build:14:8: ERROR: Unhandled python exception

    This is a Meson bug and should be reported!

eli-schwartz

This should really be reported to CPython as a regression.

mesonbuild/modules/external_project.py

test cases/common/33 run program/meson.build

MihailJP · 2024-11-11T03:23:57Z

It seems recent ObjFW update broke an Objective-C++ test case on MinGW.

ld.lld: error: undefined symbol: __getmainargs(int*, char***, char***, int, int*)

This pull request cannot be merged until the test case is fixed.

MihailJP · 2024-11-11T03:53:29Z

__getmainargs is in msvcrt.dll which is the runtime library of VC++. In MinGW it is available through CRT.

MihailJP · 2024-11-11T13:46:35Z

mingw-w64-clang-x86_64-libobjfw seems now fixed.

mesonbuild/modules/external_project.py

test cases/common/33 run program/check-mingw.py

test cases/frameworks/25 hdf5/test.json

eli-schwartz · 2024-11-18T22:25:57Z

@lazka

MihailJP · 2024-11-18T23:29:21Z

This is a quick fix AFK. Amended commit will be done tonight JST.

eli-schwartz · 2024-11-19T04:13:10Z

OK thanks. For the commit message of the first commit I'd also recommend a slight reformatting tweak while you are amending anyway:

On Python 3.12, self.prefix.relative_to(self.prefix.drive) no longer
works and raises error like:

```
ValueError: 'C:/msys64/mingw64' is not in the subpath of 'C:'
```

lazka · 2024-11-19T05:45:00Z

This should really be reported to CPython as a regression.

Looks like this change was on purpose: python/cpython#99031 since:

>>> os.path.isabs("C:")
False

lazka · 2024-11-19T05:46:04Z

In some reason, recent version of MSYS2 is shipped with /usr/bin/cmd
which makes the test fail

This has always been the case. So likely a Python behavior change as well. I'll try to figure out why.

lazka · 2024-11-19T05:56:26Z

mesonbuild/modules/external_project.py

-            self.prefix = self.prefix.relative_to(self.prefix.drive)
+            self.prefix = Path(relpath(self.prefix, self.prefix.drive))

        # self.prefix is an absolute path, so we cannot append it to another path.
-        self.rel_prefix = self.prefix.relative_to(self.prefix.root)
+        self.rel_prefix = Path(relpath(self.prefix, self.prefix.root))


Using relative_to(self.prefix.anchor) should also work in both cases as far as I understand. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.anchor

As long as prefix is absolute.

ymmv

lazka · 2024-11-19T06:34:27Z

This has always been the case. So likely a Python behavior change as well. I'll try to figure out why.

It's a regression introduced in python/cpython#103179 which makes shutil.which return paths that are not executable, from what I can tell.

eli-schwartz · 2024-11-19T06:45:39Z

Looks like this change was on purpose: python/cpython#99031 since:

Sigh, as with all things pathlib nothing can be fixed because every bad behavior is intentional to be bad. Hence dropping back to os.path.relpath, oh well... :)

lazka · 2024-11-19T06:53:14Z

It's a regression introduced in python/cpython#103179 which makes shutil.which return paths that are not executable, from what I can tell.

I've filed a bug python/cpython#127001 though that doesn't help us here I guess.

edit, this maybe?:

import shutil
import os

windows_exts = ('exe', 'msc', 'com', 'bat', 'cmd')

def which_fixed(cmd, *args, **kwargs):
    res = shutil.which(cmd, *args, **kwargs)
    # Work around https://github.com/python/cpython/issues/127001
    # for the common case at least
    if res is not None and not os.path.splitext(res)[1]:
        for ext in windows_exts:
            res = shutil.which(cmd + '.' + ext, *args, **kwargs)
            if res is not None:
                break
    return res

print(which_fixed("cmd"))

On Python 3.12, self.prefix.relative_to(self.prefix.drive) no longer works and raises error like: ``` ValueError: 'C:/msys64/mingw64' is not in the subpath of 'C:' ```

In some reason, recent version of MSYS2 is shipped with /usr/bin/cmd which makes the test fail

lazka · 2024-11-19T15:58:15Z

thanks @MihailJP

This computation of prefix and rel_prefix was re-written in mesonbuild#13886 but it introduced another bug where the leading slash was missing. In addition drive root should have trailing slash, or it would use different path as base of relpath in some cases.

MihailJP requested a review from jpakkane as a code owner November 9, 2024 13:16

MihailJP force-pushed the MihailJP-patch-1 branch from 22873d7 to 0fb6fa0 Compare November 9, 2024 16:08

eli-schwartz requested changes Nov 10, 2024

View reviewed changes

mesonbuild/modules/external_project.py Outdated Show resolved Hide resolved

mesonbuild/modules/external_project.py Outdated Show resolved Hide resolved

eli-schwartz reviewed Nov 10, 2024

View reviewed changes

test cases/common/33 run program/meson.build Outdated Show resolved Hide resolved

MihailJP force-pushed the MihailJP-patch-1 branch 11 times, most recently from a6e7e39 to 7ae4d8a Compare November 10, 2024 18:12

MihailJP force-pushed the MihailJP-patch-1 branch from 7ae4d8a to 63b7f8e Compare November 11, 2024 13:45

MihailJP force-pushed the MihailJP-patch-1 branch from 63b7f8e to 4d7de83 Compare November 12, 2024 07:45

MihailJP requested a review from eli-schwartz November 12, 2024 10:19

res2k mentioned this pull request Nov 17, 2024

make the testsuite handle MSYS2's addition of cmd #13915

Closed

eli-schwartz reviewed Nov 17, 2024

View reviewed changes

mesonbuild/modules/external_project.py Outdated Show resolved Hide resolved

test cases/common/33 run program/check-mingw.py Outdated Show resolved Hide resolved

eli-schwartz requested changes Nov 18, 2024

View reviewed changes

test cases/frameworks/25 hdf5/test.json Outdated Show resolved Hide resolved

MihailJP force-pushed the MihailJP-patch-1 branch from 0a5a992 to 9165d0b Compare November 18, 2024 21:58

MihailJP requested a review from eli-schwartz November 18, 2024 22:28

lazka reviewed Nov 19, 2024

View reviewed changes

lazka mentioned this pull request Nov 19, 2024

shutil.which() can return non-executable path starting with 3.12 on Windows python/cpython#127001

Closed

MihailJP added 3 commits November 19, 2024 17:30

make external_project work again on MinGW

75a9d69

On Python 3.12, self.prefix.relative_to(self.prefix.drive) no longer works and raises error like: ``` ValueError: 'C:/msys64/mingw64' is not in the subpath of 'C:' ```

make the testsuite handle MSYS2's addition of cmd

e889161

In some reason, recent version of MSYS2 is shipped with /usr/bin/cmd which makes the test fail

check if test 33 running under MinGW

071135d

MihailJP force-pushed the MihailJP-patch-1 branch from 57e2ab3 to 071135d Compare November 19, 2024 08:34

eli-schwartz merged commit 93f5931 into mesonbuild:master Nov 19, 2024
35 checks passed

lb90 mentioned this pull request Nov 19, 2024

ExternalProgram: Ensure that the path has no forward slashes on Windows #13924

Closed

na-trium-144 added a commit to na-trium-144/meson that referenced this pull request Nov 20, 2024

fix regression of relpath on external_project prefix in mesonbuild#13886

d413795

na-trium-144 mentioned this pull request Nov 20, 2024

fix external_project failing due to the drive letter in the path on Windows #13916

Open

lazka mentioned this pull request Nov 23, 2024

Revert msys2 python shutil workarounds #13949

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

external_project: workaround for MinGW #13886

external_project: workaround for MinGW #13886

MihailJP commented Nov 9, 2024

eli-schwartz left a comment

MihailJP commented Nov 11, 2024

MihailJP commented Nov 11, 2024 •

edited

Loading

MihailJP commented Nov 11, 2024

eli-schwartz commented Nov 18, 2024

MihailJP commented Nov 18, 2024

eli-schwartz commented Nov 19, 2024

lazka commented Nov 19, 2024 •

edited

Loading

lazka commented Nov 19, 2024 •

edited

Loading

lazka Nov 19, 2024 •

edited

Loading

lazka commented Nov 19, 2024 •

edited

Loading

eli-schwartz commented Nov 19, 2024

lazka commented Nov 19, 2024 •

edited

Loading

lazka commented Nov 19, 2024

external_project: workaround for MinGW #13886

external_project: workaround for MinGW #13886

Conversation

MihailJP commented Nov 9, 2024

eli-schwartz left a comment

Choose a reason for hiding this comment

MihailJP commented Nov 11, 2024

MihailJP commented Nov 11, 2024 • edited Loading

MihailJP commented Nov 11, 2024

eli-schwartz commented Nov 18, 2024

MihailJP commented Nov 18, 2024

eli-schwartz commented Nov 19, 2024

lazka commented Nov 19, 2024 • edited Loading

lazka commented Nov 19, 2024 • edited Loading

lazka Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

lazka commented Nov 19, 2024 • edited Loading

eli-schwartz commented Nov 19, 2024

lazka commented Nov 19, 2024 • edited Loading

lazka commented Nov 19, 2024

MihailJP commented Nov 11, 2024 •

edited

Loading

lazka commented Nov 19, 2024 •

edited

Loading

lazka commented Nov 19, 2024 •

edited

Loading

lazka Nov 19, 2024 •

edited

Loading

lazka commented Nov 19, 2024 •

edited

Loading

lazka commented Nov 19, 2024 •

edited

Loading