import('python') seperate python binary from libs

[Neumann-A]

So vcpkg separates the python tool from the python libs as such whatever I try the layout will not be as meson hardcodes/expects it to be and as thus won't be able to find it. I have pkg-config files for python but somehow the python module is not clever enough to use the pkg_config_path setup I fed it but rather wants LIBPC to be set (whatever that is; it is not documented and google didn't bring up any useful).
So please provide proper injection points for solving these path issue instead of being a now-it-all because apparently that doesn't work.

The current behavior blocks migrating vcpkg from meson 0.63 to 0.64.1 since extension_module is unusable due to not being able to find the libs. (vcpkg solved it previously by manually supplying libpython_dep=dependency(....))

[Neumann-A]

also vcpkg as theoretically 5 python locations:

a) the python executing meson
b) the python binary location for the target
c) the python binary location for the host ( which can be different from the python executing meson )
d) the python library location for the target
( e) the python library location for the host )

I doubt meson is capable of correctly distinguishing these cases.

[dnicolodi]

So vcpkg separates the python tool from the python libs

What does this mean in practice? How do you build Python extensions with setuptools for such a system? How does a Python interpreter know from where to load libraries if there is no relation between the interpreter and the library locations?

You can build extensions modules for a Python different than the one used to execute Meson just fine by using --native-file to specify the path to the interpreter you want to build for, or you can rely on meson-python to do that for you.

[Neumann-A]

What does this mean in practice? How do you build Python extensions with setuptools for such a system? How does a Python interpreter know from where to load libraries if there is no relation between the interpreter and the library locations?

That is totally irrelevant for the issue. Meson shouldn't be concerned about those issues. setuptools won't be used. vcpkg is not going to package python stuff it just needs the capability to execute build scripts wanting to also build python extensions.

You can build extensions modules for a Python different than the one used to execute Meson just fine by using --native-file to specify the path to the interpreter you want to build for,

How? via python = '<path_to_python>' ? Every time I tried that it crashed meson.

[dnicolodi]

You are not providing enough information to understand where the issue is and whether there is any possibility to support your use case. Vague statements about the environment in which you are trying to build Python extensions and a generic "It does not work" and are not very actionable. Setuptools are the de facto standard solution to build Python extensions. If setuptools works in the vcpkg environment, most likely meson works too or can be made to work.

How? via python = '<path_to_python>' ? Every time I tried that it crashed meson.

Can you be more precise about what you tried and how it failed?

[Neumann-A]

meson/mesonbuild/modules/python.py

Line 292 in 8a72558

if DependencyMethods.PKGCONFIG in methods:

Did not find python3. I provide have a pkg_config_path setup with the correct python-3.10.pc but it wasn't used/found.

meson/mesonbuild/modules/python.py

Lines 319 to 322 in 8a72558

	# We only need to check both, if a python install has a LIBPC. It might point to the wrong location,
	# e.g. relocated / cross compilation, but the presence of LIBPC indicates we should definitely look for something.
	if pkg_libdir is not None:
	candidates.append(functools.partial(PythonPkgConfigDependency, pkg_name, env, kwargs, installation))

It should be always go looking? Removing the if makes the lookup work....

[dnicolodi]

I still have no idea of what you are trying to do, how you tried to do it, and how it failed. Would you mind going one step back and provide an example meson.build and the meson command you are running with its output?

[Neumann-A]

I still have no idea of what you are trying to do

building stuff in vcpkg using meson. In more detail: Updating meson from 0.63 to 064.1. The error is e.g.

Could not find Python3 library 'D:\\downloads\\tools\\python\\python-3.10.7-x64\\libs\\python310.lib'

..\src\spection-1-c68dee862a.clean\giscanner\meson.build:112:0: ERROR: Python dependency not found

because meson is not using the pc file provided by vcpkg for python because it never looks for it and just tries to find some system installed libs....
failing meson.build line is here https://gitlab.gnome.org/GNOME/gobject-introspection/-/blob/main/giscanner/meson.build#L113

and example for a native file is (for gstreamer in this case but it is virtually the same for gobject-introspection):
meson-x64-windows-dbg.log

I previously patched it by adding/modifying the meson.build to include:

+libpython_dep = dependency('python-3.10', method : 'pkg-config')
+python_ext_dep = libpython_dep
- python_ext_dep = python.dependency()

but that is no longer working since meson automatically pulls in the dependency now for python.extension_module and throws and error if it does not exist.

[eli-schwartz]

but rather wants LIBPC to be set (whatever that is; it is not documented and google didn't bring up any useful)

It's a python config_var(), many of which (including this one) come from the Unix Makefile used to build python and therefore are not available when building with the Visual Studio solution.

Sadly python upstream doesn't distribute a pkg-config file at all in such a situation.

It should be always go looking? Removing the if makes the lookup work....

That would essentially say "I know that python itself doesn't provide a pkg-config file, but check if the user hand-wrote one and added it to the default pkg_config_path", which is probably wrong in all situations other than vcpkg but would surely help vcpkg.

It's probably usually harmless other than wasted time doing a lookup, but might have odd effects when using meson to build extensions for a python that came directly from python.org and also having e.g. msys2 python in the environment.

That is totally irrelevant for the issue. Meson shouldn't be concerned about those issues. setuptools won't be used.

I would say it's very relevant, as it indicates more general issues than just Meson.

vcpkg is not going to package python stuff it just needs the capability to execute build scripts wanting to also build python extensions.

Sorry, what does this mean?

Are you saying you just need meson to successfully configure the project, but you don't actually want to build code relying on meson's import('python') module?

If you're trying to build projects that have multiple build artifacts including a python module, maybe the right solution here is to ask those projects to add an option to disable building the python extension component? e.g. -Dpython-bindings=disabled?

[dnicolodi]

From the information provided so far and from the observation that Meson correctly works with Python installations of all kinds, except the one provided by vcpkg, I would lean toward saying that vcpkg provides a non functional Python installation.

What you linked as a meson setup log seems to be a Meson native file. The native file helps, but the command output would help more. The content of the pkg-config file installed by vcpkg would also be useful.

[Neumann-A]

I would lean toward saying that vcpkg provides a non functional Python installation.

No that is not the issue. Python from vcpkg can be used fine.....it works in cmake and i can manually start and use it.....

log seems to be a Meson native file

yeah i never said something different....

That would essentially say "I know that python itself doesn't provide a pkg-config file, but check if the user hand-wrote one and added it to the default pkg_config_path", which is probably wrong in all situations other than vcpkg but would surely help vcpkg.
It's probably usually harmless other than wasted time doing a lookup, but might have odd effects when using meson to build extensions for a python that came directly from python.org and also having e.g. msys2 python in the environment.

Meson should behave sane instead of having unexpected behavior. If I have a pkg-config for python it should just use it instead of being clever and have introspection on how python normally works.... meson needs to support all possible use cases and hand feeding it a valid pkg-config file is one of those.

If you're trying to build projects that have multiple build artifacts including a python module, maybe the right solution here is to ask those projects to add an option to disable building the python extension component? e.g. -Dpython-bindings=disabled?

Yeah you do that, I just fix meson in vcpkg in the meantime.

---

as I said. The issue is fixed if if pkg_libdir is not None: is removed as said in #11129 (comment)

[dnicolodi]

Can you please stop trying to convince us that Meson is doing the wrong thing and provide the required information? We are trying to help you and trying to understand how to support your use case but you are making it as hard as it can be. No one with the possibility to fix the alleged bug in Meson seems to be affected by the issue you report, therefore your only way to get the issue resolved is to collaborate. If you think you know better, I'm done loosing my time on this.

No that is not the issue. Python from vcpkg can be used fine.....it works in cmake and i can manually start and use it.....

I'm pretty sure you cannot build a package using setuptools with it. And this classifies as "broken" for me.

[Neumann-A]

and provide the required information?

What extra information do you want? I already gave all relevant information. The python module is not using pkg-config files but instead hardcoding search paths for the python libs on windows.

here is the pkg-config file if you need it:
python-3.10.pc.log
the -embed variant has the python3 lib in Libs:. The important part is that meson appends the include.

therefore your only way to get the issue resolved is to collaborate

doubt that. My issues will be solved by microsoft/vcpkg#28084 by simply patching meson used by vcpkg as described. vcpkg also fixes broken cmake modules so I just apply the same logic for meson.

I'm pretty sure you cannot build a package using setuptools with it. And this classifies as "broken" for me.

Why does meson care about that? Is meson using setuptools somehow?
vcpkg is not in the business of packaging python packages so it doesn't need a working setuptools (also I never tried it, so it might be working). vcpkg main goal is to provide working c/c++ packages and some ports have the capability to also build python bindings for those c/c++ libs.

[dnicolodi]

What extra information do you want?

It took a bit of effort but you have provided an example meson.build and python-3.10.pc. I also asked the meson setup output log. But it seems that you solved your issue and this ticket can be closed.

[eli-schwartz]

Why does meson care about that? Is meson using setuptools somehow?

distutils, but yes. (A little bit. Not for finding libraries, but yes for checking iff the libraries should be linked to.)

vcpkg is not in the business of packaging python packages so it doesn't need a working setuptools (also I never tried it, so it might be working). vcpkg main goal is to provide working c/c++ packages and some ports have the capability to also build python bindings for those c/c++ libs.

It's somewhat relevant inasmuch as ports that build python bindings for those c/c++ libs would historically tend to use setuptools to do it.

[eli-schwartz]

For a PCBuild version of python, which I assume is what vcpkg uses, Meson will check for the DLL using sys.base_prefix which is apparently wrong in vcpkg?

[Neumann-A]

I also asked the meson setup output log

Here let me throw more irrelevant logs at you: (actually need to extract them before uploading to github else they are empty):
config-x64-windows-dbg-out.log
config-x64-windows-dbg-meson-log.txt.log

[dnicolodi]

The python-3.10.pc seems to be wrong: Python requires linking with libpython on Windows, and that pkg-config file does not specify any linker flags to that effect.

[Neumann-A]

For a PCBuild version of python, which I assume is what vcpkg uses, Meson will check for the DLL using sys.base_prefix which is apparently wrong in vcpkg?

meson/mesonbuild/modules/python.py

Lines 220 to 222 in 9e8a3b9

	libpath = Path('libs') / f'python{vernum}.lib'
	# base_prefix to allow for virtualenvs.
	lib = Path(self.variables.get('base_prefix')) / libpath

Seems not to be looking for a DLL. The correct path would be located in base_prefix/../../(debug/)?lib and that is something meson/python cannot deal with by default. Which is why feeding pkg-config is superior.

The python-3.10.pc seems to be wrong: Python requires linking with libpython on Windows, and that pkg-config file does not specify any linker flags to that effect.

Have not yet seen linker errors due to missing symbols. So I assume this isn't generally true.

[dnicolodi]

Have not yet seen linker errors due to missing symbols. So I assume this isn't generally true.

Are you saying you want to use a pkg-config file to specify the path where to find a library you are not going to link with?

[tristan957]

I can't find authoritative docs on this, but python/cpython#65735 (comment)

This completely breaks building extension modules on Windows-based platforms like Cygwin and MinGW, where it is necessary when building even shared libraries for all global symbols to resolvable at link time.

This was also originally discussed in #4117.

Window has this documentation too

https://learn.microsoft.com/en-us/visualstudio/python/working-with-c-cpp-python-in-visual-studio?view=vs-2022#troubleshoot-compiling-failures

Error: Unable to locate Python libraries

Solution: Verify that the path: Linker > General > Additional Library Directories in the project properties points to your Python installation's libs folder. See step 6 under Create the core C++ project.

[Neumann-A]

@eli-schwartz: Even if I modify _PREFIX and _BASE_PREFIX in sysconfig.py meson still searches in the wrong dirs. (I mean libs will always be wrong but at least the base prefix should be correct which it isn't)

[eli-schwartz]

Can you check if this is still a problem after commit aa69cf0?

vcpkg solved it previously by manually supplying libpython_dep=dependency(....))

Since the commit I mentioned, dependency('python3') and py_inst.dependency() share much of the implementation and yield the same _PythonDependencyBase.

Did not find python3. I provide have a pkg_config_path setup with the correct python-3.10.pc but it wasn't used/found.

We don't skip checking for pkg-config even on Windows, if you get it via dependency('python3'), which is how that dependency behaved even before the refactor. But the refactor likely means that dependency('python3') satisfies the internal linkage requirements for extension_module.

[Neumann-A]

Can you check if this is still a problem after commit aa69cf0?

Is that commit part of 1.1.0 ? I currently use https://github.com/microsoft/vcpkg/pull/28084/files#diff-252e4529dcfc3d1f5cdda010e195917889e89a0dacc875caa16446add749343e to fix one issue. Had do deactivate the x86 build of gobject-introspection since it tried to build and link x64 libs. (I assume that had to do with the fact that python was x64 no it was because gobject-introspection build something [g-ir-scanner] for x64 and used it in cross mode without informing it about the x86 detail; so not related to meson)

[eli-schwartz]

Is that commit part of 1.1.0 ?

Yes.