-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
import('python').find_installation('python3').get_install_dir() adds an extra 'local' to the path #10459
Comments
This is the output of the introspection command in the Python module.
|
@eli-schwartz @xclaesse the issue here seems to be that the prefix is defaulting to On Ubuntu 18:
On Fedora 36:
On Fedora 35:
@jameswestman I think this is a Fedora issue. One of these is not like the others. |
I'm going to hazard a wild guess that continued inconsistent patching by a dozen different distros has once again broken our heuristics, given it changes depending on the version of Fedora. The emergency solution of last recourse here, is to define the option The proper solution will be to try to figure out what, exactly, we are supposed to do in a situation where Fedora is plainly trying to tell us that even when we configure with --prefix=/usr, python files should be installed into /usr/local because they want |
I think we should bring this up with Fedora to be honest. Tried opening a discussion on bugzilla, but unsuccessful trying to create an account. |
@bonzini any chance you can report this on the RedHat Bugzilla? I have been trying to create an account, but have been unsuccessful. |
I filed this downstream at https://bugzilla.redhat.com/show_bug.cgi?id=2097183. I'll post here with any updates. |
As a datapoint, this is also on fedora36, it also fails if you specify |
Possibly the same/similar issue on MacOS with python3 installed via brew, I see a duplicate
edit: diving into this a bit, it looks like:
I'll submit a bug against homebrew (Homebrew/homebrew-core#120032), hopefully they can inject their own base vars and let purelib/platlib get constructed based on those. |
Is there a workaround in meson we can apply in the meantime? I'm having the same problem in a project I work on, on MacOS. |
Presumably, Meson could use the |
Unfortunately, there's (I think) still a bug with that because instead of using the system folders for purelib/platlib, meson concats meson/mesonbuild/modules/python.py Line 486 in ded5770
So this still results in invalid paths on MacOS because purelib/platlib are absolute paths. Using .get_path('purelib') directly works though.
|
@ret2libc my currently implemented workaround is is to avoid using
then in meson.build:
Caveat being this is fine because my module is pure python so I don't need platlib. This approach defaults to meson's values and will preserves compatibility if this gets fixed in a later release, but for now I can instruct my MacOS users to set |
This was fixed in Fedora 37 and still works in Fedora 38. Closing since this was a Fedora issue the whole time. |
This is still a potential possibility however for future issues. |
The conflict between CPython and distro install schemes and the selection methods of adapting to magic environment variables serves to highlight a big problem with how install schemes are designed. Namely that they weren't, really, designed. And they're being retrofitted for purposes that they don't do a good job of handling. There's plans to redesign sysconfig in a future version of CPython, which is specifically intended to provide a future-proof way to do stuff like this. |
Fine with closing it, by the way – had to migrate away from meson for this reason before F37 already, hope to revisit once development time resurfaces. |
I'm getting this issue still on macosx
|
I wonder if #12113 is related. |
It certainly looks similar... |
I'm seeing it in Ubuntu (at least in 23.10). I tried installing the following gnome modules:
All three wind up with stuff going into
All other distros (so far) seem to not have this problem. I'll look for and/or file a downstream issue. Update: Just filed https://bugs.launchpad.net/ubuntu/+source/python3.11/+bug/2052443 |
Apparently distros can make changes which cause the install dir to be wrong, such as including an extra "local". See: * mesonbuild/meson#10459 * https://bugzilla.redhat.com/show_bug.cgi?id=2097183 Ubuntu is the latest one with this problem, with a downstream issue to either be found or filed. In the meantime, highlight the weirdness in the summary.
I've already made patches and justified the solution here. Could a
maintainer please revisit?
…On Mon, 5 Feb 2024 at 14:13, joanmarie ***@***.***> wrote:
I'm seeing it in Ubuntu (at least in 23.10). I tried installing the
following gnome modules:
- Orca (which I maintain)
- Accerciser
- Secrets
All three wind up with stuff going into /usr/local/local. It seems
similar to the Fedora 36 issue:
$ python3 -c "import sysconfig; print(sysconfig.get_paths(vars={'base': '', 'platbase': '', 'installed_base': ''}))"
{'stdlib': '/lib/python3.11', 'platstdlib': '/lib/python3.11', 'purelib': '/local/lib/python3.11/dist-packages', 'platlib': '/local/lib/python3.11/dist-packages', 'include': '/include/python3.11', 'platinclude': '/usr/include/python3.11', 'scripts': '/local/bin', 'data': '/local'}
All other distros (so far) seem to not have this problem.
I'll look for and/or file a downstream issue.
—
Reply to this email directly, view it on GitHub
<#10459 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB4O4GKQBUL22UPKEPXSVTYSDSI7AVCNFSM5X2XXJKKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJSG4YTAMJRGE2A>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Please revisit what? |
The solution and PR discussed in #12113 |
I have asked a Homebrew maintainer to take a look at that issue. |
It's kind of not a homebrew issue per se -
`python_info.py:get_install_paths` is trying to install in system library locations, and that causes the problem.
That could be the issue on ubuntu as well - fundamentally its a sysconfig issue.
One general work around could be to just always install to posix-prefix?
…On Fri, 9 Feb 2024 at 14:31, Tristan Partin ***@***.***> wrote:
I have asked a Homebrew maintainer to take a look at that issue.
—
Reply to this email directly, view it on GitHub
<#10459 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB4O4HU3OQ7SLZ6KZD65A3YSYXMDAVCNFSM5X2XXJKKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJTGYYDGNZWGQ4A>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
You mean because meson is a build system for system software, right?
At least on Linux, that's the default... Debian folds, mutilates, and spindles theirs via third party patching, so we special case this by checking their system library location as "deb_system". |
Installing in system locations should be done by the system packaging
mechanism. sure if meson wants to grow the ability to do that, that would
make sense, but right now it can’t - hence the issues being faced.
…On Fri, 9 Feb 2024 at 15:04, Eli Schwartz ***@***.***> wrote:
It's kind of not a homebrew issue per se -
python_info.py:get_install_paths is trying to install in system library
locations, and that causes the problem.
You mean because meson is a build system for system software, right?
That could be the issue on ubuntu as well - fundamentally its a sysconfig
issue.
One general work around could be to just always install to posix-prefix?
At least on Linux, that's the default...
Debian folds, mutilates, and spindles theirs via third party patching, so
we special case this by checking their system library location as
"deb_system".
—
Reply to this email directly, view it on GitHub
<#10459 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB4O4ANVF6W7SVDTJGIMV3YSY3HNAVCNFSM5X2XXJKKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJTGYYDSMJQGEZA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
The issue being faced sounds like a broken Homebrew build. |
Nope, its purely that python_info.py is choosing osx_framework_library scheme, and installing to that location isn't something that is expected behaviour - its OSX system installed libraries only. |
That's not how system packaging works. Build systems have the ability to be configured with a --prefix option. This use case is fundamental to real build systems, and meson unconditionally supports this use case for generic language support encompassing C, C++, Fortran, Rust, Swift, Java, D, Vala, and numerous others, on top of data files, executable scripts, configuration files, and a wide range of other types of build system artifacts. This is all 100% configurable to suit a wide range of use cases. The builtin defaults are designed for the common case. ... Python is a special case here because python doesn't have a usability story for defining or selecting installation locations. There is work ongoing to make this better but no timeframe for solving it as far as I'm aware. The defaults which meson selects are intended to integrate with the rest of meson's configuration schema, which for example works well with Due to the ever-changing and undocumented chaos that is a dozen vendors each patching python in incompatible ways, sometimes this miscalculates, and we may have to add more bizarroland hacks, but it's important to document what the expected behavior is in such cases and why the hack is correct. Probably it was a mistake to ever trust sysconfig. It's not a standard if no one follows it. |
If you need to override the builtin detection you can use |
well, indeed, and agreed - the issue seems to me that sysconfig doesn't distinguish between locations that are system installed (i.e. part of the operating system), and places where installs should go |
homebrew is at fault here as well, as sysconfig.get_preferred_scheme("prefix") shouldn't ever return a system install (i.e. non-prefix install) location, as I raised here https://github.com/orgs/Homebrew/discussions/4858 |
The issue on mac is that you always have this issue, thanks to the above issue, and no-one at any level seems to want to fix it.... |
Aside: this ticket was originally reported about Fedora and now Ubuntu is being mentioned. Then we pivot to macOS? A solution for macOS definitely won't actually close this fedora ticket. Unfortunately I'm not immediately aware what the solution for fedora should be at all -- my understanding is that fedora's modifications to sysconfig don't involve schemes at all, they just modified the existing schemes to point at /usr/local with no recourse. I have no idea what's up with Ubuntu as last I checked, it was fine because we used deb_system. |
I'm not sure what this has to do with "deb_system isn't a prefix scheme" but as I feared, this conversation isn't looking likely to come to a conclusion about what is going on, let alone how to solve it, so I'm going to retire from the ticket for now. |
if only this was actually followed by macos.. https://packaging.python.org/en/latest/specifications/externally-managed-environments/#marking-an-interpreter-as-using-an-external-package-manager @tristan957 this might help with unravelling the ubuntu/fedora issues: https://packaging.python.org/en/latest/specifications/externally-managed-environments/#implementation-notes |
relevant PEP668 discussion: https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302/111?u=pradyunsg |
The Fedora issue has been fixed for a while |
Ugh, no. Sorry. Ignoring the technically correct answer as suggested during discussion, victim blaming, and then threatening users for utilizing their system to acquire software that isn't available as a wheel is not acceptable and that PEP is the worst thing to happen to Python in a decade. It's almost as bad as the python 2 -> 3 migration pain points. It's only less bad because it doesn't affect Windows users. You don't solve a technical problem by training users to add a flag to pip.conf acknowledging that they agree their concerns aren't valuable to others and they should not ask for help achieving their goals. And with that, I'm taking a break from thinking about the woes of the python ecosystem for at least a week. Unsubscribed, do not @ me, suffer your self inflicted issues in silence for all I care, G.O.O.D.B.Y.E. |
@eli-schwartz was that directed at me?.... I've certainly not intended to ignore any technicallty correct answer, victim blame or threaten users so quite confused at your response. |
I think Eli is just done with any python packaging issues for a bit :). Don't blame yourself! Like I said, I have reached out to my homebrew contact. I'll see what he says on Monday. |
Thanks @tristan957 that's appreciated. Having spent a while on this, I totally get his frustration... |
also my apologies for hijacking! I initially confused which ticket I saw the activity on. |
No problem. Yes, Python packaging is one of the worst aspects about it. Leaves much to be desired :(. |
Describe the bug
Meson is installing my Python project to the wrong location. Specifically,
import('python').find_installation('python3').get_install_dir()
is returning a wrong location with an extra 'local' path segment.To Reproduce
meson.build:
No other files are necessary to demonstrate the bug. Just run
meson _build
. Themessage()
line printsMessage: /usr/local/local/lib/python3.10/site-packages/
.Expected behavior
The
message
line should print something like/usr/local/lib/python3.10/site-packages/
.system parameters
The text was updated successfully, but these errors were encountered: