Wheel violates PEP517 Build requirements

[jameshilliard]

Specifically the build requirements state:

Project build requirements will define a directed graph of requirements (project A needs B to build, B needs C and D, etc.) This graph MUST NOT contain cycles. If (due to lack of co-ordination between projects, for example) a cycle is present, front ends MAY refuse to build the project.

However front ends MAY have modes where they do not consider wheels when locating build requirements, and so projects MUST NOT assume that publishing wheels is sufficient to break a requirement cycle.

This specific PEP517 Build requirements violation can be identified by the dependency cycle validator in build that I'm working on.

This flags the problem:

ERROR Failed to validate `build-system` in pyproject.toml, dependency cycle detected: `wheel` -> `setuptools.build_meta` -> `wheel`

[agronholm]

Does this mean wheel cannot ever have a standard pyproject.toml file?

[pfmoore]

In theory, it probably means that either wheel or setuptools should somehow use the in-tree backend mechanism in PEP 517 to break the loop.

In practice, though, people seem mostly happy with just providing wheels to break the build loop - the people who were focused on building everything from source (which was the motivating case for that feature) don't seem to have followed up by pushing build backends to self-host. Flit has a bootstrapping process that doesn't use that mechanism, I believe.

[layday]

Flit will be bundling tomli to break the loop from next version.

[agronholm]

But setuptools already uses the in-tree backend mechanism, here? Just what is wheel expected to do here?

[agronholm]

I've backed out of PEP 517 build metadata in pyproject.toml until a solution can be devised. I've left the tooling configuration there as it shouldn't pose a problem.

[layday]

Per PEP 518 a pyproject.toml without a build-system table is assumed to use setuptools and wheel. In pip, the presence of pyproject.toml is sufficient to trigger a PEP 517 build with the PEP 518 defaults.

[layday]

$ python -m pip list
Package    Version
---------- -------
pip        21.3.1
setuptools 60.1.1
$ python -m pip -v install --no-build-isolation https://github.com/pypa/wheel/archive/refs/heads/main.zip
Using pip 21.3.1 from /[...]/venv/lib/python3.10/site-packages/pip (python 3.10)
Collecting https://github.com/pypa/wheel/archive/refs/heads/main.zip
  Using cached https://github.com/pypa/wheel/archive/refs/heads/main.zip
  Running command /[...]/venv/bin/python /[...]/venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpgwsjdnm5
  running dist_info
  creating /private/tmp/pip-modern-metadata-ifnp9e4_/wheel.egg-info
  writing manifest file '/private/tmp/pip-modern-metadata-ifnp9e4_/wheel.egg-info/SOURCES.txt'
  warning: no files found matching '*.dynlib' under directory 'tests'
  no previously-included directories found matching 'tests/testdata/*/build'
  no previously-included directories found matching 'tests/testdata/*/dist'
  no previously-included directories found matching 'tests/testdata/*/*.egg-info'
  writing manifest file '/private/tmp/pip-modern-metadata-ifnp9e4_/wheel.egg-info/SOURCES.txt'
  error: invalid command 'bdist_wheel'
  Preparing metadata (pyproject.toml) ... error
WARNING: Discarding https://github.com/pypa/wheel/archive/refs/heads/main.zip. Command errored out with exit status 1: /[...]/venv/bin/python /[...]/venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpgwsjdnm5 Check the logs for full command output.
ERROR: Command errored out with exit status 1: /[...]/venv/bin/python /[...]/venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpgwsjdnm5 Check the logs for full command output.

Installing with isolation currently works because there's no published release with pyproject.toml but expect no binary installs to break.

[agronholm]

Per PEP 518 a pyproject.toml without a build-system table is assumed to use setuptools and wheel. In pip, the presence of pyproject.toml is sufficient to trigger a PEP 517 build with the PEP 518 defaults.

Wasn't that bug fixed quite a while ago?

[layday]

It's not a bug, it's intentional.

[agronholm]

So what you're saying that I cannot make a release before doing away with pyproject.toml entirely?

[layday]

You don't have to delete it I suppose so long as you exclude it from the sdist (and nobody tries to pip install from git).

[agronholm]

To play it safe, I've gone and removed it entirely. I hope I can put it back one of these days. Right now I just don't see how.

[jameshilliard]

people seem mostly happy with just providing wheels to break the build loop

PEP517 explicitly states:

projects MUST NOT assume that publishing wheels is sufficient to break a requirement cycle

Does this mean wheel cannot ever have a standard pyproject.toml file?

Maybe setuptools should vendor wheel like flit_core is vendoring tomli?

[pfmoore]

@jameshilliard I understand that, but is there a practical issue here, or is this just a matter of being careful to follow the standards to the letter, is what I'm asking?

Maybe setuptools should vendor wheel like flit_core is vendoring tomli?

Yes, setuptools is the build backend here (it exposes the PEP 517 hooks), and it's their job to ensure that they can be built from source without circular dependencies.

My point here is that wheel doesn't need to do anything, and I'm trying to confirm whether leaving it to setuptools to address the issue will have any practical implications. It seems like it hasn't until now, at least. It's a shame if wheel has to avoid using pyproject.toml just because setuptools isn't yet able to build without a cycle.

[layday]

The practical issue is that wheel can't be built with PEP 517 because there's a circular dependency between setuptools and wheel.

1. initiate wheel build of wheel
2. read pyproject.toml metadata
3. wheel declares setuptools as build dependency
4. initiate wheel build of setuptools
5. invoke get_requires_for_build_wheel
6. setuptools declares wheel as build dependency
7. return to 1

[agronholm]

Wouldn't the front-end download a previous version of wheel in step 7 though?

[layday]

I don't know what frontend might do that but pip fails with a lookup error.

[agronholm]

I don't know what frontend might do that but pip fails with a lookup error.

How can I reproduce that?

[layday]

I suppose you could try publishing a pyproject.toml version of wheel on test-pypi or use --find-links --no-index. This is what we observed with tomli when flit had a circular dependency on tomli.

[pfmoore]

I don't know what frontend might do that but pip fails with a lookup error.

Can you provide a way of reproducing this? I believe pip should work in this case, as @agronholm says.

The following is with wheel's trunk:

❯ type .\pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
❯ py -m pip wheel .
Processing c:\work\projects\wheel
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: wheel
  Building wheel for wheel (pyproject.toml) ... done
  Created wheel for wheel: filename=wheel-0.37.1-py3-none-any.whl size=35423 sha256=d946da414c650efbc04b3f3219682468a9b38a1e8cc05aa89d743c6aa68b432a
  Stored in directory: C:\Users\Gustav\AppData\Local\Temp\pip-ephem-wheel-cache-4g2qpl23\wheels\05\b7\91\0e22e5be2d8ef4ba274b325f65c12a97ea2de64e0936eb6aee
Successfully built wheel

Edit: I also tried other variations, including py -m pip wheel -f .\dist\ --no-binary :all: . and couldn't get a failure.

[layday]

No, I cannot provide a way of reproducing this because a version of wheel with pyproject.toml has not been published (and I suspect that pip is special-casing wheel because --use-pep517 doesn't propagate to wheel). You can see pypa/flit#451 for how it manifested for flit.

[jameshilliard]

I understand that, but is there a practical issue here, or is this just a matter of being careful to follow the standards to the letter, is what I'm asking?

For us our build infrastructure is simply not capable of resolving circular package dependencies, so not having dependency cycles really is something we do want. We also don't(by policy) support downloading pre-built pypi wheels, we always use sdists.

Yes, setuptools is the build backend here (it exposes the PEP 517 hooks), and it's their job to ensure that they can be built from source without circular dependencies.

Well I think it should probably be resolved in setuptools by vendoring, however based on my reading of PEP517 until that happens wheel can't use setuptools.

My point here is that wheel doesn't need to do anything, and I'm trying to confirm whether leaving it to setuptools to address the issue will have any practical implications. It seems like it hasn't until now, at least. It's a shame if wheel has to avoid using pyproject.toml just because setuptools isn't yet able to build without a cycle.

Yeah, probably best to resolve this in setuptools, I opened an issue there.

[pfmoore]

however based on my reading of PEP517 until that happens wheel can't use setuptools.

You could use that logic to say that nothing should use setuptools, because doing so triggers a loop. More accurately, though, wheel and setuptools simply can't claim 100% compliance with PEP 517. But 99% compliance is good enough for most users. In reality, it's "just" a bug, and wheel has implemented a workaround for now, until setuptools fixes the issue. But it's a low-priority bug IMO, because nearly everyone will simply use the wheels and not encounter the problem.

[jameshilliard]

You could use that logic to say that nothing should use setuptools, because doing so triggers a loop.

Yeah, that is the bigger issue IMO, the wheel PEP517 cycle was just the most clear obvious indication of the circular dependency problem with setuptools here.

More accurately, though, wheel and setuptools simply can't claim 100% compliance with PEP 517.

The solution to me seems to be to vendor wheel in setuptools which would eliminate the dependency cycle and allow wheel to use setuptools for PEP517 builds, AFAIU vendoring would bring setuptools/wheel into full compliance with PEP517 build requirements.

But 99% compliance is good enough for most users. In reality, it's "just" a bug, and wheel has implemented a workaround for now, until setuptools fixes the issue. But it's a low-priority bug IMO, because nearly everyone will simply use the wheels and not encounter the problem.

It's mostly a problem for distributions like us that have package tooling which is not capable of resolving circular package dependencies.

[layday]

But it's a low-priority bug IMO, because nearly everyone will simply use the wheels and not encounter the problem.

If pip, distros and their package managers are happy special casing setuptools indefinitely, then I suppose it is a low-priority bug.

[jameshilliard]

If pip, distros and their package managers are happy special casing setuptools indefinitely, then I suppose it is a low-priority bug.

Yeah, for us we should be fine as long as we can use legacy setup.py style builds/installs, but it would be good to fix for bootstrapping purposes in the future.

[agronholm]

Yeah, for us we should be fine as long as we can use legacy setup.py style builds/installs, but it would be good to fix for bootstrapping purposes in the future.

For how much longer should we maintain the capability for legacy installs, and for what purpose?

[jameshilliard]

For how much longer should we maintain the capability for legacy installs, and for what purpose?

For a package like wheel which is important for bootstrapping other build tools I would generally say as long as feasible(ie as long as it doesn't introduce significant maintenance burden) to simplify installs on legacy environments.

It looks like python 3.11(last version compatible with distutils) isn't going to be EOL until 2027, so that would probably be the upper range it would make sense to keep legacy install support around for.

[agronholm]

For the specific case of Buildroot, is there something preventing you from using PEP 517/518 tooling?

[jameshilliard]

For the specific case of Buildroot, is there something preventing you from using PEP 517/518 tooling?

Well circular dependencies(which out tooling is pretty much unable to handle in any way remotely cleanly) with flit_core were causing issues although that's at least mostly fixed by it vendoring tomli.

In general...the PEP517 tooling for cross builds+installs(since we pretty much can't use pip for anything at all due to it having extremely poor/non-existent cross compilation support) is really immature and difficult to bootstrap at the moment due to a bunch of pain points all over the place, for example even basic functionality like installer cli support is still a work in progress.

I have a work in progress PEP517 branch here but it's not exactly in a state that's mergeable upstream yet.

[pfmoore]

It looks like python 3.11(last version compatible with distutils) isn't going to be EOL until 2027, so that would probably be the upper range it would make sense to keep legacy install support around for.

This sounds like an incredibly long period to expect tools to retain support for legacy tooling (about 10 years since PEP 517 was approved, and still 5 years from now). If you really do not expect to be able to use more modern tools/techniques before then, I'd be really grateful if you'd open up a discussion on the packaging Discourse area to explore what the blockers are, and try to work out how we address them. I don't think it's reasonable to expect under-resourced, volunteer-driven projects to commit to such long support timescales.

the PEP517 tooling for cross builds+installs

How is that relevant to wheel? As a pure Python project (like basically all of the core packaging components) it isn't platform specific, so building anywhere should get you a universal wheel, which can be used in any environment.

since we pretty much can't use pip for anything at all due to it having extremely poor/non-existent cross compilation support

Pip isn't intended to support cross-compilation, so this is by design. In fact the only build tool I'm aware of that might handle cross-compilation is setuptools, and that's a backend, not a frontend. If your issue is that setuptools no longer supports the use of setup.py invocations directly, that's something you should take up with them directly. There is no standard for cross-compilation, so every build tool that supports it must (by definition) have a backend-specific means of doing a cross-build. The only relevant standard here is the "config options" support in PEP 517, but so far no-one has explored what a good UI would be for that, because no-one has come up with an important enough use case.

[jameshilliard]

This sounds like an incredibly long period to expect tools to retain support for legacy tooling (about 10 years since PEP 517 was approved, and still 5 years from now).

Yeah, that's an issue for most projects, I was just going off of the cpython EOL timelines for what might make sense here. It's subjective though.

If you really do not expect to be able to use more modern tools/techniques before then, I'd be really grateful if you'd open up a discussion on the packaging Discourse area to explore what the blockers are, and try to work out how we address them.

I've mostly been trying to work out the initial batch of project specific issues(like circular dep issues) while iterating on a buildroot integration first. Once those initial issue are resolved I should be able to do a write up on any remaining non-project specific blockers there.

Pip isn't intended to support cross-compilation, so this is by design.

Yeah, that's what I thought, which is why we need a PEP517 frontend that does support cross compilation(which seems to be build+installer).

the only build tool I'm aware of that might handle cross-compilation is setuptools

Legacy distutils does as well, but yeah we currently just have infrastructure for setuptools and distutils based python packages, I'm working on adding PEP517 support but it's rather non-trivial due to bootstrapping.

If your issue is that setuptools no longer supports the use of setup.py invocations directly, that's something you should take up with them directly.

I mean we definitely have to support PEP517(due to many critical packages requiring flit_core for building) I think so I'm really trying to figure out a more generalized migration/bootstrapping path, the bootstrapping process for PEP517 right now is not very straight forward with existing tooling(other than pip which as you mentioned is not designed/capable of cross compilation) being very immature. I think the bigger issue is that there's no clear replacement in a lot of cases for setup.py for PEP517 tooling bootstrapping.

There is no standard for cross-compilation, so every build tool that supports it must (by definition) have a backend-specific means of doing a cross-build. The only relevant standard here is the "config options" support in PEP 517, but so far no-one has explored what a good UI would be for that, because no-one has come up with an important enough use case.

Most of the techniques we're using seem to not be that difficult to use with PEP517, I mean even stuff like sysconfig overrides are fairly generalized and tend to work automatically in most cases. The lack of a mature pip alternative seems to be the bigger problem. This is more of a build+install frontend issue it seems than anything.

[pfmoore]

I was just going off of the cpython EOL timelines

CPython has orders of magnitude more people working on it than the packaging ecosystem, as well as having corporate sponsors. What makes sense for them is totally different from what makes sense for packaging tools.

Yeah, that's what I thought, which is why we need a PEP517 frontend that does support cross compilation

In the sense that you mean, PEP 517 doesn't handle cross-compilation. As I said, there's no standard for anything related to cross-compilation, so you shouldn't be expecting any tool support for it presently. Setuptools (and as you note, distutils before it) provided cross-compilation support¹ via setup.py invocations, but these were never standardised and were never supported by any other tool apart from pip's "pass random options to setuptools" support that was part of the legacy code that invoked setup.py directly. And which is now in the process of being removed in favour of pyproject.toml based builds. That doesn't remove the ability to do a cross-compilation based build, you just won't be able to do it via pip.

By desupporting the use of setup.py direct invocation, it's setuptools that has in effect desupported this cross-compilation support, and that's something you should take up with them. It shouldn't affect any other project, and in particular I have no idea why it's related to how wheel (which is pure Python and so completely portable) handles its own build process. This is why I suggested that you bring this up elsewhere - either with setuptools if you want to preserve the setup.py interface, or in Discourse if you want standards-based cross compilation support that can be provided by build backends other than setuptools (should any such build backends exist).

Footnotes

1. At least I assume it did from what you're saying. ↩

[jameshilliard]

In the sense that you mean, PEP 517 doesn't handle cross-compilation.

Well I mostly mean the frontend support needed for cross compilation(which was previously just a direct setup.py invocation) is rather immature.

these were never standardised and were never supported by any other tool apart from pip's "pass random options to setuptools" support that was part of the legacy code that invoked setup.py directly

We've basically always just bypassed pip entirely and hooked our tooling straight up to the setup.py.

And which is now in the process of being removed in favour of pyproject.toml based builds. That doesn't remove the ability to do a cross-compilation based build, you just won't be able to do it via pip.

The problem is not pip(since we've never used it to begin with) but the lack of a way to actually invoke a build+install from a python interpreter that doesn't have any packaging tools at all. It's unclear how this initial stage should be done without a setup.py file.

By desupporting the use of setup.py direct invocation, it's setuptools that has in effect desupported this cross-compilation support, and that's something you should take up with them. It shouldn't affect any other project, and in particular I have no idea why it's related to how wheel (which is pure Python and so completely portable) handles its own build process.

This part isn't really a cross-compilation issue so much as a general toolchain bootstrapping issue, the issue with wheel specifically is that it's a dependency of setuptools but also requires setuptools to build+install(circular dependency issue). Right now wheel has the PEP517 pyproject.toml removed to avoid the PEP517 violation but the circular dependency with setuptools still actually exists effectively as wheel still requires setuptools in the setup.py.

[agronholm]

This part isn't really a cross-compilation issue so much as a general toolchain bootstrapping issue, the issue with wheel specifically is that it's a dependency of setuptools but also requires setuptools to build+install(circular dependency issue).

I thought the issue was somehow related to cross compilation of other things...?

This part isn't really a cross-compilation issue so much as a general toolchain bootstrapping issue, the issue with wheel specifically is that it's a dependency of setuptools but also requires setuptools to build+install(circular dependency issue).

Which I am trying to fix in #436. Also, wheel has always required setuptools to build, and setuptools has always required wheel to build wheels, so why is this only becoming a problem for you now?

[jameshilliard]

I thought the issue was somehow related to cross compilation of other things...?

Well it's actually not cross compilation specific, it's just that we can't use pip at all(which is the usual way things are bootstrapped it seems). I think gentoo has a similar problem we're facing.

Which I am trying to fix in #436. Also, wheel has always required setuptools to build, and setuptools has always required wheel to build wheels, so why is this only becoming a problem for you now?

Oh, that's because setuptools only requires wheel for PEP517 builds, not non-.whl style builds+installs(ie python setup.py build && python setup.py install), we never built or used the wheel package at all previously(we don't even really want to have to build a .whl artifact at all in the first place, we just want to do a normal uncompressed in-tree package build ideally but that wasn't ever standardized in PEP517 so we're kinda stuck and now have to add .whl support now). This is a recent issue because PEP517 is introducing the .whl requirement that previous didn't exist.

[takluyver]

I think the term 'cross compilation' might be causing some confusion.

I think the people from the Python side (including me) assume this term refers to actual compilation, as in creating native machine code. This doesn't happen for Python code, so as far as we're concerned, it's a non-issue for pure Python packages, including all the packaging machinery we're talking about.

I think @jameshilliard is using the term in a rather looser sense, including getting things installed in the right places when you're not in the target system which will run the installed code. Pip is meant to install packages for the currently running Python. James, you might want to find a clearer term for this, because it's really not about compilation.

The good news is that PEP 517 conceptually makes this easier. Most packages will be built to a platform-independent (-none-any) wheel, and then it's just a matter of extracting files from the wheel to the correct target locations, which can be done regardless of the build backend. The less good news is that the tooling to do that extraction (such as the installer package) is still fairly new and rough.

[pfmoore]

Thanks @takluyver - if that is the confusion here, that makes a lot more sense.

While the installer package is new, pip can extract wheels into (relatively) arbitrary locations, using options like --target, --root and --prefix. The problem is that these options likely have rough edges, not so much because they are new as because they are infrequently used, and we've had little or no feedback from people trying to use them to unpack wheels for installation on a different system. The distro packagers typically "do their own thing" and we don't get much feedback from them - I'd optimistically hope that means that pip is mostly sufficient for them, but I suspect the reality is that they do a fair bit of hacking of their own and don't work with us as much as we'd like.

IMO, installer is likely the best tool for this sort of unpacking in the long term, but it may well be that the use case of "installing into a different Python" is best served by a custom script using the installer API¹, rather than expecting a generic CLI tool to support this case. In the short term, using pip with its existing options (noted above) possibly with some post-processing to move things around if needed, is likely the best option.

But agreed, the existing tooling for this use case is still a bit rough.

Having a discussion on the Packaging Discourse might be the best way of collecting views from people (such as distro maintainers) with the same use case here, which would help tools design a more friendly UI.

Footnotes

1. Or a specialised tool for this use case that uses the installer API under the hood. ↩

[jameshilliard]

I think the term 'cross compilation' might be causing some confusion.

Well we're effectively using a full cross compilation setup(capable of building c extensions and such) even for pure python packages.

I think the people from the Python side (including me) assume this term refers to actual compilation, as in creating native machine code. This doesn't happen for Python code, so as far as we're concerned, it's a non-issue for pure Python packages, including all the packaging machinery we're talking about.

It's an issue for packages that have native extensions, which we of course have to support and need to be compiled, our infrastructure doesn't differentiate pure from native packages in any meaningful sense so the tooling needs to support both use cases either way even if the tools or some specific pages themselves may not require native extension. Creating separate platform-independent tooling here would just IMO just add unnecessary complexity. So I'm generally approaching the problems here from the point of view that platform-specific support is required.

Most packages will be built to a platform-independent (-none-any) wheel

This doesn't make much difference in terms of complexity since we will always have to support platform specific package builds+installation, it makes no sense to special case platform-independent packages in our infrastructure as the platform-specific infrastructure should handle those.

The problem is that these options likely have rough edges, not so much because they are new as because they are infrequently used, and we've had little or no feedback from people trying to use them to unpack wheels for installation on a different system.

Yeah, I mean for the pip case at least it always seemed that pip was deliberately not supporting cross compilation use cases, which is fine but that just means we need an alternative here.

The distro packagers typically "do their own thing" and we don't get much feedback from them - I'd optimistically hope that means that pip is mostly sufficient for them, but I suspect the reality is that they do a fair bit of hacking of their own and don't work with us as much as we'd like.

I'm trying to provide feedback so that these issues can actually get solved upstream properly, although historically the python situation has not been great there and we've had to hack stuff up a good bit. Cross compilation also tends to be significantly more complex than normal compilation for python although we do try and minimize the amount of customization we make as much as feasible.

In the short term, using pip with its existing options (noted above) possibly with some post-processing to move things around if needed, is likely the best option.

From my understanding this would probably be the most difficult option since pip has such poor/non-existent support for cross compilation and customization. It also doesn't give much control around the build and install stages in general AFAIU and would likely conflict with our download infrastructure and dependency resolution infrastructure as well probably.

[pfmoore]

From my understanding this would probably be the most difficult option since pip has such poor/non-existent support for cross compilation and customization.

Pip doesn't need to support cross-compilation (in the "build binaries for a different platform" sense) as all it does is call the build backend (setuptools or whatever). As I've said on a number of occasions (but the message doesn't seem to be getting picked up) you need to ask the setuptools project to expose options to request cross-compilation via the PEP 517 API (presumably via the config_options argument). Once it's possible for a PEP 517 client to actually ask setuptools to do a cross-platform build, then the worst case situation is that pip needs to improve how it exposes the config options in the UI (and it's possible that the existing interface is enough and you don't even need that).

But please be clear - you will never have a backend-independent means of requesting cross-compilation, unless you engage with the packaging community to define a standard for such a thing. Without a standard, you'll have to handle cross-compilation with backend-specific flags (which means knowing what build backend a given project uses).

[layday]

I'm confused, what has PEP 517 changed in terms of cross-compilation with setuptools? When has setuptools ever had options for cross-compilation? Why do we keep coming back to pip when...

Yeah, that's what I thought, which is why we need a PEP517 frontend that does support cross compilation(which seems to be build+installer).

[jameshilliard]

Pip doesn't need to support cross-compilation (in the "build binaries for a different platform" sense) as all it does is call the build backend (setuptools or whatever). As I've said on a number of occasions (but the message doesn't seem to be getting picked up) you need to ask the setuptools project to expose options to request cross-compilation via the PEP 517 API (presumably via the config_options argument).

Yeah, I just mean pip seems to be difficult in general to configure it in a cross toolchain since it does too much stuff automatically.

Once it's possible for a PEP 517 client to actually ask setuptools to do a cross-platform build, then the worst case situation is that pip needs to improve how it exposes the config options in the UI (and it's possible that the existing interface is enough and you don't even need that).

Well it seems like hacking up sysconfig gets us most of the way there with setuptools, at least when it comes to configuring c extensions and such.

But please be clear - you will never have a backend-independent means of requesting cross-compilation, unless you engage with the packaging community to define a standard for such a thing. Without a standard, you'll have to handle cross-compilation with backend-specific flags (which means knowing what build backend a given project uses).

Our tooling configures specific backends anyways and probably will always have to do that to some degree since we need to be able to resolve build dependencies, setuptools+distutils was pretty close to a universal backend for cross-compilation in the past at least(although supporting it was a bit hacky). The current situation with multiple build backends is much messier to maintain downstream.

I'm confused, what has PEP 517 changed in terms of cross-compilation with setuptools?

What's changed is that PEP 517 doesn't allow direct invocation of a build from a setup.py or equivalent I guess.

When has setuptools ever had options for cross-compilation?

It seems to have had enough configuration to make it work for us, see:
pkg-python-mk

[layday]

What's changed is that PEP 517 doesn't allow direct invocation of a build from a setup.py or equivalent I guess.

seems to have had enough configuration to make it work for us, see:
pkg-python-mk

So what has changed is that you can't pass --executable to build because setuptools doesn't properly support config_settings, which is tracked at pypa/setuptools#2491. What does setuptools do with the value of --executable? Because it's not used in build_ext as far as I can see.

[jameshilliard]

What does setuptools do with the value of --executable? Because it's not used in build_ext as far as I can see.

I think it's used for setting the shebang path for the scripts, AFAIU we hard code it to the target interpreter path in a running target system which would not match the path of the target interpreter while being built on the host.

[layday]

Well, that is fantastic news. setuptools does not write scripts under PEP 517; the installer does. And installer has had an option to change the interpreter in the shebang since day one :)