pip 21.3+ in-tree-builds with setuptools, do not work with read-only source trees

[poggenhans]

Description

We are using pip as part of our build pipeline within a project containing a mix of languages. In order to keep everything clean and reproducible, we mount the source code read-only in a docker container and run the build steps there.

Even it's a mix of languages and tools, this approach used to work fine until now, when pip as the only of all the tools involved suddenly tried to touch the source tree during build.

I have read though the discussion in pypa/pip#7555 that led to this change, but I feel like the point that there might be good reasons not to touch the source directory didn't get enough attention:

• You want to keep source code and build artifacts separate
• You don't want to clutter your scm
• Files created within the docker container are usually owned by root. So in our setup, our source directory would be cluttered with files owned by root after the container exits
• You want to make sure that no build script accidentally modifies the source code at build time. This is a very important point especially in large projects where it's hard to find side-effects in build scripts

Expected behavior

All other major packaging tools I know offer an option to chose a build directory that is separate from the source dir. I would expect that pip works in a similar way.

pip version

21.3

Python version

3.10

OS

ubuntu

How to Reproduce

Create a python package, mount read-only into a docker container and try to let pip install it, e.g:

mkdir readonly
echo "from setuptools import setup\nsetup(name='hello', version='1.0.0')" > readonly/setup.py
docker run -ti --rm -v `pwd`/readonly:/src:ro python:3.10 /bin/bash -c "pip install --upgrade pip; pip install /src"
rm -rf readonly

Output

Building wheels for collected packages: hello
  Building wheel for hello (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/src/setup.py'"'"'; __file__='"'"'/src/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-gx0vxrjn
       cwd: /src/
  Complete output (8 lines):
  running bdist_wheel
  running build
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_egg_info
  running egg_info
  creating hello.egg-info
  error: could not create 'hello.egg-info': Read-only file system
  ----------------------------------------
  ERROR: Failed building wheel for hello
  Running setup.py clean for hello
Failed to build hello

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

See my comment here regarding a similar issue.

Essentially, the build backend (setuptools, in this case) is responsible for implementing the PEP 517 interface, and if it can't handle a read-only source directory, that's something you should take up with them initially. Yes, it is true that until pip switched to in-tree builds this wasn't something that users encountered, but there were plenty of downsides to out-of-tree builds, and until we switched there wasn't much incentive for build backends to consider these sorts of issue.

Note that currently, you can use --use-deprecated=out-of-tree-build to get the old behaviour back. While that will be removed at some point, it does provide a window for backends to address this sort of issue (as does the previous ability for users to opt into the new behaviour, but sadly I don't know if many people did that).

[uranusjr]

I think this needs to be transferred to the setuptools repo.

[poggenhans]

Wow, that's a quick response 👍 . I'm aware of the workaround to restore the old behavior, but as you guessed, I'm interested in a long-term solution.

I feel like it's a bit too easy to just "blame" it on setuptools as the backend. As a user, if I want to package something with an out-of-tree build, it would be great if pip could do it in a convenient way. If it can just delegate my will to setuptools, great, but if it can't, I would expect it to have some sort of workaround to make it work (as it used to). When this workaround has limitations, I have to live with them, but then it's the result of something I explicitly asked for.

[uranusjr]

It's not about putting blame on someone but how this can be resolved. pip does not know how a build tool is building a project and where it puts built stuff, so previously it copied the entire project to somewhere it knows would work. But that turns out to be terribly slow in a lot of cases, so we switched to not doing that (and iirc we did talked to some popular backends to make sure they are OK with this). Unless we switch back to the previous behaviour, ultimately the backend needs to do something to make sure it can build the project somehow. pip cannot solve this problem alone; if the backend needs help, we'll definitely help, but the backend needs to do something first so we can know what to do.

[pradyunsg]

I’m curious — did you not read the very-noisy warning that pip has printed for ~6 months about this upcoming change? I’m asking because it’s unclear to me how we can surface usecases like this before we make a behaviour change, and if there’s any suggestions you have on that front.

[pfmoore]

Also, to add a little more context here, this separation of concerns is at the very core of the design of PEP 517 which describes how "build frontends" like pip interact with "build backends" like setuptools.

The design is a practical compromise that establishes a set of conventions. It's absolutely possible (although protracted and fairly difficult) to change the interface via further standardisation, but I'd argue that until pip switched to in-tree builds, we'd not really been working in the spitit of PEP 517, and so we haven't given the current standard much of a chance yet.

Saying that this should be resolved by setuptools isn't a matter of "blaming" setuptools, it's just confirming that this is the separation of responsibility that we'd agreed some time ago, and so in the first instance that's what we should now be implementing. If setuptools hits difficulties in supporting this use case, we can explore how the standard can be modified to make it easier for everyone to handle it. But let's give them a chance to respond before assuming that this is a big issue.

[poggenhans]

@pradyunsg Well, I understand that this is hard to understand from a perspective where pip is tool used in your daily work. But when it's part of a much larger project, that message becomes a small line in a logfile of a build pipeline and will probably be never read as long as the pipeline is running fine. In that specific case, the pip call was even wrapped by another tool that suppressed its output. Perhaps writing the warning to stderr would have made a difference, but ultimately only a pipeline failure is a clear signal that something needs to be done. And it's kind of the job of a nightly build to detect such things. Actually I think there are many more people out there whose pipelines were also broken by that change, they just haven't tested the new pip version yet.

Don't get me wrong: I'm not complaining about the way this was executed. I personally think it was done excellently (first a warning with a flag for testing, remove it with a flag for backwards compability before remove it completely). I'm just questioning that removing the feature to have out-of-tree builds (or even deprecating it) is a good idea as long as there is no replacement.

Thank you all for your patient explanations of the setuptools and PEP 517 situation, but I think that even if setuptools were to offer such a feature in the future, I as a user cannot be expected to know whether a particular package that I want to install with pip uses setuptools or something else and if that thing supports out-of-tree builds or not. Only pip could help me there by offering me a pip install --but-please-dont-let-anybody-touch-my-source-tree-no-matter-the-cost to mitigate the limitations of setuptools and other possible backends.

[uranusjr]

But when it's part of a much larger project, that message becomes a small line in a logfile of a build pipeline and will probably be never read as long as the pipeline is running fine.

I'm just questioning that removing the feature to have out-of-tree builds (or even deprecating it) is a good idea as long as there is no replacement.

I think at least a part of the problem is these two are practically blocking each other. From pip's perspective, we don't know what a replacement would work because we don't what we need to fix in an abstract context, and nobody is coming forward with use cases unless we very loudly trigger people to. So I feel the current way is mostly a good way to go; we now know you problem, and can now work with build backends to come up with a solution, and you can use --use-deprecated=out-of-tree-build before then. With any luck, we'll have more people like you that are willing to talk to us, instead of sliently using --use-deprecated=out-of-tree-build and not complaining until when that breaks.

[poggenhans]

@uranusjr Ok, thank you. That was more or less what I wanted to achieve. In any case it would be great if --use-deprecated=out-of-tree-build was promoted to a --use-feature. Or at least if the deprecation warning was updated to make it clear that out-of-tree builds will live longer than 22.1 (because I doubt there will be a better solution to this problem so soon).

[pfmoore]

I'm fine with a longer deprecation period if we think it would help. But I think we should have some deadline, and stick (reasonably firmly) to it. This feels to me very like the dependency_links situation, where we dithered for years over pulling the plug on it, but basically nothing happened to address the problem while we waited. If we publicly set a deadline, maybe we can avoid that happening again.

Also, I note that setuptools is covered by tidelift - I don't know much about tidelift, but maybe that means that corporate users affected by this issue can subscribe and raise the visibility of this issue via that route. Or even provide some form of contribution direct to setuptools to help them work towards a solution. (I'm thinking specifically here of the fact that setuptools cannot build a project if that project's source tree is on read-only media).

[pradyunsg]

I’m pretty sure Tidelift only pushes for security updates.

[pradyunsg]

I'm going to go ahead and move this to setuptools, since that's where the change likely needs to happen.

[psigen]

https://pip.pypa.io/en/stable/topics/local-project-installs/#build-artifacts

Changed in version 22.1: The --use-deprecated=out-of-tree-build option has been removed.

Welp, the use-deprecated option is now deprecated. 😱

Is there any remaining workaround for the increasingly common use-case of wanting to pip install off a read-only mounted directory (like a network mounted FS or docker volume)? Currently I've just pinned the older pip and crossed my fingers that a solution is forthcoming 🤞.

[dhirschfeld]

So I've ended up here after googling the error message in my CI. My source is on a readonly filesytem, that's not changing. How can I use pip to install the package? Is my only option to pin to <22.1?!?

[psigen]

@dhirschfeld I'm not sure how to escalate this further.

I'm pretty sure that this is affecting many people using docker volume mounts in their CI pipelines, since the sane default is to use a read-only source volume from a build reproducibility and caching standpoint. I'm just hoping someone can help.

[pradyunsg]

There is one workaround that I've personally employed: copying over the source code to a read-write capable directory, and passing that to pip. Another is to stop using setuptools, switching to an alternative like Hatch or Flit, which IIRC will do the right thing in this case.

The fundamental problem here is that setuptools is trying to write to build/ in a read-only FS and, even though this was exposed by a change in pip, the fix needs to happen on setuptools.

[dhirschfeld]

even though this was exposed by a change in pip, the fix needs to happen on setuptools

I'm in the situation where I don't control the upstream package - I just want to install it. Maybe I could put in a PR to update their build to use hatch and they'd be better off for it but that's a change that'll take months and I just need to install their package now.

It would be nice if pip could provide an escape hatch so people could get on with what they need to rather than saying it's not their problem (even if that's true).

I've personally employed: copying over the source code to a read-write capable directory*

That's where I'm at now, but it seems incredibly wasteful to have to copy the entire source tree rather than e.g. specifying a writable output folder.

[pfmoore]

It would be nice if pip could provide an escape hatch

It's no easier for pip to make a change to provide an automated workaround for this than it is for the upstream package to modify their build. And any change to pip would only be a workaround and wouldn't address the underlying problem, so while I agree it would be nice if this problem would go away somehow, realistically it's better to ensure that the problem gets addressed at the source.

[psigen]

FYI: this issue is currently on the setuptools repo, it was transferred after a previous similar discussion.

Logistically, it is still unfortunate that in light of the issue, the previous workaround flag in pip was deprecated before a fix was found for setuptools. It would be nice to have a longer runway to fix the issue.

Does anyone know if the required change to do out-of-source builds is practical to implement in setuptools?

[pfmoore]

It's a little inaccurate to describe it as a "workaround flag". In reality, pip needed to change the build process to in-tree builds to handle a number of internal issues and blockages that were caused by the previous "copy and build" approach. The idea that installers should "trust the backend" to do the build has been around since the original introduction of PEP 517, 5 years ago. We kept out-of-tree builds for a long time, and then when we finally did remove them, we followed our normal deprecation process.

It's a shame that in all that time, setuptools didn't address the in-tree build issue, but they have a lot of legacy issues to address, and (as for most open source projects) very limited maintainer resource. So it's not surprising that they had other priorities. But I don't think that we can hold back development on pip indefinitely because of that, unfortunately.

Does anyone know if the required change to do out-of-source builds is practical to implement in setuptools?

Practical? Yes, I'm sure it is. Easy? Quite probably not. My understanding is that nothing is easy in setuptools, because people can do arbitrarily complex stuff in setup.py. I imagine migrating to a much more declarative UI (which is what setuptools has been working on for some time) is the best way to make out-of-source builds safe (in the majority of cases at least). Of course, if I'm right, then setuptools could end up allowing out-of-tree builds only if the project opts in, which means that you'd still need to get the project to update their build...

Sorry I can't offer a different short term solution, but copying the source code to a read-write directory is the short term solution, for what it's worth.

[markokr]

Stumbled on this issue, please consider fixing it.

Specifically, only project.egg-info/... writing seems problematic, rest of the build process respects various -b/-d/-t flags.

Also, it would be enough if at least one of the commands: sdist, install or bdist_wheel would work.

I would prefer prioritizing sdist as that should be the easiest, it's natural to continue rest of the build process on its result.