Determine what should be done about --(install|global)-option with Wheels

[dstufft]

Currently pip has --install-option and --global-option to make it possible to pass flags to the setup.py invocation. However these currently get ignored completely when installing from Wheel. This previously wasn't a big deal because commonly these options are only used for compiled packages and those often don't have Wheels on the platforms you might need these options on. However now in pip 7.0 via #2618 we now attempt to automatically build wheels for all projects. This means that, assuming we can build a wheel, these options are just completely ignored.

So what do we do with these options? Do we make their use imply --no-use-wheel? Do we attempt to convert them? Get rid of them?

[qwcode]

I had previously opened #1716 for this. I'll close that.

[qwcode]

the only extra tidbit from #1716 is just the awareness that install-options are recognized when placed in distutils config files.

[pfmoore]

The main use I've made of --install-option (actually --global-option) is the obscene hack:

pip wheel --global-option "--with-libyaml" --global-option "build_ext" --global-option "-DYAML_DECLARE_STATIC" --global-option "-LC:\Work\Scratch\pyyaml\yaml-0.1.5\win32\vs2008\Output\Release\lib" --global-option "-IC:\Work\Scratch\pyyaml\yaml-0.1.5\include"  .\PyYAML-3.10.tar.gz

This lets me add build options to the extension build (by injecting an explicit build_ext step into the setup command). The reason for doing this is to avoid having to download and edit the source distribution just to add some settings into setup.cfg.

Maybe having a means to add section.key=value items into the source archive's setup.cfg when doing the build would be a more general solution, that is explicitly a "build step" option.

Something like --build-setting [build_ext]define=YAML_DECLARE_STATIC, maybe? Then there's no implication that it has an effect when installing from wheel, it's clearly a build-only option.

(Whether this would be viable depends on what uses other people have for --install-option and --global-option...)

[xavfernandez]

According to the recently merged #2537 , people seem to have a need for these options.

An implied --no-use-wheel when --install-option or --global-option are used seems easy enough.

[rbtcollins]

So the problem (and its not unique to the wheel autobuilding) is scoping those options.

We can and should use the options from requirements file binding to pass through to the wheels. But the top level --install-option is only defined as applying to the thing being installed, isn't it? Or is it global?

Anyhow, if someone can define 'right' here, I'll happily write the code for it.

Remember too that folk can always use 'pip wheel' and then -f wheelhouse to get exactly what they want.

[dstufft]

Scoping is certainly a problem, and I've thought before that maybe we should remove those options and instead we should do something like --build-option:myproject="--bar" or something that handles the scoping.

There's also the question of what --install-option and --global-option even means in the future (tm) where we're trying to move to a situation where we only ever install from wheels and the only thing we use a source distribution for is building a wheel. In that scenario there is nothing to pass a --install-option too (and ideally anything people are using it for we'd either explicitly say we don't support, or we'd get a better, generic option for) and --global-option doesn't really make much sense because if you're only invoking a source distribution to build a wheel there is no global, only build.

On the other hand, we might not want to define the API for --build-option util the metabuild system is done so we can implement it in terms of that (e.g. if the API for metabuild is CLI based, we'd want to pass --foo options, if it's Pythonic "call this function" based, we'd want some way to translate CLI opts into Python opts).

In the short term it might just make the most sense that using --install-option and/or --global-option disables the automatic wheel cache and then open up a new issue to deprecate and replace those with a --build-option.

[rbtcollins]

@dstufft and I discussed this on IRC, he says that the resolution is: "--install-option, --global-option, --build-option will all disable the consumption of wheels, and any autobuilding of wheels." And that follow on work can be done to assess what we really want/need longer term.

[qwcode]

the question of what --install-option and --global-option even means in the future (tm)
where we're trying to move to a situation where we only ever install from wheels

why doesn't --install-option make sense when installing from wheels?
regardless of the format, you might want to set different paths for the install.

feels like a loss for wheel installs to have less flexibility than sdists.
(see https://github.com/twang817/pip/commit/e3338f7b087a9aba7e9033aa9078971fc33fc547 as an example of a pip user hacking this themselves, and wanting us to add this support)

how would this get sorted out in the "longer term"?

[dstufft]

Because --install-option, --global-option, and --build-option is conceptually "pass this option to the setup.py execution", it's documented that way and I'm sure people are using it that way. When we're installing from a Wheel we have no setup.py to execute and thus nothing to pass options to. Further more as we move to the meta build system we (most likely) aren't even going to be guaranteed to have a setup.py that exists for even a source distribution.

Longer term I think that the way to solve this is something like that:

• Figure out what people are using --install-option for that makes sense at all for wheels (one example I've seen is to be able to direct scripts to a different location) and expose a top level pip flag that supports that same use case. When installing from a source distribution we'll pass those options to setup.py and when installing from a wheel we'll take them into consideration for computing paths.
• Figure out what people are using --global-option for, and if it still makes sense in the new world order. I don't know off the top of my head what people are using this for, so I can't speak to what this would look like.
• Figure out the meta build system and what API is available for pip to call to pass options into the build system, once that is nailed down figure out a way that we can expose those build options to both pip wheel and pip install.

[dstufft]

In short, the basic problem with those options is that they are explicitly tied to executing setup.py and one of their use cases is for people being able to pass random arbitrary options to their setup.py invocations.

[qwcode]

these all seem like valid uses of install-options that could be applied to wheels.

 --home                               (Unix only) home directory to install
                                       under
  --install-base                       base installation directory (instead of
                                       --prefix or --home)
  --install-platbase                   base installation directory for
                                       platform-specific files (instead of --
                                       exec-prefix or --home)
  --root                               install everything relative to this
                                       alternate root directory
  --install-purelib                    installation directory for pure Python
                                       module distributions
  --install-platlib                    installation directory for non-pure
                                       module distributions
  --install-lib                        installation directory for all module
                                       distributions (overrides --install-
                                       purelib and --install-platlib)
  --install-headers                    installation directory for C/C++
                                       headers
  --install-scripts                    installation directory for Python
                                       scripts
  --install-data    

yes, these options are conceptually tied to setuptools, but don't have to be literally. we'd map them over to our distutils scheme utility.

for sure, it's weird to use setuptools options when there's no "setup.py" involved, but the reality is things are fractured right now, and maybe it's ok to break a conceptual line, to give people support for install flexibility in the mean time.

to be clear, I'm ok with the approach of ignoring them for wheels for now, but just making it clear, that we could do otherwise, if a lot of people wanted this.

[dstufft]

We already support --root, I'm not sure what --home does, and it seems logical to me to add the top level arguments there straight to pip install.

[qwcode]

yes, agreed, I'd like to see new top-level options that are setuptools-agnostic

[qwcode]

--home is for the "home scheme". it's similar to the user scheme, but free standing, and not on the python path by default

[pisymbol]

I too have now ran into this (or at least something similar).

I need 'wheel install-scripts' to be able to install to a custom path. Right now, it zonks the script wrapper always in '/usr/bin' without the ability to change the prefix path.

Is there really no way to have console scripts post wheel install to a custom path? I can certainly do this with eggs.

[JordonPhillips]

+1 to --install-scripts for wheels

[bsolomon1124]

The "Disabling all use of wheels due to the use of --build-options / --global-options / --install-options" when building a wheel is quite confusing. It probably should not exist at all when pip wheel is invoked.

[sbidoul]

The "Disabling all use of wheels due to the use of --build-options / --global-options / --install-options" when building a wheel is quite confusing. It probably should not exist at all when pip wheel is invoked.

I agree this is confusing. Actually --build-options / --global-options / --install-options implies --no-binary :all:. For pip wheel this means downloading and building everything from source. (and for pip install this additionally implies using the legacy setup.py install code path.)

[sbidoul]

I'm thinking about this in the context of #8102 (deprecating the setup.py install code path).

To progress with that deprecation, I think we need to decide between one of these two paths:

1. deprecating --install-option together with the setup.py install code path and asking users to switch to --build-option (and the setup.py bdist_wheel path).
2. deprecating all 3 --(global|install|build)-option, and moving foward with PEP517 config settings (Config settings support in PEP 517 #5771)

My current impression is that the road towards (2) is still quite long, and (1) could be a reasonable intermediary step with an alternative that we can support quite easily (i.e. replace --install-option with --build-option).

[pfmoore]

I agree. I think we should aim towards (2) - config settings are still relatively untried in practice, precisely because the toolchain doesn't support them yet - but I don't want improvements like #8102 to be blocked on it. So (1) as an immediate step sounds perfectly reasonable to me.

[davidtvs]

I think I just ran into this too. I'm trying to pass an argument to pip install that can be accessed in setup.py so that I can adjust the installation according to the parameter.
After googling a bit it seems like the way to do this is through --install-option or --global-option but now I see the warning UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option. and the installation fails completely because it can no longer find some packages.

I would definitely not expect this behavior and would expect install options to be easily accessible in setup.py

[TylerGubala]

I ran into a similar issue where I want to pass a prebuilt directory to pip wheel to avoid building a Python extension that I have already built from sources on the machine somewhere.

What is the best way? I don't think this is working:

"${PYBIN}/pip" wheel --global-option="--bpy-prebuilt=/build/linux/bin" /blenderpy -v -w wheelhouse/

[mzpqnxow]

I am also having (roughly) this issue- I want to set --build-option for a specific package in requirements.txt, but doing this completely disables wheels for ALL packages, which is not the behavior I want

The requirements.txt looks roughly like this

cryptography   --install-option="'CFLAGS=-I/opt/openssl-1.1.1k/include LDFLAGS=-L/opt/openssl-1.1.1k/lib'"
... some other crypto package requiring openssl-1.0.x ...
... other packages ...

The warning I see is:

venv/lib/python3.7/site-packages/pip/_internal/req/req_file.py:194: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options, line.opts)

@sbidoul @pfmoore, I'm having a hard time following this issue and the linked/referenced issues and can't quite figure out if this is. If you have a moment, I'm just looking for a quick answer for these (I think it could be a sentence):

1. Is this acknowledged as being undesired or unexpected behavior?
2. If so, when (roughly) might this be enhanced to work as desired?

There are plenty of ways for me to work around this, they're just clunkier than I would expect for what seems like a relatively simple (even if unusual) use-case. That said, I understand that if this is a non-trivial change due to the way the dependencies are handled then it won't be much of a priority and I will hack around it by installing the package with an additional (separate) command if possible

[pfmoore]

@mzpqnxow as the last comment on this issue is from 12 months ago, and there has been no activity since, I'll be honest and say that I don't recall a lot of the details. But in the interests of giving you the quick answer you asked for:

Is this acknowledged as being undesired or unexpected behavior?

No, it's not. --install-option is only relevant for an install method that we only retain for legacy projects, and we have no plans to continue with it, or to "fix" limitations in it.

If so, when (roughly) might this be enhanced to work as desired?

When someone gets round to it is the best answer I can give. This is a volunteer project and the issue has languished mostly ignored since 2015, so it's clearly not that important to anyone.

---

A longer answer follows with a bit more background, if you care.

If you were to switch to --build-option, my understanding is that we might make that work in order to enable us to progress with the work to drop the "direct install" legacy code. But all we'd likely do in that case is sort out the bare minimum to give people who currently rely on the direct install path, a short term workaround to let them continue working with the "install via wheels" approach. But even --build-option will ultimately be removed at some point.

For anything other than a stopgap fix, we need proper adoption of the PEP 517 "config settings" mechanisms for pip to communicate with the build backend. That's not in our hands - build backend projects like setuptools, flit, enscons, etc, need to start working on support for config settings before we can work out how pip can provide a UX for passing data to them.

[mzpqnxow]

@mzpqnxow as the last comment on this issue is from 12 months ago, and there has been no activity since, I'll be honest and say that I don't recall a lot of the details. But in the interests of giving you the quick answer you asked for:

Thanks for taking the time

No, it's not. --install-option is only relevant for an install method that we only retain for legacy projects, and we have no plans to continue with it, or to "fix" limitations in it.

Understood

When someone gets round to it is the best answer I can give. This is a volunteer project and the issue has languished mostly ignored since 2015, so it's clearly not that important to anyone.

Fair enough

A longer answer follows with a bit more background, if you care.

I do care, thanks again for the time, especially this bit with the background

If you were to switch to --build-option, my understanding is that we might make that work in order to enable us to progress with the work to drop the "direct install" legacy code. But all we'd likely do in that case is sort out the bare minimum to give people who currently rely on the direct install path, a short term workaround to let them continue working with the "install via wheels" approach. But even --build-option will ultimately be removed at some point.

For anything other than a stopgap fix, we need proper adoption of the PEP 517 "config settings" mechanisms for pip to communicate with the build backend. That's not in our hands - build backend projects like setuptools, flit, enscons, etc, need to start working on support for config settings before we can work out how pip can provide a UX for passing data to them.