[BUG] pip install not working while installing packages defined in setup.py

[Niladri24dutta]

setuptools version

setuptools==58.1.0

Python version

3.9.19

OS

Linux(Ubuntu 22.04)

Additional environment information

facing this issue while running my devops pipeline in Azure , it is using a Linux agent pool with Ubuntu 22.04 image.

Description

This issue is happening since today morning and our CI pipelines are failing because of this . We are trying to install python packages which are defined in our setup.py file using the below command -

python -m pip install wheel setuptools --index-url <some Url>
python -m pip install '.[testing_req]' --index-url <some Url>

"testing_req" is defined under extras_require argument of setup() . Here is how the setup.py file looks like -

 setup(
  name="some package",
    ...,
    install_requires=[
       list of python package to be installed
    ],
  extras_require={
        "testing_req": [list of python package to be installed for testing],
    }
)

I have observed that if I use the above pip install command it is only installing the packages which are present under "install_requires" but it is ignoring all the packages which are mentioned under "extras_require" . It should also install the additional packages apart from the required packages. Currently as a workaround I had to provide the path to the requirements.txt file to install the packages.
**Please find the versions used below

• Python version 3.9.19
• pip version 23.0.1
• wheel version 0.44.0/ 0.43.0 (tested with both same issue)
• setuptools version 72.1.0**

Expected behavior

While using the below command -

python -m pip install '.[testing_req]' --index-url <some Url>

It should also install the additional packages apart from the required packages. But it is only installing the packages which are present under "install_requires" but it is ignoring all the packages which are mentioned under "extras_require"

How to Reproduce

1. Create setup.py file using the above structure mentioned.
2. Run the below commands to install wheel,setuptools and the python packages -

python -m pip install wheel setuptools --index-url <some Url>
python -m pip install '.[testing_req]' --index-url <some Url>

Output

Observe that additional packages mentioned under "extras_require" are not installed . Only the packages under "install_requires" are getting installed.

[abravalheri]

Hi @Niladri24dutta, please note that v58.1.0 is quite old and therefore does not receive active support. Could you please try the newest version (of both setuptools and pip)?

I also recommend using pip install --use-pep517 to force pip packages to handle old packages using the upcoming non-deprecated default mechanism.

[Niladri24dutta]

@abravalheri there are some constraints in upgrading the version of pip to 24 and setuptools to latest version as it is not managed by me. As this was working till last friday, I am curious to know has anything changed recently which is causing this issue.

[abravalheri]

Hi @Niladri24dutta, you can start by checking which versions of which packages installed in your environment have changed since Friday. Maybe that helps.

Since you have constraints that you can not update and the version of setuptools that you are using is no longer officially supported, I will go ahead and close this issue.

[Niladri24dutta]

@abravalheri I am able to reproduce this issue locally with latest version of setuptools as well, and I also checked that no other package versions were upgraded after friday except the wheel . But I have tried with both latest and previous version of wheel it is the same issue.

I used the below version
pip : 23.0.1
setuptools: 72.1.0

[abravalheri]

I am able to reproduce this issue locally with latest version of setuptools as well, and I also checked that no other package versions were upgraded after friday except the wheel . But I have tried with both latest and previous version of wheel it is the same issue.

Thank you very much for confirming @Niladri24dutta.

Could you please provide a complete minimal reproducer including a "mock" version of the package that you are trying to install as a source code? Please also make sure to use either a brand new virtual environment and/or docker container to avoid external factors influencing in the reproducer. (This is necessary to investigate the problem further).

Can you also describe what happens when you add the --use-pep517 flag together with pip install?

no other package versions were upgraded after friday except the wheel

wheel is pretty important, I would not discard a possibility it is involved in the problem, it might be worthy to investigate more.

[Niladri24dutta]

@abravalheri No change even after using --use-pep517 flag together with pip install.
I just noticed that there was a new version released for "wheel" yesterday which the below changes -

 Canonicalized requirements in METADATA file
  Deprecated the bdist_wheel module, as the code was migrated to setuptools itself

Here is the release https://github.com/pypa/wheel/releases/tag/0.44.0 . Could there be any conflict due to this?

[abravalheri]

When you say you tested against the latest version of setuptools do you mean you wrote setuptools>="72.2.0" in the [build-system] requires session of pyproject.toml? I think that is the only way to guarantee that pip is actually using the latest version of setuptools and not something different...

Here is the release https://github.com/pypa/wheel/releases/tag/0.44.0 . Could there be any conflict due to this?

There is a report of lack of compatibility in pypa/wheel#631 (comment).

To investigate this more, I believe it would be important to have a complete minimal reproducible example.

[Niladri24dutta]

@abravalheri I have figured out a work around for this . I am not sure if this is expected way of installing the extra packages or not in the new version. This could also be due to the latest version of wheel (0.44.0).
Here is what worked for me -
Instead of using this command
python -m pip install '.[testing_req]' --index-url <some Url>
It works if I use this command
python -m pip install '<my package name in setup.py>[testing_req]' --index-url <some Url>
So it works if I use the package name instead of only . . Can we keep this issue open to see if others are also facing the same issue? may be this work around will be useful until we know the root cause.

[abravalheri]

Thanks for the feedback @Niladri24dutta.

Can we keep this issue open to see if others are also facing the same issue? may be this work around will be useful until we know the root cause.

We can, if we have a complete reproducer for the problem that help us to identify its root cause and it is related to setuptools. Without a reproducer or indication of a clear connection between setuptools and the problem I think this issue should be closed. Users will still be able find it by searching even if the issue is closed.

For now I will close the issue, if anyone has a complete reproducer, please feel free to post here and I will re-open the issue.