Support pip install . to include tool.poetry.dev-dependencies

[ace-han]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

It would be great if pip install . could pick up those dependencies under tool.poetry.dev-dependencies section.

The use case is to deploy a poetry managed app to a docker/CI environment without installing poetry, just plain pip install . in order to run uni-test.

[finswimmer]

Hello,

please use poetry export to create a requirements.txt which you can that use with pip.

fin swimmer

[ace-han]

@finswimmer Using requirements.txt means either maintaining another duplicate / unnecessary file in CVS system or extra before-script to export this requirements.txt.

Since PEP517, PEP518 are here already, it would be better that poetry-core reads pyproject.toml and installs tool.poetry.dev-dependencies by some flag or CLI option

[finswimmer]

Hello @ace-han,

the dev-dependencies are not part of the package itself, it's a feature provided by poetry to manages those dependencies as well. PEP 517/518 are describing an interface to build a package from source. Because dev-dependencies are not part of the package itself, there's nothing useful we can do here.

Why not installing poetry in your CI? It's getting installed anyway if you are doing a pip install ., because it's defined as the build-backend.

fin swimmer

[ace-han]

@finswimmer I've done some test, please refer to https://github.com/ace-han/composetest/blob/master/Dockerfile

In order to install poetry in docker/CI, there seems too many stuff to take care of, which are tricky.

It's way too complicated than just pip install .

# using pip
RUN pip install -r requirements.txt

VS

# using poetry
# (this way is hard, modifying `/etc/hosts`, installing curl/poetry are very tricky in docker image)
# (refer to https://github.com/snok/install-poetry/blob/main/action.yml)
RUN apk --no-cache add curl

# refer to https://github.com/hawtim/blog/issues/10 for `githubusercontent.com` issue
RUN echo '199.232.96.133 raw.githubusercontent.com' >> /etc/hosts \
&& curl -O -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py \
&& python get-poetry.py --yes \
&& rm get-poetry.py \
&& source $HOME/.poetry/env \
&& poetry config virtualenvs.create false \
&& poetry install --no-root

As I mentioned earlier, it would be better to support installing dev-dependencies via pip install . in a docker/CI environment, thx.

[sinoroc]

Some of us, in the larger Python packaging ecosystem make our dev dependencies optional with the extra feature. So that we can do python -m pip install .[dev] for example. I find it very practical.

[ace-han]

@sinoroc Interesting. Could you be more specific?

Maybe some demo code?

pyproject.toml, requirements.txt and the cli command to run python -m pip install .[dev]?

Thx in advance.

[sinoroc]

@ace-han Nothing particularly complicated about it. Instead of placing your development dependencies under poetry's dev-dependencies section, just make them optional dependencies under an extra named for example dev. See for example: #1941 (comment). The advantage is that it is easier to integrate with pip, tox and many other common Python packaging tools. But be aware that there are some drawbacks to this technique, for example poetry install won't install you development dependencies by default.

I wish maybe poetry would settle for a convention where if the project has extras named dev, dev_* then they are automatically installed in the development virtual environment.

[ace-han]

@sinoroc Great! Then this will be another choice to make pip install .-wise to install dev dependencies.

However, I think poetry could support this in some way would be much better.

Thx

[finswimmer]

Hello @ace-han,

# using poetry
# (this way is hard, modifying `/etc/hosts`, installing curl/poetry are very tricky in docker image)
# (refer to https://github.com/snok/install-poetry/blob/main/action.yml)
RUN apk --no-cache add curl

# refer to https://github.com/hawtim/blog/issues/10 for `githubusercontent.com` issue
RUN echo '199.232.96.133 raw.githubusercontent.com' >> /etc/hosts \
&& curl -O -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py \
&& python get-poetry.py --yes \
&& rm get-poetry.py \
&& source $HOME/.poetry/env \
&& poetry config virtualenvs.create false \
&& poetry install --no-root

as you don't need to take full advantage of working together with pyenv, you can use pipx to install poetry.

&& poetry install --no-root
I've seen this by other users here quite often and still don't get it. Why don't you want to install the root package but its dependencies? How can you than successfully run your unit tests?

Some of us, in the larger Python packaging ecosystem make our dev dependencies optional with the extra feature. So that we can do python -m pip install .[dev] for example. I find it very practical.

Yes this is practical and it works. But my understanding of "extras" is, that those package add additional features to the package itself. So using this for dev dependencies is a kind of misuse (due to the lack of better solutions until now).

I wish maybe poetry would settle for a convention where if the project has extras named dev, dev_* then they are automatically installed in the development virtual environment.

Sounds like a nice idea for a plugin (as soon as the plugin system is available)

However, I think poetry could support this in some way would be much better.

There is no clean technical way for it. While PEP 517 allows a way to pass arguments to the build backend, this would mean that one could build the same package with different dependencies (without dev-dependencies by default, with them with any kind of flag). Dynamic dependencies must be avoided in any case.

fin swimmer

[sinoroc]

Some of us, in the larger Python packaging ecosystem make our dev dependencies optional with the extra feature. So that we can do python -m pip install .[dev] for example. I find it very practical.

Yes this is practical and it works. But my understanding of "extras" is, that those package add additional features to the package itself. So using this for dev dependencies is a kind of misuse (due to the lack of better solutions until now).

Absolutely. It can be considered a misuse. I should have mentioned that. But a harmless one as far as I can tell. Never seen any issue of any kind with that pattern.

[ace-han]

@finswimmer

&& poetry install --no-root
I've seen this by other users here quite often and still don't get it. Why don't you want to install the root package but its dependencies? How can you than successfully run your unit tests?

When developing a business-oriented app (no need to publish to pipy), we just need to copy all the source file to the app server, install the deps, and then run python -m $LAUNCH_THE_APP only.

This is one of the cases to do poetry install --no-root

[sinoroc]

I don't think you should use poetry on the app server. I don't think poetry should be used for deployment. You probably should build your application as a wheel, and then copy all the wheels (app + all dependencies) to the app server, where you use pip to install (while skipping the index and any other network connection). That is what I would recommend.

[exhuma]

I've just come across this as well when doing some maintenance on our CI pipelines. And I'm in the same boat as @sinoroc and @ace-han . I have been using the dev extra in non-poetry packages. While also aware that this is a bit of a misuse as stated, this has served us very well over the past 10 years. We actually keep two of those extras in all of our projects. One for day-to-day development and one for testing. I wouldn't mind dropping one of those to have just a "dev" extra.

When working on the CI pipelines I am also faced with some cumbersome scripts to detect whether a project uses poetry or not as we have only migrated two projects for now to evaluate poetry (and to install poetry if needed)

With PEP 517 and PEP518 I was hoping I could finally just run a pip install .[dev] in my pipeline. It worked, but I forgot that poetry has its own handling of dev-dependencies and the extra dev does not exist in those projects. So none of the dev-dependencies were pulled in, making the unit-tests fail.

I could re-create the "extra" again in pyproject.toml but that's silly because 1) I could no longer do poetry add --dev ... (removing one major added-value from poetry), and 2) I would be very, very close to plain-old setup.py files.

In general - while this is a bit off-topic - this issue is yet another nail in the coffin for our team. And primarily due to the difficulties when running CI pipelines. I really enjoy the poetry CLI UX, but I don't think that alone should be a selling point.

[NightMachinery]

If I have poetry installed, what's the command to just install the dev deps into the current environment? I manage my envs with conda, and don't use poetry for that.

[MarlieChiller]

Just leaving a comment here to echo what others have said. Specifically:

It would be great if pip install . could pick up those dependencies under tool.poetry.dev-dependencies section.
The use case is to deploy a poetry managed app to a docker/CI environment without installing poetry, just plain pip install . in order to run unit-test.

The lack of this feature is preventing my teams adoption of this tool 😞

[exhuma]

After testing poetry for over a year, this issue along with the issue of having "editable" installs (as in pip install -e ...) made us go back to plain pip with setuptools. And while poetry seemed nice at first, I don't see any tangible reason why to use it other than that it looks nicer.

[dyens]

I found solution for me. It looks bad, but maybe it can be helpful for someone:

The idea is use some custom build.py script for extending dependencies with dev ones:
https://gist.github.com/dyens/a9ea6c3392ebc62865b2c308661cb68c

Then i can install dev deps from tool.poetry.group.dev.dependencies section:

pip install --config-settings dev=true .

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.