Wheelhouse Generation

[cpvandehey]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Hey Poetry Crew,

Given an enterprise python context, most devs are using private pypi registries to house/distribute their artifacts. In general, downloading artifacts from a custom registry requires a network configuration (VPN) and login credentials. While some companies have integrated their build/deploy pipelines with their custom registries, others are relying on the wheelhouse deployment approach (generate all the wheels locally and upload) to send their applications to the cloud.

Pip does support this with the pip wheel -i command. Poetry could wrap around pips functionality to ensure hashes that are stored in the lock file are used while generating a wheelhouse for deployment. I made this SO post describing ONE potential scenario where this feature would be very useful.

[abn]

@cpvandehey I am not entirely sure if this falls within the scope for poetry. If I am understanding you correct, you want all project dependencies downloaded into a local directory.

Wouldn't something like this be sufficient?

$ cat pyproject.toml 
[tool.poetry]
name = "foobar"
version = "0.1.0"
description = ""
authors = ["Some User <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
pycowsay = "*"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
$ poetry build -f wheel
Building foobar (0.1.0)
 - Building wheel
 - Built foobar-0.1.0-py3-none-any.whl
$ mkdir wheelhouse
$ cd wheelhouse/
$ pip download ../dist/foobar-0.1.0-py3-none-any.whl
Processing /tmp/foobar/dist/foobar-0.1.0-py3-none-any.whl
  Saved ./foobar-0.1.0-py3-none-any.whl
Collecting pycowsay
  Using cached pycowsay-0.0.0.1-py3-none-any.whl (3.3 kB)
  Saved ./pycowsay-0.0.0.1-py3-none-any.whl
Successfully downloaded foobar pycowsay
$ ls
foobar-0.1.0-py3-none-any.whl  pycowsay-0.0.0.1-py3-none-any.whl

Additional note, it might also make more sense to maybe support zipapp builds.

[cpvandehey]

This... This is stellar. Ill do some research on zipapp as well.

[mouse07410]

I think one problem remains: subdirectory dist where the created "wheels" are placed, isn't mentioned anywhere in the poetry Documentation (it may be obvious to some - definitely not to all).

I suggest that the doc for the build command is enhanced - add a sentence saying that sdist and wheel place the result of a successful build into the dist subdirectory of the project root dir.

[fredrikaverpil]

I have a bit of a problem where I would like poetry to download the wheels instead of pip. The reason being using this plugin/hack to allow SCM versioning.

One could argue that poetry can offer certain functionality through its upcoming plugin system, which would offer this capabilities and pip would simply bypass all such functionality.

[senpos]

@abn Hey there!

it might also make more sense to maybe support zipapp builds.

Is there an active issue for this feature or should we create one?

This sounds very convenient. zipapp itself is pretty easy to use, but it lacks some automation to bundle dependencies into it.
It would be a breathe to build those with a single command with dependencies defined in pyproject.toml / poetry.lock

[sinoroc]

@senpos You might want to read #3488 and associated tickets.

[fredrikaverpil]

About the original request; not only for poetry plugins (hacks), also credentials added for poetry won't be known by pip.

I at least would love to see this being supported by poetry, and to leverage the credentials and overall workings of poetry wrapping pip.

[sinoroc]

About the original request; not only for poetry plugins (hacks), also credentials added for poetry won't be known by pip.

I looked into this a bit, and both poetry and pip use keyring but use different keys. If they agreed on the naming of keys, that could have been solved easily. But again I only had a cursory look, it probably is not as simple as that.

[sontek]

Pip download isn't a solution because if the package doesn't ship wheels on pypi then it'll download a tarball. Make uvloop a dependency for example. You'll get something like: uvloop-0.14.0.tar.gz but we want the result of pip wheel uvloop

[sontek]

This is how I'm currently solving it, but it means not using poetry for my actual final production install. I just use it in development and to track the locking:

RUN poetry export -f requirements.txt -o requirements.txt --without-hashes
RUN pip wheel -w wheelhouse -r requirements.txt

[sinoroc]

but it means not using poetry for my actual final production install. I just use it in development and to track the locking

In my opinion, this is exactly what should be done. I would indeed recommend against using poetry to install on production.

Get all the wheels beforehand. Then on production system, something like:

python -m pip install --no-index --no-deps wheelhouse/*.whl

If possible, I would go one step futther and not even using (or installing) pip at all on the production system. This can be done with zipapp (as mentioned in this thread).

Prepare a zipapp before hand and then just run it on the production system, no need to install anything but the Python interpreter:

• https://docs.python.org/3/library/zipapp.html
• https://pypi.org/project/pex/
• https://pypi.org/project/shiv/
• https://pypi.org/project/zapp/ (my project, for very simple use cases)

[neersighted]

Closing as the download use case is better tracked in #2184 -- pip wheel already does a good job of building a wheelhouse folder using either a distfile or a requriements.txt

zipapp support should be tracked on the bundle plugin.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.