Repeatability: offline mirror

[Korijn]

The lockfile goes a long way towards reliable and repeatable builds. One reason that pipenv does not yet provide a 100% repeatability guarantee is its dependency on remote resources, mainly package repositories. If a package is removed from the online repository, pipenv will not be able to install from the lockfile! Yarn is listed as one of the inspirers of the pipenv project, and so I propose to implement their solution to this issue!

You can read about the “Offline Mirror” feature on their blog and in their docs:

https://yarnpkg.com/blog/2016/11/24/offline-mirror/

I think it should be possible for pipenv with relatively simple changes. The Pipfile can include a feature toggle and an option to specify the directory, by default a directory within the current directory. Pipenv can configure pip’s cache directory for the duration of the pipenv invocation to make it work!

The more complicated component of this feature is managing the cache folder. I think it should be managed by pipenv to always match the lockfile. That means it should only include the tarballa for the packages in the lockfile and nothing else. This means you can check in the folder into source control (perhaps using git-lfs if filesize is prohibitive), to reach the 100% repeatability goal!

What do you think of this proposal? Can we make it work? How can I help?

[kennethreitz]

I think bandersnatch is the more appropriate solution to this in Python land:

https://pypi.python.org/pypi/bandersnatch

[kennethreitz]

P.S. While new ideas are always welcome, we're having a hard time keeping up with the features we already have, so we're more likely to remove features than add new ones at this point.

[Korijn]

I hear you. There many projects that mirror or cache packages as a sort of proxy mirror, also eg Artifactory can do it. In particular I am not enthusiastic about maintaining that solution separately from my version control and toolchain.

Is there some way I could convince you, or otherwise write a plugin / parallel tool for pipenv?

[Korijn]

Perhaps a hook mechanism could be supported, allowing plugins to hook into eg the lockfile update process?

[kennethreitz]

setuptools entrypoints could be considered — maybe.

[gsemet]

Nexus and artifactory (I only know and used them) provided this feature at an enterprise level (nexus is free). For a personal level, the official pypi should provide this stability.

At really personal level, one can always build its own wheelhouse we all versionned package he wishes to backup

[kennethreitz]

Bandersnatch.

[Korijn]

I appreciate you all taking the time to respond and explain your views to me. If nothing can be done to change your minds, I will pursue the alternative of writing an external tool for this purpose. It would be nice to hook into pipenv, but otherwise I can work around that as well.

Preferably I would collaborate though. That’s why I opened the issue. Can I help out with implementing setuptools entrypoints? Do you know of any existing tools that have an extension mechanism based on setuptools entrypoints that I can look at to learn more?

[kennethreitz]

@Korijn pytest!

[Korijn]

Thanks! It seems easy enough to implement. I suppose the main challenge would be to define the right hooks and supporting them over time. Do you already have any insights to share on this?

Either way, I can have a look at the pipenv source code later and see what would be required in terms of hooks to support an offline mirror plugin. Are you open to me pursuing this further? Or should I back off. :)

[kennethreitz]

I'm open — but skeptical :)

[Korijn]

Hah! Can you share with me what aspects in particular you are skeptical of? It might lead to new insights on my part, potentially saving the both of us some time. If you are pressured for time already then don’t worry about it. :) I’ll get on it soon anyway then.

[kennethreitz]

Just, maintainability. I don't want more maintenance burden. It'd have to be bullet-proof.

[Korijn]

Got it. A strong argument for sure! I think in particular then I would need to keep backward and forward compatibility in mind.

[Korijn]

Looks like pluggy is a stable library that can handle setuptools loading and hooks quite nicely, with forward compatibility to boot. The fact that pytest and tox depend on it give me the impression that maintenance is already well taken care of. Are you open to taking on a dependency on it?

[uranusjr]

Bandersnatch + #2281 seems like a good solution to this. Any opinions?

[Korijn]

That could work, although its another service to maintain as opposed to a self-contained project-scope solution.