Adding CIBW_FROM_PYPI option

[YannickJadoul]

First step towards addressing #93.

A nice extra is that I expect this will also work for a value my-pypi-package==0.42.1, building (or downloading, if the wheel already exists) that specific release.

Things to do, if this approach works:

• Does this need to be an environment variable option (as I've made it, for the moment), or the suggested --from-pypi=... argument?
• Document this feature in README.md (depending on the actual form the option will take).
• Not sure what's the best way to test this without actually using a project that's already been uploaded to PyPI. @astrofrog, maybe you could give it a shot, using this branch to build your project?
• The list of arguments to the different platform functions is getting quite long. What about introducing a collections.namedtuple (e.g., BuildOptions) to nicely wrap all the options and only pass one argument to linux.build/macos.build/windows.build?

EDIT: Sorry about initially pushing the commit to the master. Something went wrong, there.

[joerick]

I'm not 100% sure about adding this option, since we've discussed previously the possibility of moving off the pip wheel style of building wheels, to the python setup.py bdist_wheel method.

Once we have this feature that transition is going to be quite a bit harder :/

Added to that, I'm not sure I totally understand the workflow when running in Travis CI or Appveyor - since those tools are designed to run as a result of a git change, not manually or reacting to a change in PyPI 🤔

Right now, (unless someone can change my mind 😄) I'd suggest writing a script to download the tarball from pypi, expand, and then invoke cibuildwheel on the resulting project directory. (so mark this a wontfix)

[YannickJadoul]

@astrofrog Up to you to argue, here. I don't really mind either way. I just saw an easy way of implementing the requested feature in this PR.

[astrofrog]

moving off the pip wheel style of building wheels, to the python setup.py bdist_wheel method

Note that as far as I can tell, using setup.py to do various things is going to be deprecated in the long term in favor of using pip - see for example some of the replies here:

pypa/setuptools#1156

In particular the sentence in one of the replies Also, running setup.py directly will (after a long period of deprecation) be transitioned to implementation-defined behavior

So I'm not sure it makes sense to transition to setup.py from ``pip`.

Added to that, I'm not sure I totally understand the workflow when running in Travis CI or Appveyor - since those tools are designed to run as a result of a git change, not manually or reacting to a change in PyPI 🤔

I do something similar for conda packages - basically I have Travis and AppVeyor running as cron jobs, so that they can watch for changes in e.g. PyPI rather than react to changes in the git repo.

In any case, I'm fine with just doing the downloading from PyPI myself - this was just an idea in case it was useful for others too :)

[joerick]

Cool, thanks for the info! I didn't know that pip will eventually inherit those commands.

How do you run Travis/Appveyor as cron jobs? Just push empty commits to them every day or something?

If we were to add something like this to cibuildwheel (which could be useful), I'd like it to be implemented separately from the pip wheel <pypi_package> feature. So an explicit cibuildwheel --from-pypi my-project - then on getting that command, cibuildwheel would download the tarball, expand it into a temp dir and then run as normal on that. That would keep the feature localised to one bit of the code base. Just trying to keep things consistent.

(By the way, do we even need package_name any more? Maybe I'll file an issue to remove it.)

[astrofrog]

@joerick - Travis and AppVeyor both have options in the settings for setting up cron jobs, e.g. https://docs.travis-ci.com/user/cron-jobs/

I'll have a think about how cibuildwheel --from-pypi my-project could be implemented.

[joerick]

Oh, interesting! I'm starting to get it :)

…

Sent from my phone

On 16 Sep 2018, at 22:02, Thomas Robitaille ***@***.***> wrote: @joerick - Travis and AppVeyor both have options in the settings for setting up cron jobs, e.g. https://docs.travis-ci.com/user/cron-jobs/ I'll have a think about how cibuildwheel --from-pypi my-project could be implemented. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[YannickJadoul]

So, should we close this particular PR, then?

[joerick]

Yes, I think so. I agree on the potential usefulness, just not this implementation. If you're happy to, lets close the PR, but keep the issue to track the feature request.

[YannickJadoul]

Nope, no problem. Just saw the issue, thought of this approach, and wanted to test/propose it :-)