Contents of cryptography 0.5.2 source distribution on PyPI

[xolox]

Hi cryptography developers,

When I download the cryptography 0.5.2 source distribution from PyPI, unpack it and run the command python setup.py egg_info the setup.py script spontaneously compiles a bunch of C files:

peter@macbook> wget https://pypi.python.org/packages/source/c/cryptography/cryptography-0.5.2.tar.gz
peter@macbook> tar xf cryptography-0.5.2.tar.gz
peter@macbook> cd cryptography-0.5.2
peter@macbook> python setup.py egg_info
running egg_info
writing requirements to cryptography.egg-info/requires.txt
writing cryptography.egg-info/PKG-INFO
writing top-level names to cryptography.egg-info/top_level.txt
writing dependency_links to cryptography.egg-info/dependency_links.txt
running build_ext
building '_Cryptography_cffi_684bb40axf342507b' extension
creating /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/cryptography
creating /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat
creating /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives
creating /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.c -o /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.o -o /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.so
running build_ext
building '_Cryptography_cffi_8f86901cxc1767c5a' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.c -o /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.o -o /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.so
running build_ext
building '_Cryptography_cffi_79a5b0a3x3a8a382' extension
creating /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/cryptography
creating /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat
creating /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings
creating /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_79a5b0a3x3a8a382.c -o /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_79a5b0a3x3a8a382.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_79a5b0a3x3a8a382.o -lcrypto -lssl -o /home/peter/tmp/2014-07-11/cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_79a5b0a3x3a8a382.so
reading manifest file 'cryptography.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'docs/_build'
warning: no previously-included files matching '*' found under directory 'vectors'
writing manifest file 'cryptography.egg-info/SOURCES.txt'

There's two reasons I'm reporting this issue:

The first reason is that I've worked with hundreds of Python packages and I have never encountered one that ran a C compiler when python setup.py egg_info is invoked. My problem with this is that I wrote and use pip-accel and this behavior of compiling during the egg_info command slows down my deployments a lot. What happens is this:

• pip-accel is given a requirements file that includes cryptography==0.5.2 so it uses pip to download the source distribution archive (if not already in the download cache), unpack the source distribution archive and run python setup.py egg_info in the resulting directory (to gather information about the package).
• After this is done pip-accel receives the information from pip and realizes that the cryptography==0.5.2 package has previously been built so it re-uses the cached binary distribution to speed up my deployments.
• This process which is supposed to take less than a second instead takes 16 to 20 seconds because the C compiler is run by python setup.py egg_info.

The second reason is that all of the C files compiled by the python setup.py egg_info command are contained inside __pycache__ directories. These directory names make me wonder if maybe these files ended up in the source distribution archive unintentionally? In that case resolving my issue would be easy, basically just find -type d -name __pycache__ -exec rm -R {} \; and publish a new source distribution archive cryptography==0.5.3 to PyPI :-).

My suggestion: If the files shouldn't be there they should be removed. If the files should be there and should be compiled and installed, then please change your setup.py script so that the files are only compiled by commands like python setup.py build and python setup.py install.

One last thing: I tried to do my homework and searched through the existing issues and pull requests. I found some issues that seem related but I could find no definitive answer on whether these __pycache__/*.c files are really meant to be included in the source distribution (just search for __pycache__ in any of these issues):

• Cannot install on OS X (v0.2.1) #693
• Error installing 0.2.1 on Ubuntu 11.10 #712
• etc, there are 13 issues that match a GitHub search on __pycache__

Thanks for your time.

[alex]

Hmm. So the question is "Why does setup.py egg_info invoke setup.py build_ext?", build_ext is the thing that's actually doing the compilation. /cc @dstufft

[xolox]

@alex: Thanks for the quick reply. Yes, if setup.py build_ext is to blame for this then that's half of my question. The other half is if those __pycache__/*.c files are really meant to be included in the source disitribution releases, because it seems like a mistake and it's what causes the problem. And they're definitely included in the source distribution archive, they're not created locally:

peter@macbook> wget https://pypi.python.org/packages/source/c/cryptography/cryptography-0.5.2.tar.gz
peter@macbook> tar tf cryptography-0.5.2.tar.gz | grep __pycache__                            
cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/
cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_79a5b0a3x3a8a382.c
cryptography-0.5.2/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_d62b3d91x972e1c0b.c
cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/
cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_684bb40axf342507b.c
cryptography-0.5.2/cryptography/hazmat/primitives/__pycache__/_Cryptography_cffi_8f86901cxc1767c5a.c

If these were packaged unintentionally (seems like it but I might be missing something) then I would humbly suggest to start packaging from git archive output or an equivalent command (if git is not your upstream VCS). These files have apparently been included in the cryptography packages for a while, see the 13 issues I mentioned.

[reaperhulk]

Releases are done from a fresh checkout (we do not have any pycache directories in our tree), but just calling python setup.py sdist triggers cffi's build process so we end up packaging the generated C files. This is probably due to the same core issue that is causing the egg_info build.

[xolox]

@reaperhulk: Thanks for pointing that out, my hint was meant in a friendly way. Certainly good to hear you guys are already doing this!

If I would have to venture a guess I would point out setup_requires as the guilty one here. It has a lot of sharp edges and I would generally recommend to avoid it at almost all costs (however I don't know why you guys decided to start using it).

The setup_requires option basically breaks the (IMHO valid) assumption of most the Python world that setup.py should do as little work as possible until python setup.py build (or similar) is called. Right now it executes the C compiler even if you run python setup.py --version...

_{Disclaimer: Sorry if I'm stating the obvious here, just trying to get this issue fixed :-)}

[alex]

Yup, it's known that setup_requires breaks lots of stuff, unfortunately
there's not really any other way to acheive what we want (compiling CFFI
stuff at install time).

On Fri, Jul 11, 2014 at 7:52 AM, Peter Odding [email protected]
wrote:

@reaperhulk https://github.com/reaperhulk: Thanks for pointing that
out, my hint was meant in a friendly way. Certainly good to hear you guys
are already doing this!

If I would have to venture a guess I would point out setup_requires as
the guilty one here. It has a lot of sharp edges and I would generally
recommend to avoid it at almost all costs (however I don't know why you
guys decided to start using it).

The setup_requires option basically breaks the (IMHO valid) assumption of
most the Python world that setup.py should do as little work as possible
until python setup.py build (or similar) is called. Right now it executes
the C compiler even if you run python setup.py --version...

Disclaimer: Sorry if I'm stating the obvious here, just trying to get this
issue fixed :-)

—
Reply to this email directly or view it on GitHub
#1253 (comment).

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[dstufft]

Yes, setup_requires is special and it means "these projects must be installed in order to run this setup.py". Projects in a setup_requires could do any manner of thing that controls how a setup.py is handled. For instance the pbr uses setup_requires to install pbr, which inserts all of the metadata from a static setup.cfg instead of from a setup.py. So hopefully you can see that if the stuff in setup_requires weren't installed, then the fundamental meaning of the setup.py changes.

So we add into that cffi, which needs to be in a setup_requires in order to get the Extension definition that the setup.py needs in order to compile the extension module. So executing the cryptography setup.py (and the same holds true for basically any CFFI using project) mandates installing CFFI which involves compiling a C-extension on CPython.

However I do see that build_ext is also being ran and I'm not sure why that is the case. It might be required for some reason but without source diving setuptools and distutils it's hard to say why it's doing that.

[xolox]

Given that I'm working on a fix in pull request #1257 I'll go ahead and close this issue.