check-dependencies.py failing for pytz on Ubuntu

[obfuscurity]

A handful of requirements are being installed in /opt/graphite/lib/python2.7/site-packages rather than the default system location, when using pip install -r requirements.txt.

This is causing check-dependencies.py to fail for pytz unless PYTHONPATH is set to include the site-packages directory.

# ls -l /opt/graphite/lib/python2.7/site-packages/
total 44
drwxr-xr-x  7 root root 4096 Jan  2 00:15 gunicorn
drwxr-xr-x  2 root root 4096 Jan  2 00:15 gunicorn-19.1.1.dist-info
drwxr-xr-x  6 root root 4096 Jan  2 00:15 pygments
drwxr-xr-x  2 root root 4096 Jan  2 00:15 Pygments-2.0.1.dist-info
drwxr-xr-x  3 root root 4096 Jan  5 01:13 pytz
drwxr-xr-x  2 root root 4096 Jan  5 01:13 pytz-2014.10.dist-info
drwxr-xr-x 14 root root 4096 Jan  2 00:15 sphinx
drwxr-xr-x  2 root root 4096 Jan  2 00:15 Sphinx-1.2.3.dist-info
drwxr-xr-x  3 root root 4096 Jan  2 00:15 sphinx_rtd_theme
drwxr-xr-x  2 root root 4096 Jan  2 00:15 sphinx_rtd_theme-0.1.6.dist-info
drwxr-xr-x  5 root root 4096 Jan  2 00:15 tests

I'm not sure what the correct fix is here. I hate to just remove pytz from check-dependencies.py but I don't have any better workarounds. FWIW, none of the other packages listed above are in check-dependencies.py or they'd fail the same way.

[brutasse]

I followed the instructions as updated in #1080 and couldn't replicate this behavior. pytz ended up in the global site-packages as expected, as all other dependencies (/usr/local/lib/python2.7/dist-packages/ on an ubuntu 14.04 instance).

[obfuscurity]

@brutasse This was actually via source installation. Steps to reproduce on Ubuntu 14.10:

$ sudo su -
# apt-get update
# apt-get install -y libcairo2-dev libffi-dev pkg-config python-dev python-pip fontconfig apache2 libapache2-mod-wsgi git-core memcached
# cd /usr/local/src/
# git clone https://github.com/graphite-project/whisper.git
# git clone https://github.com/graphite-project/carbon.git
# git clone https://github.com/graphite-project/graphite-web.git
# cd whisper/
# git checkout 0.9.13-pre1
# python setup.py install
# cd ../carbon/
# git checkout 0.9.13-pre1
# pip install -r requirements.txt
# python setup.py install
# cd ../graphite-web/
# git checkout 0.9.13-pre1
# pip install -r requirements.txt
# python check-dependencies.py

[obfuscurity]

Reproduced on Ubuntu 14.04.

[bitprophet]

Recreated on an Ubuntu 14.10 Rackspace Cloud VPS:

• Whisper install: whisper ends up in /usr/local/lib/python2.7/dist-packages (not site-packages).
  • I forget when/why dist-packages vs site-packages is a thing offhand, but I know I have seen it occasionally in the past.
  • Installing whisper doesn't create /opt/graphite.
• Carbon requirements: Twisted does at least some C compilation, as does zope.interface; however they & the other pure-Python requirements all end up in /usr/local/lib as with Whisper.
• Carbon install: this is what creates /opt/graphite; it also appears to copy itself & a portion of Twisted into /opt/graphite/lib/ but not into any site-packages.
• Graphite requirements: this seems to be the primary culprit. Afterwards, things like Django and cairocffi end up in /usr/local/lib alongside their friends (again, this is always dist-packages not site-packages but I doubt this matters) but as noted above, gunicorn, pygments etc are in /opt/graphite/lib/python2.7/site-packages/.
  • Nothing in the pip output seems to explain the discrepancy, nor are any patterns jumping out, eg some sub-dependencies end up in both places, some things requiring C compilation end up in both places.

All output of my pip/python commands are in this gist: https://gist.github.com/bitprophet/794bb907dfa30460b08e

I'm wondering if perhaps this has something to do with individual packages' sensitivity to the local setup.cfg files (which specify prefix = /opt/graphite) but that's a long shot; I/we need to poke their individual setup.py files to check on this. It would also have likely affected some carbon deps, but those were all fine.

I also noticed that some of the /opt/graphite installed packages mention explicit /opt/graphite/bin/xxx paths in their RECORD metadata, so I thought perhaps it had to do with installing non-lib elements, but sadly no - a couple of these 'misplaced' packages lack any such elements.

Should also try to reproduce on a non Ubuntu platform to see if it's related to Ubuntu/Debian specific Python shenanigans, as I've run into that before. I have a CentOS 6.5 instance cooking now.

[obfuscurity]

@bitprophet Looks lik you're onto something. Tested with CentOS 7, unable to reproduce.

# ls -l /opt/graphite/lib/
total 8
drwxr-xr-x. 3 root root 4096 Jan  5 17:21 carbon
-rw-r--r--. 1 root root  291 Jan  5 17:21 carbon-0.9.13-py2.7.egg-info
drwxr-xr-x. 3 root root   20 Jan  5 17:21 twisted

# python check-dependencies.py
[WARNING]
Unable to import the 'ldap' module, do you have python-ldap installed for python 2.7.5?
Without python-ldap, you will not be able to use LDAP authentication in the graphite webapp.

All necessary dependencies are met.
1 optional dependencies not met. Please consider the warning messages before proceeding.

[obfuscurity]

Also unable to reproduce on Debian 7.7. This appears to be an Ubuntu-ism.

[bitprophet]

Yea I remember having wackadoodle Python issues on Ubuntu back when I opsed on it (which is now a few years ago).

Quickly peeped some setup.py files and both 'good' and 'bad' packages do things like require setuptools, so I'm largely out of ideas here and would probably just be Googling terms related to ubuntu, python and prefixes until something rang a bell ¯\_(ツ)_/¯

Comedy option: have the checker test for this specific case, either VERY specific ("if ubuntu -> look in /opt/graphite/<prefix> for some things"), or just "if not found in expected location, check some common trees inside /opt/graphite in case this issue is present".

[obfuscurity]

This is probably a stab in the dark, but perhaps @warsaw might have some insight into what's causing this issue.

[obfuscurity]

Unable to reproduce in Ubuntu 13.10. Whatever is causing this appears to have surfaced in Ubuntu 14.04.

[warsaw]

Just a quick note until I get a chance to try to reproduce this. The Debian wiki briefly describes why dist-packages instead of site-packages on Debuntu systems:

"dist-packages instead of site-packages. Third party Python software installed from Debian packages goes into dist-packages, not site-packages. This is to reduce conflict between the system Python, and any from-source Python build you might install manually."

https://wiki.debian.org/Python

It's completely expected that apt-get installed third party packages end up in /usr/lib/pythonX.Y/dist-packages and manually installed packages end up in /usr/local/lib/pythonX.Y/dist-packages. AFAIK, none of the standard Debuntu Python tools touch /opt.

[warsaw]

It's not at all surprising that carbon ends up installing in /opt, since the setup.cfg is explicitly telling it to do that. Having [install]prefix point to /opt/graphite and having install-lib variable are exactly equivalent to calling "python setup.py install --prefix=/opt/graphite --install-lib=/opt/graphite/lib"

Note that if you enable setuptools by doing "USE_SETUPTOOLS=1 python setup.py install" on Ubuntu 15.04 you'll get a nice warning from setuptools. stdlib distutils doesn't include this warning.

I tested the recipe above in Ubuntu 15.05 and Debian sid (unstable). Same behavior in both (i.e. installs to /opt/graphite).

Seems like everything's working as intended (where "intended" means "what the setup.cfg" is saying to do). :)

[obfuscurity]

@warsaw Thanks for the feedback. The problem isn't that Carbon is being installed to /opt/graphite/lib, but rather than random(?) dependencies are being installed into /opt/graphite/lib/python2.7/site-packages/ and the rest are going to the standard system location. See my original comment at the top.

[warsaw]

Right, sorry I missed that. I still think it's the setup.cfg that's messing you up. If I comment out the [install] section in carbon's setup.cfg, the pip install -r requirements.txt doesn't go into /opt. (Tested in Debian sid just to take Ubuntu out of the picture).

Sid's version of pip is 1.5.6 so my guess is that it applies setup.cfg to all requirements installed via -r.

Or IOW, I think it's pip's problem. :)

[brutasse]

That doesn't make sense… When you pip install -r something from a directory containing a setup.cfg file, I'm 99% sure pip doesn't take setup.cfg into account. @dstufft should be able to confirm.

@obfuscurity what's the pip version on the debian 7.7 you tried?

[obfuscurity]

vagrant@packer-debian-7:~$ pip --version
pip 1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

[dstufft]

I wonder if this is related to Wheels. If I remember correctly Ubuntu 14.04 has pip 1.5.x which will install Wheels by default and if I look at pytz on PyPI it has a wheel file. Does it still install into different locations if you do pip install -r requirements.txt --no-use-wheel?

[warsaw]

@dstufft Well that's interesting. Again, in Debian sid (which has pip 1.5.6), adding --no-use-wheel seems to avoid installing into /opt. Do you know why that would be?

[dstufft]

My guess is this code https://github.com/pypa/pip/blob/develop/pip/locations.py#L211-L259 in particular the use of the Distutils command is probably checking the current directory for setup.cfg. When not installing from a Wheel we cd into the build directory and out of the current directory.

[obfuscurity]

Adding --no-use-wheel is a workaround for Ubuntu 14.10. I'll test the others as well.

[obfuscurity]

Yeah, it looks like every distro with a pip that includes Wheels also works well with --no-use-wheel for our purposes. Others return a usage error.

Would it be reasonable to check for the specific version of pip that started using Wheels and base our logic around that?

[SEJeff]

Nope, just put the pip version in the requirements and I believe it will do the right thing. For several internal webapps at $real_job we deploy via wheels and have to do this.

[dstufft]

File a pip bug please.

On Jan 10, 2015, at 2:02 PM, Jeff Schroeder [email protected] wrote:

Nope, just put the pip version in the requirements and I believe it will do the right thing. For several internal webapps at $real_job we deploy via wheels and have to do this.

—
Reply to this email directly or view it on GitHub.

[obfuscurity]

@SEJeff Can you give an example how this would work?

[deniszh]

So, what's preffered fix? pinnin pip to <= 1.4.x (not sure how it will work when we have system pip version >= 1.5) or just documenting "--no-use-wheel" (IMO better for now) ?
@SEJeff ? @obfuscurity ? @dstufft ?

[brutasse]

+1 for documenting --no-use-wheel for now.

[SEJeff]

I agree with @brutasse here

[hggh]

@obfuscurity you can remove the check for pytz, because pytz is shipped with graphite itself in https://github.com/graphite-project/graphite-web/tree/0.9.x/webapp/graphite/thirdparty

[brutasse]

Fixed in #1233