python setup.py egg_info is broken

[ayoshi]

This is a very common issue, that breaks an ability to move ta-lib to requirements.txt,
making tox builds unusable.

The problem is that is simple cases ( --help and egg_info ) setup.py shouldn't import any requirements, or build requirements.

 Cloning https://github.com/mrjbq7/ta-lib to ./.tox/py27/src/ta-lib
    Traceback (most recent call last):
      File "<string>", line 20, in <module>
      File "/Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/src/ta-lib/setup.py", line 3, in <module>
        from Cython.Distutils import build_ext
    ImportError: No module named Cython.Distutils
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

      File "<string>", line 20, in <module>

      File "/Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/src/ta-lib/setup.py", line 3, in <module>

        from Cython.Distutils import build_ext

    ImportError: No module named Cython.Distutils

    ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in /Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/src/ta-lib

Simplest fix is to move all imports into conditional block, example from :

https://github.com/Unidata/netcdf4-python/blob/master/setup.py

if any('--' + opt in sys.argv for opt in Distribution.display_option_names +
       ['help-commands', 'help']) or sys.argv[1] == 'egg_info':
pass
else:
# append numpy include dir.
import numpy
    inc_dirs.append(numpy.get_include())

[mrjbq7]

So the idea is to make dependencies (like cython and numpy) be only imported in the else case? This is for a fresh install to not require dependencies? I'm a little confused about the use-case.

[ayoshi]

No, this is to fix projects relying on systems like tox ( actually any build automation software or CI ), which use pip -r requirements.txt to install projects dependancies.

Any project you will want to build using ta_lib, if you put ta_lib in requirements.txt, you will not be able to install ta_lib.

The reason, is that pip internally uses python setup.py egg_info before package dependancies are installed, and that operation will fail, because setup.py will mistakenly will try to import Cython.Distutils, though no dependancies are installed yet!

To verify, try to run python setup.py --help or python setup.py egg_info in your source directory from any clean virtualenv, or using system python ( i.e. with no dependancies installed ). Both commands should succeed.

This is already fixed by most upstream packages ( numpy, etc. ) which used to have the same issue.
It had been extensively discussed here pypa/pip#25

[mrjbq7]

So I want to hide import errors in the case of --help or egg-info, but let them through in other cases because we actually need them to build stuff?

It seems kinda lame that pip leaks this build dependency problem to everyone's setup.py.

[mrjbq7]

Does this work for you?

53cbbab

[ayoshi]

This is more about the correct usage of setuptools - which are notoriously hard to use properly and are almost unusable.

Unfortunately no:

at minimum three use cases should work:

python setup.py (exit 0)
python setup.py --help ( exit 0 )
python setup.py egg_info (exit 0)

You would also need to add:

from setuptools import setup, find_packages

I apologize for not submitting proper pull request, I'm a bit constrained, let me know if you want me to fix it I will try to submit proper pull request later.

[ayoshi]

After looking at setup, it seems more work is required - here is better example, with cythonized extensions:

stan-dev/pystan@5634448

[mrjbq7]

I don't mind fixing it -- any idea how to get egg_info to work? It says error: invalid command 'egg_info'?

[ayoshi]

Yes,
just add

from setuptools import setup, find_packages

on top

[mrjbq7]

That makes egg_info work, but is there a way to do that using only distutils?

[ayoshi]

Unforutnately, no - I know, that's a mess - there is no clean way to do everything, that's why numpy does this for example:

if len(sys.argv) >= 2 and ('--help' in sys.argv[1:] or
        sys.argv[1] in ('--help-commands', 'egg_info', '--version',
                        'clean')):
    # Use setuptools for these commands (they don't work well or at all
    # with distutils).  For normal builds use distutils.
    try:
        from setuptools import setup
    except ImportError:
        from distutils.core import setup

I think that's probably the best way

[mrjbq7]

Ok, thanks @ayoshi, I will work on this today.

[mrjbq7]

Okay, pushed some fixes so it works on a fresh install without numpy and cython and all three of those setup.py commands exit with 0.

[ayoshi]

Sorry - there is still something missing - setup.py develop will not work properly, that will break installing from URL I think. It might be the cython requirement is missing in setupy.py .

 Running setup.py develop for ta-lib
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/src/ta-lib/setup.py", line 58, in <module>
        from Cython.Distutils import build_ext
    ImportError: No module named Cython.Distutils
    Complete output from command /Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/bin/python2.7 -c "import setuptools, tokenize; __file__='/Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/src/ta-lib/setup.py'; exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps:
    Traceback (most recent call last):

      File "<string>", line 1, in <module>

      File "/Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/src/ta-lib/setup.py", line 58, in <module>

        from Cython.Distutils import build_ext

    ImportError: No module named Cython.Distutils

    ----------------------------------------
    Command "/Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/bin/python2.7 -c "import setuptools, tokenize; __file__='/Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/src/ta-lib/setup.py'; exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps" failed with error code 1 in /Users/alexg/Dropbox/workspace/personal/oanda/fxgrunt/.tox/py27/src/ta-lib

[mrjbq7]

Hmm, this might be a good time to fix that - I've wanted ta-lib to not require cython to install unless you're making changes or compiling the .pyx files yourself.

[mrjbq7]

Okay, I think cython is now optional, if you wouldn't mind checking that it works well?

[ayoshi]

Thank you! It works!