No completions of the functions of SciPy, NumPy, Matplotlib, and PyLab

[mrjohannchang]

This issue is very similar to this.

I'm currently using Vim with YouCompleteMe. I found that Python's code completion went well except of the follow modules' functions: SciPy, NumPy, Matplotlib, and PyLab.

I traced the YouCompleteMe, finding it just pass the code completion job to Jedi, so then gave a quick try:

import jedi
print(jedi.Script('import numpy; a = numpy.array([0]); a.').completions())

Got output: '[]'.

But

import jedi
print(jedi.Script('import numpy; numpy.').completions())

went well.

I also found that IPython completes both 'numpy.' and 'a.' well.

There is a test unit for NumPy in the file 'test/completion/thirdparty/pylab_.py' around the line containing 'na.shape', so I think the root cause may be the wrong configuration on my machine. But I cannot find it out from the documents.

I'm using Vim with +python flag and YouCompleteMe plug-in on Ubuntu 13.10. I also installed SciPy by apt-get:

apt-get install python-numpy python-scipy python-matplotlb

And then upgrade them by pip (without virtualenv)

pip install --upgrdae numpy scipy matplotlib

[davidhalter]

Doesn't work for me either. Don't know what it is exactly, but it's probably related to the kind of complicated array function, which Jedi basically should understand.

[davidhalter]

I'm currently looking into making numpy completions fast. It's working pretty well, I've already got a speedup of >100ms (from 200ms originally). This is pretty cool.

However, completions of numpy are mostly bad, because a lot of it are builtin functions. Understanding the return value of <built-in function array> is pretty hard. We basically only have the docstring available, that hints us with this (and more):

Returns
-------
out : ndarray
    An array object satisfying the specified requirements.

While this could probably be enough to actually understand a lot of the numpy stuff, it creates the question of how many different docstring formats should we understand.

@asmeurer While sympy makes the impression to be pure Python, you seem to have a good understanding of the numeric/scientific Python community. What are the conventions in the most popular libraries (Scipy, Numpy, Matplotlib, ...) regarding builtin returns? Are there even conventions and if so, are they helpful in a completion context?
Would it be better to use something like "fake builtins" for numpy? We're doing this already for some builtins: https://github.com/davidhalter/jedi/blob/master/jedi/mixin/builtins.pym. This we would have maximal flexibility.

[asmeurer]

NumPy has a very standardized docstring format, which is used by many projects. See https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt. I would highly recommend just learning to understand it, because a lot of projects use it.

[asmeurer]

It might be worth creating a fake builtin at least for array. That is the principle datatype for numpy and scipy. Must functions input arrays and output arrays.

[spillz]

It might be worth creating a fake builtin at least for array. That is the principle datatype for numpy and scipy. Must functions input arrays and output arrays.

Makes me wonder if there shouldn't be some general purpose way to provide helpers as extensions to jedi to hint return types. As it is, jedi's inference can often fail or take a very long time to figure out what type a particular object is as it gets passed through several calls.

e.g.

import pandas
import numpy
sims = 100
df = pandas.to_csv("data.csv") #returns a DataFrame
gr = df.groupby(['year']) #returns a DataFrameGroupBy object
df0 = gr.agg(numpy.sum)/sims #returns a DataFrame
df0.

The type inference on df0 can take a very long time (up to 10s). Maybe it would be faster if there were some hints along the way.

[fedeinthemix]

The IPython Qt console appear to understand numpy, scipy and matplotlib. Maybe it would be instructive to look into how they did solve the problem.

[davidhalter]

The IPython Qt console appear to understand numpy, scipy and matplotlib. Maybe it would be instructive to look into how they did solve the problem.

It's very easy for IPython to understand it, because you can just do a dir() on any object in the REPL. It's way harder to do that without execution of code (which has already been done by the user at that point in IPython).
Even Python's readline.parse_and_bind("tab: complete") understands all of these. However, Jedi is of a different nature. Jedi does static analysis. That's way harder to get right. We need both. (And as a side note: Jedi's REPL completion also understands numpy and so forth. But in that case, Jedi's also using the Python objects that are available.)

[llcc]

@davidhalter I found some links about tricky temporary solutions in Eclipse. Please refer to them if it can give you some inspiration. I tried it in emacs without any luck, however hope that it would help to you.

http://stackoverflow.com/questions/11593472/code-completion-for-e-g-numpy-scipy-or-matplotlib-does-not-work-in-eclipse-pyd

and http://pydev.org/manual_adv_type_hints.html.

But I am sure that this is not a common issue, because I used it very well in Windows Eclipse. The problem came after I changed my pc.

[davidhalter]

@llcc What is not a common issue? Type hints work pretty well with Jedi (if you stick to the Jedi manual).

[immerrr]

FYI, you can get numpydoc return type hints with something like:

In [32]: numpydoc.docscrape.NumpyDocString(inspect.getdoc(np.interp))._parsed_data['Returns']
Out[32]: [('y', '{float, ndarray}', ['The interpolated values, same shape as `x`.'])]

[davidhalter]

@immerrr
Which is now a part of Jedi, thanks to your efforts!

[pe224]

With the latest version (as of today), there are still some problems with detecting numpy.
Here is a literal excerpt from a Python session.
There is some variation in responses to jedi.Script() calls.
Sometimes completion is working, sometimes not.
https://gist.github.com/pe224/195d69717028c97cadde

[Knusper]

any updates on this?

[davidhalter]

No.

[ronxin]

@davidhalter Does jedi currently support extracting the return types specified in numpydoc-style docs? @immerrr mentioned a solution at #372 (comment) , and you mentioned that this "is now a part of Jedi", but in https://github.com/davidhalter/jedi/blob/master/jedi/evaluate/docstrings.py (or https://github.com/davidhalter/jedi/blob/dev/jedi/evaluate/docstrings.py) I still don't see return type extraction out of numpydocs (there are only parameter extraction), and in real testing I found that jedi doesn't seem to be tracking the return types (e.g., of matplotlib). Am I missing something?

[davidhalter]

@roxin Maybe it's working in the linter branch?

[ronxin]

@davidhalter I didn't seem to find it in the linter branch either.

[tyalie]

Why are you just not running a python shell in the background. I know that sounds ambiguous, because there are many cases which could go wrong. But having instead a complete autocompletion is something I would really like. You could also give the option to activate / deactivate it or something like that.

[bcolsen]

Pull request #796 can help with this issue.

This doesn't work for numpy.array() or numpy.zeros() but it does for numpy.asarray()

Also plt.Figure() works. but the matplotlib docstrings often don't list returns. You can add them yourself though. It's handy for Figure().add_subplot to return Axes for example

[djangoliv]

I have a similar problem (but different I think)

print(jedi.Script('import numpy; numpy.a').completions())                                                                                                                                                                                
[<Completion: abs>, <Completion: absolute>, <Completion: add>, <Completion: add_newdocs>, <Completion: alen>, <Completion: all>, <Completion: allclose>, <Completion: ALLOW_THREADS>, <Completion: alltrue>, <Completion: allTypes>, <Complet
ion: alterdot>, <Completion: amax>, <Completion: amin>, <Completion: any>, <Completion: arange>, <Completion: arccos>, <Completion: arccosh>, <Completion: arcsin>, <Completion: arcsinh>, <Completion: arctan>, <Completion: arctan2>, <Comp
letion: arctanh>, <Completion: argmax>, <Completion: argmin>, <Completion: argsort>, <Completion: argwhere>, <Completion: around>, <Completion: array>, <Completion: array2string>, <Completion: array_equal>, <Completion: array_equiv>, <Co
mpletion: array_precision>, <Completion: array_repr>, <Completion: array_str>, <Completion: array_type>, <Completion: Arrayterator>, <Completion: as_strided>, <Completion: asanyarray>, <Completion: asarray>, <Completion: asbytes>, <Compl
etion: asbytes_nested>, <Completion: ascontiguousarray>, <Completion: asfarray>, <Completion: asfortranarray>, <Completion: asmatrix>, <Completion: asscalar>, <Completion: asstr>, <Completion: atleast_1d>, <Completion: atleast_2d>, <Comp
letion: atleast_3d>, <Completion: AxisConcatenator>]

doesn't contains average method.

Regards

[bcolsen]

I cannot reproduce this on Anaconda 4.2

print(jedi.Script('import`` numpy; numpy.a').completions())
[<Completion: abs>, <Completion: absolute>, <Completion: absolute_import>,
<Completion: add>, <Completion: add_docstring>, <Completion: add_newdoc>, 
<Completion: add_newdoc_ufunc>, <Completion: add_newdocs>, <Completion: alen>, 
<Completion: all>, <Completion: allclose>, <Completion: ALLOW_THREADS>, 
<Completion: alltrue>, <Completion: allTypes>, <Completion: alterdot>, 
<Completion: amax>, <Completion: amin>, <Completion: angle>, <Completion: any>, 
<Completion: append>, <Completion: apply_along_axis>, <Completion: apply_over_axes>, 
<Completion: arange>, <Completion: arccos>, <Completion: arccosh>, <Completion: arcsin>,
 <Completion: arcsinh>, <Completion: arctan>, <Completion: arctan2>, <Completion: arctanh>, 
<Completion: argmax>, <Completion: argmin>, <Completion: argpartition>, 
<Completion: argsort>, <Completion: argwhere>, <Completion: around>, <Completion: array>, 
<Completion: array2string>, <Completion: array_equal>, <Completion: array_equiv>, 
<Completion: array_precision>, <Completion: array_repr>, <Completion: array_split>, 
<Completion: array_str>, <Completion: array_type>, <Completion: Arrayterator>, 
<Completion: as_strided>, <Completion: asanyarray>, <Completion: asarray>, 
<Completion: asarray_chkfinite>, <Completion: asbytes>, <Completion: asbytes_nested>, 
<Completion: ascontiguousarray>, <Completion: asfarray>, <Completion: asfortranarray>, 
<Completion: asmatrix>, <Completion: asscalar>, <Completion: asstr>, 
<Completion: atleast_1d>, <Completion: atleast_2d>, <Completion: atleast_3d>, 
<Completion: average>, <Completion: AxisConcatenator>]

Versions
python 3.5

>>> import numpy
>>> numpy.__version__
'1.11.3'
>>> import jedi
>>> jedi.__version__
'0.9.0'

[oracleyue]

@bcolsen Yes! It seems that the bug has been fixed in newest jedi. I can properly auto-complete numpy.linalg.d etc. Seems no problem now.

[bcolsen]

@oracleyue this bug is about this code not completing:

import jedi
print(jedi.Script('import numpy; a = numpy.array([0]); a.').completions())

This still doesn't work. Howerver, this is fixed by applying PR: #796 and PR: #818

If anyone is interested this issue is patched for Spider-IDE 3.1

[oracleyue]

Regarding the bug proposed here: ndarray in numpy has no problem but numpy.array has, really strange.
If you use a = numpy.ndarry([0,1]); a., it works well. For array, no idea, but IPython works well.

Just brainstorm: is it possible to ask IPython developers to get the idea on how they do completions for numpy,scipy etc.?

---

you might ignore the following; I just suddenly want to mention it. I thought it has been fixed.

I found one more bug (It's really due to the strange organization of numpy scipy and their "inheritance" relations, I guess.)

import jedi
print(jedi.Script('import numpy; import scipy; import scipy.linalg; scipy.linalg.i').completions())

the output is empty.
However, it should give some completions, e.g. as in ipython:

import numpy
import scipy
import scipy.linalg

when you use scipy.linalg.i, you will get 3 candidates.

I say they are due to the "inheritance", since "scipy.linalg.inv()", for example, is actually first defined in numpy. When we use numpy.linalg.i, we will get completions. However, the tutorial from scipy teaches and recommend us to use like scipy.linalg.inv() instead of numpy.linalg.inv(). Really puzzled, they are supposed to have certain differences. Maybe I should read the source files.
The issue could similar the bug reported here. numpy might the similar trick to redefine array.

But, if you use import scipy.linalg as linalg; linalg.i, jedi can give correct completion. I put this bug before. Now I have kind of give up; no hope unless diving into the organizations of numpy, scipy. I just changed my way to import modules.

FYI,
IPython 3.1.0
Python 2.7.11
Scipy 0.18.0
Numpy 1.12.0

[bcolsen]

Those are some good workarounds, but you would have to wrap all array creation routines in np.ndarray to get completions like np.ndarray(np.zeroes(2,3))

>>> print(jedi.Script('import numpy; a = numpy.ndarray(numpy.zeros(10)); a.a').completions())
[<Completion: all>, <Completion: any>, <Completion: argmax>, <Completion: argmin>, <Completion: argpartition>, <Completion: argsort>, <Completion: astype>]
>>> print(jedi.Script('import numpy; a = numpy.zeros(10); a.a').completions())[]

but this is fixed by applying PR: #796 and PR: #818

IPython has it easy...You are constantly executing your code.
From davidhalter earlier in this thread

It's very easy for IPython to understand it, because you can just do a dir() on any object in the REPL. It's way harder to do that without execution of code (which has already been done by the user at that point in IPython).
Even Python's readline.parse_and_bind("tab: complete") understands all of these. However, Jedi is of a different nature. Jedi does static analysis. That's way harder to get right. We need both. (And as a side note: Jedi's REPL completion also understands numpy and so forth. But in that case, Jedi's also using the Python objects that are available.)

[davidhalter]

AFAIK the numpydoc fixes have been applied. You need to install numpydoc to actually have completions. Just sayin'. :)

If you still have an issues (I'm sure there are with this numpy beast), please open a new issue, this one is old and a lot of things have been fixed/changed.