Ask the IPython kernel for editor code completions

[bcolsen]

So here's an idea that might have been shot down before, but I think some people could like it. In fact after thinking about #2162, it seems people might even expect this behavior.

My usual work-flow involves writing a script in the editor and running it in the same ipython kernel without resetting the name-space. After running my script, I make changes in the editor. I find myself often copying a line of code from the editor into ipython to get the code completion from there, because it knows more about some of objects(eg. pandas and matplotlib) then jedi.

So I thought as a last resort it would be cool if Spyder did this automatically.

Basic Idea

1. Look to jedi and rope to get the code completion when requested.
2. If there are no suggestions, then give that current line of code with the cursor position to the ipython client for completion requests. (Do we have access to this from Spyder?)

The Problem

The completions might be for the wrong object if if that name is different in the current kernel. This should be infrequent, but it might be worse than I think. I can see the bug reports already. However, I think people could recognize the occasional wrong completion in exchange for any completion at all.

What do you think?

[rlaverde]

Maybe this is related with some errors in the introspection plugin server (like #4410), I think that a better solution for completion problems is the language server implementation #4751

[ccordoba12]

@bcolsen, I think this solution would work only if users select the Run in dedicated console option. Else it'd create more confusion by not giving the right completions when users move from console to console to run their code.

[andfoy]

As @rlaverde, we will soon provide code completions via Language Server Protocol: #4751 (comment)

[gepcel]

As @bcolsen said:

I find myself often copying a line of code from the editor into ipython to get the code completion from there.

Me too. Me too. Me too. (I have to say it 3 times to agree.

For the time being, let's refer them as the editor autocompletion and the ipython autocompletion while I explain why this may be a good idea.

---

Who most likely need this feature, and why?

From the very first beginning I learned python, I found out a few things I needed to know are:

• Tools like ipython, spyder.
• Packages like numpy, scipy, matplotlib, pandas.

Most of the people using spyder are the people familiar with ipython, which are exactly the people wondering why the autocompletion works in ipython (console) but not spyder. And numpy, matplotlib, pandas are almost exactly the codes that fail in editor autocompletion but works in ipython completion.

And spyder defines itself as A Scientific Python Development Environment.

The most working scenarios for the people described above:

I know people use spyder and work with python in all different ways. For example, maybe expert programmers write hundreds of lines of codes, add breakpoints, run them, and debug. Maybe that's why so many people like pycharm and don't understand the popularity of ipython (I don't want to argue this with anyone whether I'm right or wrong). But there are also people analyzing data, who are not experts of python, who always need to look into the doc for names and parameters of many functions. They write a line of code or a cell of codes and execute them line by line or cell by cell (F9 or Shift+ Enter), see the result/plot, modify the code, and execute them again, and again.

It may be arguable that to make the ipython autocompletion work, the codes need to be executed in the kernel first, which is not the case with codes in editor.
But this may not be a problem in the scenarios described above. And again spyder defines itself as A Scientific Python Development Environment.

Also the so many Ctrl+I issues

Besides a lot of issues about "autocomplete works in console but not in the editor", there are also a lot of issues about "Ctrl+I fails in the editor but works in console".

Most of codes or packages related to those Ctrl+I issues are exactly those related with the autocompletion issues. Seems like this can kill two birds with one stone？

How about the risks and confusions about ipython autocompletion?

I can understand some of the risks like a variable may be changed before it needs to be changed, or a file is deleted before it needs to be deleted. And some confusions about the current console, the dedicated console and the external system terminal. And also confusions about the autocompletion may only work after previous codes execution.

But I think the risks and confusions may be overlooked. The people who need this feature, in the working scenarios which need this feature, mostly are familiar with these kinds of risks and confusions. It's just like working in ipython or notebook, right? With the benefits of an extra ipython console, a plot window, a variable explorer, a help window, a file explorer, a project explorer, outline window, debug functionality, et al. The fact that "Run the line or the selection", or "Run the cell" is widely used proves that. And also there's a "Run the current line or selection (Ctrl+Enter)" very popular in Rstudio.

Again, spyder defines itself as A Scientific Python Development Environment.

And the risks and confusions can be minimized

Maybe this feature can be disabled by default, and let people choose to enable it when they know what they are doing. By doing this, the risks and confusions can be minimized.

ipython autocompletion may not be the best solution but still be a good solution.

I found that it took many efforts and time to make jedi to work with compiled objects through docstrings with numpy-style (#868), and still unfinished about google-style docstrings (963). And spyder team puts lots of efforts migrating to use Language Server Protocol #4751, already merged.

With the spyder from the master branch (2019-04-01), I tested the following code in spyder editor, spyder ipython console, jupyter qtconsole, jupyterlab/notebook, visual studio code with python extension, atom with hydrogen extension. Seems like ipython autocompletion is still a good solution.

import matplotlib.pyplot as plt
ax = plt.subplot(111)
ax.plot([1,2,3])

---

So

I don't know the actual difficulty, but if it's possible, my suggestion would be:

• Let ipython autocompletion co-exists with the existing one.
• Maybe disabled by default to minimize the fisks and confusions.

[ccordoba12]

I don't know the actual difficulty

There are several:

1. About the actual implementation: there's no one-to-one association between consoles and files, as I clearly said in this Ask the IPython kernel for editor code completions #5252 (comment) above.

   So, the completions that would be useful for the variable a, generated by running the file A.py could be completely useless for the same variable a generated by running another B.py file in the same console (the first a could refer to Matplotlib figure and the second one to a dataframe).

   Also, if there are several consoles open at the same time, which one should we choose to get completions from? The first one, last one, the one with the current focus? This is simpler to solve for projects like RStudio or the (now defunct) Rodeo because they only have one console. So it's easy for them to ask that one console to give the completions it can gather for an object.

2. Another problem is that until the user reruns a file, the console would give the same completions as with the previous run. So let's suppose that variable a in A.py now holds a list in the editor, but in the last evaluation it was an int. Completion would not return append, index, etc, but the methods associated with int. Again, that would be terribly confusing for users.

Maybe disabled by default to minimize the fisks and confusions.

I know you're well intended and clearly want to help, but you don't understand what it means to run an open source project of this scale. Whatever solution we'd add to make this a reality (even an experimental one), we'd be forced to maintain it for years (two or three years, depending on the time we'd need to get a new major version) and to make it work as users expect.

This has happened to us several times already, so I don't want to waste my time supporting a half-done solution.

And spyder team puts lots of efforts migrating to use Language Server Protocol

In my mind, the LSP provides a real path to solve this the right way, and that's why we're putting so much effort on it. The problem is @andfoy (our LSP expert) is not working with us anymore because he's busy with his Master.

My plan is to build an LSP plugin with an associated IPython kernel in the background. This plugin will parse the file, detect all definitions (like ax = plt.subplot(111) in your example above), create a dummy object in the kernel (or reuse a previous one) and ask for completions to it.

Unfortunately, we won't have time to do this for Spyder 4 because we already have a ton of other things to do. But it's in our plans for next year.

[gepcel]

the completions that would be useful for the variable a, generated by running the file A.py could be completely useless for the same variable a generated by running another B.py file in the same console

if there are several consoles open at the same time, which one should we choose to get completions from? The first one, last one, the one with the current focus?

1. We can face the same problem when a=[1,2,3] in the first line and then a=None in the 300th line, in the same .py file. So just give what the kernel gives. 2. And choose the same consoles as the F9 does. It's just I'm using spyder like jupyter notebook to run a bunch of codes. And you are using spyder to manage a project with a bunch of files.

But this doesn't matter, because I get your points of not supporting a half-done solution. And I like your idea of build an LSP plugin with an associated IPython kernel in the background, sounds like a better solution than my suggestion.

Thank you for your response. I like spyder.

[ccordoba12]

Closing as a duplicate of issue #2162.