pickling=True in save_module seems to slow down completions

[jogo]

Environment:

• python 2.7
• jedi from master branch (270f70e)
• test script:

import timeit

test_scripts = [
        'import matplotlib.',
        'from numpy import ',
        'from collections import ',
        'from io import ',
]

# timeit
for test_script in test_scripts:
    stmt = "x = jedi.Script('{0}').completions()".format(test_script)
    print stmt
    print "No Cache  : {0:.4f}".format(timeit.timeit(stmt, 'import jedi', number=1))
    print "With Cache: {0:.4f}".format(timeit.timeit(stmt, 'import jedi;{0}'.format(stmt), number=5)/5.0)

If I force pickling to False in save_module, getting recommendations with no cache is much faster.

change:

diff --git a/jedi/parser/cache.py b/jedi/parser/cache.py
index 182a8a47..fb6fdaf0 100644
--- a/jedi/parser/cache.py
+++ b/jedi/parser/cache.py
@@ -110,6 +110,7 @@ def save_module(grammar, path, module, lines, pickling=True):
         p_time = None
         pickling = False
 
+    pickling = False
     item = _NodeCacheItem(module, lines, p_time)
     parser_cache[path] = item
     if settings.use_filesystem_cache and pickling and path is not None:

Results from master:

x = jedi.Script('import matplotlib.').completions()
No Cache  : 0.3465
With Cache: 0.0035
x = jedi.Script('from numpy import ').completions()
No Cache  : 8.0198
With Cache: 0.1907
x = jedi.Script('from collections import ').completions()
No Cache  : 0.5308
With Cache: 0.0119
x = jedi.Script('from io import ').completions()
No Cache  : 0.0233
With Cache: 0.0022

Results with picking = False:

x = jedi.Script('import matplotlib.').completions()
No Cache  : 0.3344
With Cache: 0.0034
x = jedi.Script('from numpy import ').completions()
No Cache  : 2.5304
With Cache: 0.1995
x = jedi.Script('from collections import ').completions()
No Cache  : 0.1368
With Cache: 0.0120
x = jedi.Script('from io import ').completions()
No Cache  : 0.0089
With Cache: 0.0024

[davidhalter]

Can you test on the parso branch? You might need to install parso for that (pip install parso).

I have changed quite a few things about pickling (and moved the parser out of Jedi).

[jogo]

Hit a stacktrace

Commit: b9271cf

and I installed parso

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    import jedi
  File "jedi/jedi/__init__.py", line 41, in <module>
    from jedi.api import Script, Interpreter, NotFoundError, set_debug_function
  File "jedi/jedi/api/__init__.py", line 28, in <module>
    from jedi.api import helpers
  File "jedi/jedi/api/helpers.py", line 9, in <module>
    from parso.tokenize import tokenize
ImportError: cannot import name tokenize

[jogo]

I ran pip install git+https://github.com/davidhalter/parso.git and installed directly from git, that fixed the stacktrace, but imports on cold cache are still very slow.

x = jedi.Script('import matplotlib.').completions()
No Cache  : 0.6887
With Cache: 0.0036
x = jedi.Script('from numpy import ').completions()
No Cache  : 7.6750
With Cache: 0.1775
x = jedi.Script('from collections import ').completions()
No Cache  : 0.5176
With Cache: 0.0119
x = jedi.Script('from io import ').completions()
No Cache  : 0.0248
With Cache: 0.0022

[kbatbouta]

The Problem in my opinion

That even when you have a lot of ram Jedi won't use it to it's benefit rather is caches everything ( caching is important but doesn't always require you to drop data from memory ) , I have a MacBook Pro 15' 2017 and still with the blazing fast NVME ssd it's struggling to read the cash and also creating wear in the ssd due the constant I/O So...

why not create RamDisk as a temporally solution

it solve the problem with pickling read and write low performance and even use less cpu ( I know it's not perfect but consider this until Jedi is able handle cashing and memory usage reliably we don't have a choice )

So this is my setup for now:

This is only intended for MacOS , This is not for everyone. Feel free to create it for your OS.

the whole thing depend on your python environment and python usage habits ex: I use numpy , pandas , scikit-learn , scipy , TensorFlow and etc... ( all in one ENV ) but won't use it when using nltk because then the needed ram disk size will be huge.

CACHEDIR="/Volumes/RamDiskCache/$USER"

#
#  change  ram://2097152  to the amount of ram you want to allocate to Jedi cache 
#

DISK=`/usr/bin/hdiutil attach -nobrowse -nomount ram://2097152`
/usr/sbin/diskutil erasevolume HFS+ "RamDiskCache" $DISK

/bin/mkdir -pv $CACHEDIR/jedi
/bin/ln -v -s $CACHEDIR/jedi/ ~/Library/Caches/jedi

also this consume a lot of memory but to be honest I rather have a snappy coding experience with less multitasking than the other way around.

[ghost]

I think we should start using the JSON standard for serialization. Check this out: https://konstantin.blog/2010/pickle-vs-json-which-is-faster/. JSON is several orders of magnitude faster than Pickle. Best of all, it is human-readable! This is great for debugging purposes too.

[davidhalter]

Be my guest in implementing it with JSON. If you do, I just want to see the performance difference. :)

[balta2ar]

@kbatbouta Is pickling really the source of the problem? What's your environment/editor? Does using RAM make any noticeable difference in your case?

In fact, I'm having a similar issue. I'm using vscode + vscode-python and tensorflow 1.5. Completions for the root tensorflow module (import tensorflow as tf; tf.) take nearly 2 seconds every time the completion menu pops up. It's feels like in-memory cache is discarded every time, otherwise why would it be so slow, even though measurements from the first message of this issue show relatively low latency:

x = jedi.Script('from tensorflow import ').completions()
No Cache  : 0.6029
With Cache: 0.3839

[davidhalter]

I'm not sure. How new is your computer? SSD or HD?

0.38 is extremely slow for jedi. I consider this to be a very bad time performance. It's hard to get it better at the moment, but still there's a lot of room for improvements.

[balta2ar]

Well, it's few years old already (5-2500K @ 3.30GHz), but still it's not the slowest piece of hardware. Usually, ~/.cache/jedi is located on an SSD disk, but I've just made an experiment: I deleted jedi cache and symlinked ~/.cache/jedi into /dev/shm to make sure it's only in memory and does not hit the disk. The initial run was slow of course, but after cached were generated, it didn't make any difference, the numbers are pretty much the same. That's why I doubted the problem is with caches:

$ python perf2.py
x = jedi.Script('from tensorflow import ').completions()
No Cache  : 0.6097
With Cache: 0.3822

But that's just "synthetic" tests. I haven't checked it in VSCode yet (gotta do it in the evening at home).

I wonder, whether there are any low-hanging fruits in terms of jedi performance, such as a) running jedi (or parts of it) in pypy b) parallelizing completion (inference, analysis)? I'm not sure whether b) is trivial or even possible at all, though.

[davidhalter]

I have tried pypy, but it seems like it's slower than normal jedi (even after disabling pickling, which is what Armin Rigo told me could be an issue).

For parallelizing there's certainly possibilities, but I doubt I'll ever tackle it. It just doesn't make sense in Python for jedi. I guess one of my next projects is to create something like an "index" that caches most of these things.

This means that jedi would work a bit more like PyCharm in a way. The pre-calculated index would generally mean that you have very good speed. However until that is done, performance for tensorflow and numpy is probably suffering. Sorry for that, but it's really not easy to fix, because of the huge size of the code bases.

[jogo]

looking at the with cache tensorflow case and using cProfile, most of the time is spent in get_definition and get_defined_names. the time to deserialize the pickled object only impacts uncached times.

foo% sort time
foo% stats 10
Wed Jan 31 16:14:32 2018    foo

         8044995 function calls (7720781 primitive calls) in 8.188 seconds

   Ordered by: internal time
   List reduced from 283 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   762311    1.911    0.000    4.393    0.000 /usr/local/lib/python2.7/site-packages/parso/python/tree.py:198(get_definition)
   379918    1.049    0.000    2.045    0.000 /usr/local/lib/python2.7/site-packages/parso/python/tree.py:1005(get_defined_names)
683683/392766    0.765    0.000    0.861    0.000 /usr/local/lib/python2.7/site-packages/parso/python/tree.py:980(_defined_names)
    72358    0.525    0.000    6.680    0.000 /usr/local/lib/python2.7/site-packages/jedi/evaluate/filters.py:199(_filter)
   762311    0.489    0.000    5.803    0.000 /usr/local/lib/python2.7/site-packages/jedi/evaluate/filters.py:204(_is_name_reachable)
   188144    0.429    0.000    0.687    0.000 /usr/local/lib/python2.7/site-packages/jedi/parser_utils.py:230(get_parent_scope)
   762311    0.312    0.000    4.705    0.000 /usr/local/lib/python2.7/site-packages/parso/python/tree.py:192(is_definition)
   817542    0.243    0.000    0.243    0.000 /usr/local/lib/python2.7/site-packages/jedi/parser_utils.py:226(is_scope)
    26180    0.198    0.000    0.273    0.000 /usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py:329(normpath)
      605    0.143    0.000    0.143    0.000 {imp.find_module}

[davidhalter]

That is pretty much what I expected. I have tried to optimize these things before, but I think it's pretty hard without a lot of additional caching.

[jogo]

It looks like python3.6 is significantly faster for the with cache tensorflow case (and most other cases)

Python 3.6

x = jedi.Script('import matplotlib.').completions()
No Cache  : 0.0885
With Cache: 0.0049
x = jedi.Script('from numpy import ').completions()
No Cache  : 0.7122
With Cache: 0.1602
x = jedi.Script('from collections import ').completions()
No Cache  : 0.0622
With Cache: 0.0250
x = jedi.Script('from io import ').completions()
No Cache  : 0.0068
With Cache: 0.0038
x = jedi.Script('from tensorflow import ').completions()
No Cache  : 1.5363
With Cache: 0.3697

Python 2.7:

x = jedi.Script('import matplotlib.').completions()
No Cache  : 0.0816
With Cache: 0.0023
x = jedi.Script('from numpy import ').completions()
No Cache  : 1.8017
With Cache: 0.2706
x = jedi.Script('from collections import ').completions()
No Cache  : 0.1014
With Cache: 0.0161
x = jedi.Script('from io import ').completions()
No Cache  : 0.0085
With Cache: 0.0030
x = jedi.Script('from tensorflow import ').completions()
No Cache  : 3.5505
With Cache: 0.7843

[davidhalter]

IMO the issues with pickling=True have been fixed in parso. Please refer to #910 for numpy/matplotlib/tensorflow slowness.