Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically or manually unload modules to free memory #1340

Closed
felixhao28 opened this issue Jun 3, 2019 · 9 comments
Closed

Automatically or manually unload modules to free memory #1340

felixhao28 opened this issue Jun 3, 2019 · 9 comments
Labels
database-index Needs a database index/Rewrite in Rust (#1059) feature

Comments

@felixhao28
Copy link

Hi I am using jedi in a service and it processes a lot of code a day. That leads to the issue of a very large memory consumption because all modules are loaded. Each jedi process takes 3Gb of memory and I have 4 such processes. My suggestion is can we have a way to unload less frequently used modules to keep the memory consumption in check?
@davidhalter
Copy link
Owner

Yes, I agree. I'm actually surprised that nobody has brought this up previously. I've known for a while that it's a problem, but a lot of people seem to be working with less modules (myself included).

If you want to change something, please consider working on parso's caching mechanism. There's probably the problem.

My personal take on it is that we probably have to get rid of "old" parsed modules every once in a while, but I don't really have time to do that at the moment. Jedi itself doesn't really do caching, it's just parso.

BTW: One of the issues with Python is that it doesn't really release memory, it just holds it so you can reuse it later.

@davidhalter davidhalter added the database-index Needs a database index/Rewrite in Rust (#1059) label Dec 22, 2019
@davidhalter
Copy link
Owner

Now instead of 3 GB, those processes should take probably like 600MB. I hope that's not too much. But in general, it's just hard to optimize with Python. Let me know if this is OK for you. Long term I might rewrite Jedi in Rust and at that point, it would definitely take less memory (and Rust can free memory as well).

@felixhao28
Copy link
Author

I installed 1TB of memory on our server as a "workaround". It is still good to know there is a software-based solution.

@davidhalter
Copy link
Owner

LOL. Can you please tell me what you are working on? I would be very interested, because that sounds like an extreme amount of Python files.

@felixhao28
Copy link
Author

py

A code completion service powered by deep learning. It used to be a web service. Now we are also releasing a version to run the service on client-side. It supports python and java for now and javascript is coming next.

@davidhalter
Copy link
Owner

So basically you're working for a competitor of kite.com? Or https://tabnine.com/?

@felixhao28
Copy link
Author

Exactly

@davidhalter
Copy link
Owner

Is there a product that one can try/look at?

@felixhao28
Copy link
Author

Here 👉 https://github.com/aixcoder-plugin/localservice

clrpackages pushed a commit to clearlinux-pkgs/parso that referenced this issue Jan 29, 2020
@fenrus75
parso: Autospec creation for update from version 0.5.2 to version 0.6.0
ee9ecc5 
Dave Halter (10):
      Make sure to limit the amount of cached files parso stores, fixes davidhalter/jedi#1340
      Remove Python 2.6 grammar
      Get rid of Python 3.3 artifacts
      Fix a Python 2.7 issue
      Add a Changelog note about dropping 2.6/3.3
      Forgot to increase the pickle version
      del_stmt is now considered a name definition
      Add a bit to the changelog
      Make sure iter_funcdefs includes async functions with decorators, fixes #98
      Bump version to 0.6.0

Hugo (4):
      Drop support for EOL Python 2.6
      Drop support for EOL Python 3.3
      Add python_requires to help pip
      Fix string/tuple concatenation

Jarry Shaw (2):
      Revision on assignment errors (#97)
      Revision on fstring issues (#100)

Tim Gates (1):
      Fix simple typo: utitilies -> utilities
blueyed added a commit to blueyed/jedi-vim that referenced this issue Jan 29, 2020
@blueyed
Update submodule pythonx/parso fb010f2...c864ca6
b6e74c9 
    * pythonx/parso fb010f2...c864ca6 (14):
      > Bump version to 0.6.0
      > Make sure iter_funcdefs includes async functions with decorators, fixes davidhalter#98
      > Add a bit to the changelog
      > del_stmt is now considered a name definition
      > Forgot to increase the pickle version
      > Revision on fstring issues (davidhalter#100)
      > Revision on assignment errors (davidhalter#97)
      > Add a Changelog note about dropping 2.6/3.3
      > Fix a Python 2.7 issue
      > Get rid of Python 3.3 artifacts
      > Remove Python 2.6 grammar
      > Merge branch 'rm-2.6' of https://github.com/hugovk/parso
      > Make sure to limit the amount of cached files parso stores, fixes davidhalter/jedi#1340
      > Fix simple typo: utitilies -> utilities
blueyed added a commit to blueyed/jedi-vim that referenced this issue Feb 3, 2020
@blueyed
Update submodule pythonx/parso fb010f2...c864ca6
41ecaa5 
    * pythonx/parso fb010f2...c864ca6 (14):
      > Bump version to 0.6.0
      > Make sure iter_funcdefs includes async functions with decorators, fixes davidhalter#98
      > Add a bit to the changelog
      > del_stmt is now considered a name definition
      > Forgot to increase the pickle version
      > Revision on fstring issues (davidhalter#100)
      > Revision on assignment errors (davidhalter#97)
      > Add a Changelog note about dropping 2.6/3.3
      > Fix a Python 2.7 issue
      > Get rid of Python 3.3 artifacts
      > Remove Python 2.6 grammar
      > Merge branch 'rm-2.6' of https://github.com/hugovk/parso
      > Make sure to limit the amount of cached files parso stores, fixes davidhalter/jedi#1340
      > Fix simple typo: utitilies -> utilities
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
database-index Needs a database index/Rewrite in Rust (#1059) feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@davidhalter @felixhao28