OSError: [Errno 24] Too many open files

[maldil]

Thank Jedi for the amazing tool.
I am using Jedi to determine method invocations belongs to couple of libraries in python projects.

Basically what I do is,
First, I parsed the Python source code to get its AST.
tree = ast.parse(source)
From that AST, I could get the lineno and col_offset of each node.
When I needed the type information of an expression (associated with an AST node), I created a jedi.Script object using the source code and pointed it to the place of that expression:

script = jedi.Script(source, node.lineno, node.col_offset)

Using that script object, I can get the type information using goto_definitions():

defs = script.goto_definitions()

Basically, I do this for almost all the method invocations in the projects.

When I am processing a project with large number of line of code code or large number of files ,the OS kills the project as below saying too many open files.

Any suggestions to solve my problem. Should I close the opened files frequently, if then how ?

NOTE:

1. I am using python virtual environment on MacOS.
2. I believe the issue Jedi creating too many processes when VIRTUAL_ENV not set? #1242 is relevant to this issue. Even though the issue says it is fixed, I am experiencing the use on the version 0.13.3.
3. However, I haven't updated any environment variable as described in the issue Jedi creating too many processes when VIRTUAL_ENV not set? #1242

StackTrace

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xxxx/Documents/Python/VMS/65399782/lib/python3.7/site-packages/jedi/api/environment.py", line 74, in _get_subprocess
    info = self._subprocess._send(None, _get_info)
  File "/Users/xxxx/Documents/Python/VMS/65399782/lib/python3.7/site-packages/jedi/evaluate/compiled/subprocess/__init__.py", line 229, in _send
    pickle_dump(data, self._process.stdin, self._pickle_protocol)
  File "/Users/xxxx/Documents/Python/VMS/65399782/lib/python3.7/site-packages/jedi/cache.py", line 143, in wrapper
    result = method(self, *args, **kwargs)
  File "/Users/xxxx/Documents/Python/VMS/65399782/lib/python3.7/site-packages/jedi/evaluate/compiled/subprocess/__init__.py", line 177, in _process
    bufsize=-1
  File "/Users/xxxx/Documents/Python/VMS/65399782/lib/python3.7/site-packages/jedi/_compatibility.py", line 530, in __init__
    super(GeneralizedPopen, self).__init__(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 769, in __init__
    restore_signals, start_new_session)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1406, in _execute_child
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./MLLibFunctionCallDesityFinder.py", line 486, in <module>
    getNameOfAllMLPythonScripts(repo, pro,num)
  File "./MLLibFunctionCallDesityFinder.py", line 19, in getNameOfAllMLPythonScripts
    AllVisitors().visit(root)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 270, in generic_visit
    self.visit(item)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 270, in generic_visit
    self.visit(item)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 270, in generic_visit
    self.visit(item)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 272, in generic_visit
    self.visit(value)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "./MLLibFunctionCallDesityFinder.py", line 143, in visit_Call
    source = jedi.Script(f.read(), node.lineno, col_length)
  File "/Users/xxxx/Documents/Python/VMS/65399782/lib/python3.7/site-packages/jedi/api/__init__.py", line 108, in __init__
    project, environment=environment, script_path=self.path
  File "/Users/xxxx/Documents/Python/VMS/65399782/lib/python3.7/site-packages/jedi/evaluate/__init__.py", line 94, in __init__
    self.compiled_subprocess = environment.get_evaluator_subprocess(self)
  File "/Users/xxxx/Documents/Python/VMS/65399782/lib/python3.7/site-packages/jedi/api/environment.py", line 113, in get_evaluator_subprocess
    return EvaluatorSubprocess(evaluator, self._get_subprocess())
  File "/Users/xxxx/Documents/Python/VMS/65399782/lib/python3.7/site-packages/jedi/api/environment.py", line 79, in _get_subprocess
    exc))
jedi.api.environment.InvalidPythonEnvironment: Could not get version information for '/Users/xxxx/Documents/Python/VMS/65399782/bin/python3': OSError(24, 'Too many open files')
/Users/xxxx/Documents/Python/Pancakes_ML_2point7_set1/65399782/src/work_in_progress/run_dnn.py
File found: /Users/xxxx/Documents/Python/Pancakes_ML_2point7_set1/65399782/src/work_in_progress/run_dnn.py
/Users/xxxx/Documents/Python/Pancakes_ML_2point7_set1/65399782/src/work_in_progress/run_dnn.py

[davidhalter]

Jedi is not done for this. Sorry. If ever, #1059 is the solution. Not saying this will never be the case, but what you are doing is just not feasible at the moment.

You can work around it by recreating Script (or recreate it after a certan number of completions), but it will be even slower.

[maldil]

Thank you so much!

Actually the error is, the OS kills the process when it opens more files than the maximum allowed number. I am just thinking, isn't there a way to close the opened files from the operation jedi.Script(source, node.lineno, node.col_offset)

a pseudo code is below for understanding.

source = open(file)
For node in [every call sites in source]
    jedi.Script(source, node.lineno, node.col_offset)`
    defs = script.goto_definitions()

When the len( [every call site]) is too large the OS kills the process.

I am just wandering is there a way to close all the files as below?

source = open(file)
For node in [every call site]
    jedi.Script(source, node.lineno, node.col_offset)`
    defs = script.goto_definitions()
    jedi.close(). #CLOSE ALL OPENED FILES BY JEDI

Or any workaround highly appreciated.

[davidhalter]

As I said, this is not really supported. I meant to close before. My comment above still applies. You cannot reuse Script too much. It's not built for that at the moment.

[armansamimi]

same issue, I faced similar problem when I tried to upload many images ( more than 6000 ) via django.

[armansamimi]

same issue, I faced similar problem when I tried to upload many images ( more than 6000 ) via django.
what should I do guys ?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Asus\Anaconda3\lib\wsgiref\handlers.py", line 137, in run
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\contrib\staticfiles\handlers.py", line 66, in __call__
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\core\handlers\wsgi.py", line 146, in __call__
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\core\handlers\base.py", line 81, in get_response
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\core\handlers\exception.py", line 37, in inner
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\core\handlers\exception.py", line 87, in response_for_exception
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\core\handlers\exception.py", line 122, in handle_uncaught_exception
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\views\debug.py", line 94, in technical_500_response
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\views\debug.py", line 331, in get_traceback_html
  File "C:\Users\Asus\Anaconda3\lib\pathlib.py", line 1176, in open
  File "C:\Users\Asus\Anaconda3\lib\pathlib.py", line 1030, in _opener
OSError: [Errno 24] Too many open files: 'C:\\Users\\Asus\\Anaconda3\\lib\\site-packages\\django\\views\\templates\\technical_500.html'
[28/Oct/2020 10:44:30] "POST /train HTTP/1.1" 500 59
Exception ignored in: <function TemporaryFile.__del__ at 0x000001F16A1EE488>
Traceback (most recent call last):
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\core\files\temp.py", line 61, in __del__
  File "C:\Users\Asus\Anaconda3\lib\site-packages\django\core\files\temp.py", line 49, in close
AttributeError: 'TemporaryFile' object has no attribute 'close_called'

---