-
-
Notifications
You must be signed in to change notification settings - Fork 30.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
subprocess.Popen leaks file descriptors opened for DEVNULL or PIPE stdin/stdout/stderr arguments #87474
Comments
TL;DR: subprocess.Popen's handling of file descriptors opened for DEVNULL or PIPE inputs/outputs has serious problems, and it can be coerced into leaking file descriptors in several ways. This can cause issues related to resource exhaustion. # The basic problem As part of its setup, Popen.__init__() calls Popen._get_handles(), which looks at the given stdin/stdout/stderr arguments and returns a tuple of 6 file descriptors (on Windows, file handles) indicating how stdin/stdout/stderr should be redirected. However, these file descriptors aren't properly closed if exceptions occur in certain cases. # Variant 1: Bad argument errors (introduced in 3.9) The first variant of this bug is shockingly easy to reproduce (note that this only works on platforms with /proc/self/fd, like Linux): import os, subprocess
def show_fds():
for entry in os.scandir("/proc/self/fd"):
print(entry.name, "->", os.readlink(entry.path))
print("Before:")
show_fds()
try:
subprocess.Popen(["ls"], stdin=subprocess.PIPE, user=1.0)
except TypeError as e: # "User must be a string or an integer"
print(e)
print("After:")
show_fds() This produces something like:
The process never got launched (because of the invalid This happens because the code that validates the Variant 2: Error opening file descriptors (seems to have been around in
|
This fixes several ways file descriptors could be leaked from `subprocess.Popen` constructor during error conditions by opening them later and using a context manager "fds to close" registration scheme to ensure they get closed before returning. --------- Co-authored-by: Gregory P. Smith [Google] <[email protected]>
…GH-96351) This fixes several ways file descriptors could be leaked from `subprocess.Popen` constructor during error conditions by opening them later and using a context manager "fds to close" registration scheme to ensure they get closed before returning. --------- (cherry picked from commit 3a4c44b) Co-authored-by: cptpcrd <[email protected]> Co-authored-by: Gregory P. Smith [Google] <[email protected]>
) (#104563) gh-87474: Fix file descriptor leaks in subprocess.Popen (GH-96351) This fixes several ways file descriptors could be leaked from `subprocess.Popen` constructor during error conditions by opening them later and using a context manager "fds to close" registration scheme to ensure they get closed before returning. --------- (cherry picked from commit 3a4c44b) Co-authored-by: cptpcrd <[email protected]> Co-authored-by: Gregory P. Smith [Google] <[email protected]>
Thanks for all your work diagnosing and fixing these. It is a good cleanup! |
* main: (26 commits) pythonGH-101520: Move tracemalloc functionality into core, leaving interface in Modules. (python#104508) typing: Add more tests for TypeVar (python#104571) pythongh-104572: Improve error messages for invalid constructs in PEP 695 contexts (python#104573) typing: Use PEP 695 syntax in typing.py (python#104553) pythongh-102153: Start stripping C0 control and space chars in `urlsplit` (python#102508) pythongh-104469: Update README.txt for _testcapi (pythongh-104529) pythonGH-103092: isolate `_elementtree` (python#104561) pythongh-104050: Add typing to Argument Clinic converters (python#104547) pythonGH-103906: Remove immortal refcounting in the interpreter (pythonGH-103909) pythongh-87474: Fix file descriptor leaks in subprocess.Popen (python#96351) pythonGH-103092: isolate `pyexpat` (python#104506) pythongh-75367: Fix data descriptor detection in inspect.getattr_static (python#104517) pythongh-104050: Add more annotations to `Tools/clinic.py` (python#104544) pythongh-104555: Fix isinstance() and issubclass() for runtime-checkable protocols that use PEP 695 (python#104556) pythongh-103865: add monitoring support to LOAD_SUPER_ATTR (python#103866) CODEOWNERS: Assign new PEP 695 files to myself (python#104551) pythonGH-104510: Fix refleaks in `_io` base types (python#104516) pythongh-104539: Fix indentation error in logging.config.rst (python#104545) pythongh-104050: Don't star-import 'types' in Argument Clinic (python#104543) pythongh-104050: Add basic typing to CConverter in clinic.py (python#104538) ...
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
The text was updated successfully, but these errors were encountered: