asyncio subprocesses allow pids to be reaped, different behavior than regular subprocesses

[JustinLebar]

BPO	40078
Nosy	@asvetlov, @1st1, @c-lewin

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-03-26.17:06:33.546>
labels = ['3.8', 'type-bug', '3.7', '3.9', 'expert-asyncio']
title = 'asyncio subprocesses allow pids to be reaped, different behavior than regular subprocesses'
updated_at = <Date 2020-04-19.03:14:10.157>
user = 'https://bugs.python.org/JustinLebar'

bugs.python.org fields:

activity = <Date 2020-04-19.03:14:10.157>
actor = 'c-lewin'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['asyncio']
creation = <Date 2020-03-26.17:06:33.546>
creator = 'Justin.Lebar'
dependencies = []
files = []
hgrepos = []
issue_num = 40078
keywords = []
message_count = 2.0
messages = ['365095', '366752']
nosy_count = 4.0
nosy_names = ['asvetlov', 'Justin.Lebar', 'yselivanov', 'c-lewin']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40078'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[JustinLebar]

From https://bugs.python.org/issue1187312 about regular subprocesses:

So as long as the application keeps a reference to the
subprocess object, it can wait for it; auto-reaping only
starts when the last reference was dropped [in Popen.__del__].

asyncio subprocesses seem to behave differently. When we notice the process has exited in BaseSubprocessTransport._process_exited, we call _try_finish(), which -- if all pipes are closed -- calls _call_connection_lost and sets self._proc to None.

At this point, my understanding is that once self._proc is GC'ed, we'll run Popen.__del__ and may reap the pid.

I would expect asyncio subprocesses to behave the same way as regular Popen objects wrt pid reaping.

[c-lewin]

Very first time engaging in such a forum. Apologies is advance if I am doing it wrong!

Observation: ps -ef shows "Defunct" process until calling script terminates

Scenario: equivalent test scripts in BASH, Python 2.7 and 3.6 that:

1. Start a ping
2. SIGTERM (kill -15) the associated PID
3. wait for a user input (hence stopping the script terminating)

I tried P.Open and threading but behaviour is same.

BASH script does not show any "defunct" process.

Is this "Child Reaping" the cause of this observed behaviour?

Problem comes when the Parent script is required to run constantly (server type scenario) as the "defunct" processes will presumably eventually consume all system resources?

[kumaraditya303]

Duplicate of #87744

This is basically same as #87744, both will be taken care of when rewriting subprocess handling in 3.14 without child watchers.