multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw)

[poxthegreat]

BPO	31804
Nosy	@terryjreedy, @pitrou, @applio, @poxthegreat
PRs	bpo-31804: multiprocessing calls flush on sys.stdout at exit even if … #5492 bpo-31804: Fix multiprocessing.Process with broken standard streams #6079 [3.7] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) #6080 [3.6] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) #6081
Files	process.py.patch: patch file with trivial bugfix

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-03-11.19:10:52.978>
created_at = <Date 2017-10-17.14:59:34.932>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw)'
updated_at = <Date 2018-03-11.19:10:52.977>
user = 'https://github.com/poxthegreat'

bugs.python.org fields:

activity = <Date 2018-03-11.19:10:52.977>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2018-03-11.19:10:52.978>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2017-10-17.14:59:34.932>
creator = 'Pox TheGreat'
dependencies = []
files = ['47224']
hgrepos = []
issue_num = 31804
keywords = ['patch']
message_count = 12.0
messages = ['304512', '304678', '309708', '309716', '309721', '309759', '309762', '309853', '313608', '313613', '313616', '313617']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'pitrou', 'davin', 'Pox TheGreat']
pr_nums = ['5492', '6079', '6080', '6081']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31804'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[poxthegreat]

If you start Python by pythonw then sys.stdout and sys.stderr are set to None. If you also use multiprocessing then when the child process finishes BaseProcess._bootstrap calls sys.stdout.flush() and sys.stderr.flush() finally. This causes the process return code to be not zero (it is 1).

[pitrou]

This looks to be a duplicate of https://bugs.python.org/issue28326

[poxthegreat]

Unfortunately this is NOT a duplicate of https://bugs.python.org/issue28326. That issue is about a closed output stream. In that case sys.stdout and sys.stderr are file like objects which have been closed.

This issue is about sys.stdout and sys.stderr being None! This is because pythonw was used not python.

[pitrou]

@pox, nevertheless, the fix committed in #4073 should also fix this issue. Do you disagree?

[terryjreedy]

Pox, please retest with either 3.6.4 or 3.7.0a3 (or .0a4 when released, soon). Both were released after the merge that should fix this.

[poxthegreat]

Retested it with a freshly installed 3.6.4 version. Used the following code to test:

import sys
import multiprocessing


def foo():
    return 'bar'


if __name__ == '__main__':
    proc = multiprocessing.Process(target=foo)
    proc.start()
    proc.join()
    with open('process_exit_code.txt', 'w') as f:
        f.write(sys.version)
        f.write('\nprocess exit code: ')
        f.write(str(proc.exitcode))

It is very important to run the script with pythonw, not just with python. This is the content of the resulting process_exit_code.txt file on my machine:
3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]
process exit code: 1

As it can be seen the problem was not fixed. The process exit code should be 0. By default the new multiprocessing process created uses the same interpreter as the creator process, so it uses pythonw too.

[pitrou]

I'm not using Windows, so I'm unable to test using pythonw. You'll have to provide a fix, or someone else will have to.

[poxthegreat]

I have already uploaded a patch file but it is not in the required format. Also I realize that most of the confusion was because I forgot to provide the OS version. Perhaps it would be good to have a separate field for that.

I will upload a patch as it is described in the developer guide.

[pitrou]

New changeset e756f66 by Antoine Pitrou in branch 'master':
bpo-31804: Fix multiprocessing.Process with broken standard streams (bpo-6079)
e756f66

[pitrou]

New changeset ff5d213 by Antoine Pitrou (Miss Islington (bot)) in branch '3.7':
bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) (GH-6080)
ff5d213

[pitrou]

New changeset 069b8d2 by Antoine Pitrou in branch '3.6':
[3.6] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) (GH-6081)
069b8d2

[pitrou]

This is all fixed in the Python 3 branches now. I won't bother with Python 2 as it has quite a different code structure and backporting would take too much of my time.

Pox TheGreat, thanks for reporting and the initial patch!