gh-94026: Buffer regrtest worker stdout in temporary file (GH-94253)

[tiran]

Co-authored-by: Victor Stinner [email protected]

• Issue: Emscripten test worker sometimes fails witth "BlockingIOError: write could not complete without blocking" #94026

[tiran]

@vstinner are you ok with my patch?

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @tiran for commit df2f3fbb63b946f5e709262c37cfad76d7372158 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[tiran]

@vstinner please take a look. I would like to merge the PR today.

[vstinner]

The overall change LGTM. I added a few comments.

IMO it's ok to use a temporary file to worker process stdout and stderr. If we can any issue later, we can reconsider this choice later. But so far, buildbots are happy, so I'm happy as well :-)

[miss-islington]

Thanks @tiran for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

[miss-islington]

Sorry, @tiran, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 199ba233248ab279f445e0809c2077976f0711bc 3.11

[bedevere-bot]

GH-94408 is a backport of this pull request to the 3.11 branch.

[tim-one]

Since this was checked in, parallel regrtest runs (-j) on Windows always fail for me. They just die midstream, skipping loads of tests, but not in any reproducible place.

After a long time, I eventually figured out that this happens if, and only if, test_disutils is being run. Add -x test_distutils, and no problem. OTOH, running test_distutils by itself fails even with -j1:

C:\Code\Python\PCbuild>rt -q -j1 test_distutils

C:\Code\Python\PCbuild>"C:\Code\Python\PCbuild\amd64\python.exe"  -u -Wd -E -bb -m test  -j1 test_distutils
0:00:00 Run tests in parallel using 1 child processes
0:01:00 load avg: 0.00 running: test_distutils (60.0 sec)
Warning -- regrtest worker thread failed: Traceback (most recent call last):
Warning --   File "C:\Code\Python\Lib\test\libregrtest\runtest_mp.py", line 334, in run
Warning --     mp_result = self._runtest(test_name)
Warning --                 ^^^^^^^^^^^^^^^^^^^^^^^^
Warning --   File "C:\Code\Python\Lib\test\libregrtest\runtest_mp.py", line 293, in _runtest
Warning --     stdout = stdout_fh.read().strip()
Warning --              ^^^^^^^^^^^^^^^^
Warning --   File "C:\Code\Python\Lib\tempfile.py", line 483, in func_wrapper
Warning --     return func(*args, **kwargs)
Warning --            ^^^^^^^^^^^^^^^^^^^^^
Warning --   File "<frozen codecs>", line 322, in decode
Warning -- UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 3149: invalid start byte

== Tests result: NO TEST RUN ==

1 test omitted:
    test_distutils

Total duration: 1 min 17 sec
Tests result: NO TEST RUN

1 test omitted:
    test_distutils

Total duration: 55.8 sec
Tests result: NO TEST RUN

All the failures have "something like that" in them, although in context an irrelevant test usually appears to be at fault, because this unhandled exception causes all sorts of other crud to get skipped, and then Windows usually dies when teardown code is trying to delete a file that's still in use by some other test.

I don't understand what the code is trying to do, so just note that all visible problems go away if, when stdout_fh is created, the encoding is set to "latin1".

[vstinner]

Warning -- UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 3149: invalid start byte

Issue now tracked by #95027