Pass --fail-rerun option on the main branch

[vstinner]

• Treat regrtest exit code 5 (EXITCODE_RERUN_FAIL) as WARNINGS, rather than FAILURE.
• Rename ONLY_MASTER_BRANCH to ONLY_MAIN_BRANCH.
• Add MAIN_BRANCH_NAME and JUNIT_FILENAME constants.
• Always pass the branch to Test() steps.
• Test steps use lists for test options, rather than strings.
• Remove unused SlowSharedUnixBuild and FreezeBuild factories.

[vstinner]

I'm waiting for python/cpython#108896 to be merged.

cc @zware

[vstinner]

I'm waiting for python/cpython#108896 to be merged.

I merged my PR. This PR is now ready for review :-)

[vstinner]

I restarted the server manually:

$ ps aux|grep buildbot.tac
buildbot 2533539 50.9  5.8 1285740 119816 ?      Sl   08:45   2:03 /srv/buildbot/venv/bin/python3.9 -c from twisted.scripts import twistd; twistd.run() --no_save --logfile=twistd.log --python=buildbot.tac

I'm now curious if the cron job will kill this process to spawn a new one.

[vstinner]

I'm now curious if the cron job will kill this process to spawn a new one.

15 min later, it didn't:

$ ps aux|grep buildbot.tac
buildbot 2533539 13.4 15.5 1452288 316140 ?      Sl   08:45   3:53 /srv/buildbot/venv/bin/python3.9 -c from twisted.scripts import twistd; twistd.run() --no_save --logfile=twistd.log --python=buildbot.tac

[vstinner]

I scheduled a job on FreeBSD on the main branch and it worked successfully! https://buildbot.python.org/all/#/builders/1223/builds/128

make buildbottest 'TESTOPTS=-j2 --junit-xml test-results.xml -j4 --fail-rerun ${BUILDBOT_TESTOPTS}' TESTPYTHONOPTS= TESTTIMEOUT=900
 in dir /buildbot/buildarea/3.x.ware-freebsd/build (timeout 1200 secs) (maxTime 14400 secs)
 watching logfiles {}
 argv: [b'make', b'buildbottest', b'TESTOPTS=-j2 --junit-xml test-results.xml -j4 --fail-rerun ${BUILDBOT_TESTOPTS}', b'TESTPYTHONOPTS=', b'TESTTIMEOUT=900']
(...)
./python -E  ./Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml -j4 --fail-rerun
(...)
1 re-run test:
    test_tarfile

Total duration: 5 min 27 sec
Total tests: run=40,945 skipped=1,320
Total test files: run=461/463 failed=1 skipped=15 resource_denied=3 rerun=1
Result: FAILURE then FAILURE

test_tarfile failure is known and unrelated

[vstinner]

I scheduled a Windows job: https://buildbot.python.org/all/#/builders/146/builds/6354

The --fail-rerun option is passed as expected!

Tools\buildbot\test.bat -p x64 -j4 --junit-xml test-results.xml --fail-rerun --timeout 900
 in dir D:\buildarea\3.x.bolen-windows10\build (timeout 1200 secs) (maxTime 14400 secs)
 watching logfiles {}
 argv: [b'Tools\\buildbot\\test.bat', b'-p', b'x64', b'-j4', b'--junit-xml', b'test-results.xml', b'--fail-rerun', b'--timeout', b'900']
(...)
D:\buildarea\3.x.bolen-windows10\build>"D:\buildarea\3.x.bolen-windows10\build\PCbuild\amd64\python_d.exe"  -u -Wd -E -bb -m test  -uall -rwW --slowest --timeout 1200 --fail-env-changed -j1 -j4 --junit-xml test-results.xml --fail-rerun --timeout 900 

My change works as expected: Result: FAILURE then SUCCESS, the build is marked as ORANGE 22:37 warnings test (warnings) WARNINGS.

1 re-run test:
    test_threading

Total duration: 20 min 31 sec
Total tests: run=40,689 skipped=2,242
Total test files: run=463/463 skipped=37 resource_denied=1 rerun=1
Result: FAILURE then SUCCESS
D:\buildarea\3.x.bolen-windows10\build>exit /b 5 
program finished with exit code 5

The orange color is a hint that something went wrong. Here is problem is that test_default_timeout was re-run:

Test the barrier's default timeout ... Warning -- Unraisable exception
Exception ignored in thread started by: <function Bunch.__init__.<locals>.task at 0x00000290E578B7D0>
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\lock_tests.py", line 48, in task
Warning -- Unraisable exception
Exception ignored in thread started by: <function Bunch.__init__.<locals>.task at 0x00000290E578B7D0>
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\lock_tests.py", line 48, in task
Warning -- Unraisable exception
Exception ignored in thread started by: <function Bunch.__init__.<locals>.task at 0x00000290E578B7D0>
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\lock_tests.py", line 48, in task
    f()
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\lock_tests.py", line 1020, in f
    f()
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\lock_tests.py", line 1020, in f
    f()
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\test\lock_tests.py", line 1020, in f
    i = barrier.wait()
        ^^^^^^^^^^^^^^
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\threading.py", line 709, in wait
    i = barrier.wait()
        ^^^^^^^^^^^^^^
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\threading.py", line 709, in wait
    i = barrier.wait()
        ^^^^^^^^^^^^^^
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\threading.py", line 709, in wait
    self._wait(timeout)
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\threading.py", line 749, in _wait
    self._wait(timeout)
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\threading.py", line 749, in _wait
    self._wait(timeout)
  File "D:\buildarea\3.x.bolen-windows10\build\Lib\threading.py", line 749, in _wait
    raise BrokenBarrierError
threading.BrokenBarrierError: 
    raise BrokenBarrierError
threading.BrokenBarrierError: 
    raise BrokenBarrierError
threading.BrokenBarrierError: 
ERROR

This issue is tracked as: python/cpython#108987

[vstinner]

For comparison, I scheduled a FreeBSD job on the 3.11 branch: https://buildbot.python.org/all/#/builders/1218/builds/35

On the 3.11 branch, tests are not run with --fail-rerun as expected.

make buildbottest 'TESTOPTS=-j2 --junit-xml test-results.xml -j4 ${BUILDBOT_TESTOPTS}' TESTPYTHONOPTS= TESTTIMEOUT=900
 in dir /buildbot/buildarea/3.11.ware-freebsd/build (timeout 1200 secs) (maxTime 14400 secs)
 watching logfiles {}
(...)
./python -E  ./Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml -j4 
== CPython 3.11.5+ (heads/3.11:f1f85a42ea, Sep 14 2023, 04:56:23) [Clang 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386
== FreeBSD-13.2-RELEASE-amd64-64bit-ELF little-endian
== Python build: debug
== cwd: /buildbot/buildarea/3.11.ware-freebsd/build/build/test_python_60282æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
Using random seed 879875
0:00:00 load avg: 1.30 Run tests in parallel using 4 child processes (timeout: 15 min, worker timeout: 20 min)