bpo-11105: use a lower recursion limit for infinite recursion tests #26550

isidentical · 2021-06-05T18:39:55Z

https://bugs.python.org/issue11105

isidentical · 2021-06-05T18:45:50Z

@Fidget-Spinner does this patch prevent the crash for your end (do you have resources to test it out?). I highly doubt so (since these functions are giant I don't expect them to be inlined), though if it does then it would be good to know.

Fidget-Spinner · 2021-06-05T19:02:53Z

@Fidget-Spinner does this patch prevent the crash for your end (do you have resources to test it out?). I highly doubt so (since these functions are giant I don't expect them to be inlined), though if it does then it would be good to know.

Thanks for working on this! Yup I can test it out, I use MSVC often.

Your doubts were right, this is sadly still causing a stack overflow. BTW, testing with buildbots should catch it, the AMD64 Windows10 Pro 3.x and AMD64 Windows10 3.x buildbots both caught it the previous commit. For PR pushes, their names are AMD64 Windows10 Pro PR and AMD64 Windows10 PR (the workers are bolen-windows10 and kloth-win64). Both seem to have errored out though they haven't updated their statuses yet.

isidentical · 2021-06-05T19:12:09Z

Both seem to have errored out though they haven't updated their statuses yet.

Cool! I was worried that they are not available on the public list, but apparently, they are! Thanks for letting me know and testing this out.

bedevere-bot · 2021-06-05T22:39:21Z

🤖 New build scheduled with the buildbot fleet by @isidentical for commit f31c8411cc51590aee9bc8c2b3042911316b546e 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

isidentical · 2021-06-05T23:14:32Z

Seems like the error in test_ast doesn't happen anymore though we still get a stack overflow from somewhere else which doesn't make much sense since there is no ast compilation related cover on that test.

test_recursion_in_except_handler (test.test_exceptions.ExceptionTests) ... 
D:\buildarea\3.x.bolen-windows10\build>exit /b -1073741571 
Windows fatal exception: stack overflow
Current thread 0x00000a54 (most recent call first):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_exceptions.py", line 1132 in test_recursion_in_except_handler
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\case.py", line 549 in _callTestMethod
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\case.py", line 592 in run
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\case.py", line 652 in __call__
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 122 in run
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 122 in run
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 122 in run
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\buildarea\3.x.bolen-windows10\build\lib\unittest\runner.py", line 176 in run
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\support\__init__.py", line 960 in _run_suite
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\support\__init__.py", line 1083 in run_unittest
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 210 in _test_module
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 246 in _runtest_inner2
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 282 in _runtest_inner
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 154 in _runtest
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\runtest.py", line 194 in runtest
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\main.py", line 321 in rerun_failed_tests
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\main.py", line 698 in _main
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\main.py", line 641 in main
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\libregrtest\main.py", line 719 in main
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\__main__.py", line 2 in <module>
  File "D:\buildarea\3.x.bolen-windows10\build\lib\runpy.py", line 86 in _run_code
  File "D:\buildarea\3.x.bolen-windows10\build\lib\runpy.py", line 196 in _run_module_as_main

Fidget-Spinner · 2021-06-06T04:37:05Z

Seems like the error in test_ast doesn't happen anymore though we still get a stack overflow from somewhere else which doesn't make much sense since there is no ast compilation related cover on that test.
test_recursion_in_except_handler (test.test_exceptions.ExceptionTests) ... 

I just checked on main branch, this specific test also causes a stack overflow there. So maybe that test failure was previously masked by this one. Your patch fixes the AST stack overflow stack overflow issue for windows. Thanks!

Something I found strange is that the exceptions recursion test was added in 4e7a69b but it passed all along. I'm starting to suspect maybe there's some other change elsewhere that caused both test_exception and test_ast to fail. I'll start bisecting with test_exception soon. So please wait before merging this PR, thanks!

Fidget-Spinner · 2021-06-06T10:59:46Z

Ok, bisect tells me that this commit caused test_exception to fail:

commit f3fa63ec75fdbb4a08a10957a5c631bf0c4a5970
[bpo-39573](https://bugs.python.org/issue39573): Py_TYPE becomes a static inline function (GH-26493)

    Convert the Py_TYPE() and Py_SIZE() macros to static inline
    functions. The Py_SET_TYPE() and Py_SET_SIZE() functions must now be
    used to set an object type and size.

bedevere-bot · 2021-06-08T11:25:30Z

🤖 New build scheduled with the buildbot fleet by @isidentical for commit 1bf3518db90f3e1c6c8eb1d8daf925e0ae5ee649 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

Fidget-Spinner · 2021-06-08T11:28:04Z

@isidentical I think if you rebase to the latest 6d518bb, test with buildbots should mostly pass. commit f3fa63e was reverted.

bedevere-bot · 2021-06-08T11:30:12Z

🤖 New build scheduled with the buildbot fleet by @isidentical for commit e01e32d 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

Fidget-Spinner

LGTM. The windows 10 tests are passing. Thanks for your efforts. Without your help I would've taken a lot longer to track down the other windows buildbot failures.

Fidget-Spinner · 2021-06-08T13:33:33Z

Lib/test/support/__init__.py

@@ -1999,3 +1999,12 @@ def check_disallow_instantiation(testcase, tp, *args, **kwds):
        qualname = f"{name}"
    msg = f"cannot create '{re.escape(qualname)}' instances"
    testcase.assertRaisesRegex(TypeError, msg, tp, *args, **kwds)
+
+@contextlib.contextmanager
+def infinite_recursion(max_depth=75):


Minor suggestion: I think adding a docstring would help others know where/why/when they need this.

Ah, missed this one! Yeap, forgot to do that. Will add a follow up PR.

miss-islington · 2021-06-08T17:01:06Z

Thanks @isidentical for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2021-06-08T17:01:06Z

Thanks @isidentical for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

miss-islington · 2021-06-08T17:01:09Z

Sorry, @isidentical, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker e58d762c1fb4ad5e021d016c80c2bc4513632d2f 3.9

miss-islington · 2021-06-08T17:01:11Z

Sorry @isidentical, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker e58d762c1fb4ad5e021d016c80c2bc4513632d2f 3.10

(cherry picked from commit e58d762) Co-authored-by: Batuhan Taskaya <[email protected]>

bedevere-bot · 2021-06-08T17:03:40Z

GH-26605 is a backport of this pull request to the 3.9 branch.

(cherry picked from commit e58d762) Co-authored-by: Batuhan Taskaya <[email protected]>

bedevere-bot · 2021-06-08T17:04:27Z

GH-26607 is a backport of this pull request to the 3.10 branch.

the-knights-who-say-ni added the CLA signed label Jun 5, 2021

bedevere-bot added the awaiting core review label Jun 5, 2021

isidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021

isidentical added the skip news label Jun 5, 2021

isidentical force-pushed the bpo-11105-recursion-windows branch from b246c21 to 7f83161 Compare June 5, 2021 20:29

isidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021

isidentical force-pushed the bpo-11105-recursion-windows branch from 7f83161 to d7b496f Compare June 5, 2021 20:47

isidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021

isidentical force-pushed the bpo-11105-recursion-windows branch from d7b496f to f31c841 Compare June 5, 2021 22:39

isidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 5, 2021

python deleted a comment from bedevere-bot Jun 5, 2021

isidentical force-pushed the bpo-11105-recursion-windows branch from f31c841 to 1bf3518 Compare June 8, 2021 11:08

isidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2021

isidentical changed the title ~~bpo-11105: [WIP] reduce stack usage~~ bpo-11105: use a lower recursion limit for infinite recursion tests Jun 8, 2021

bpo-11105: reduce the recursion limit for tests

e01e32d

isidentical force-pushed the bpo-11105-recursion-windows branch from 1bf3518 to e01e32d Compare June 8, 2021 11:29

isidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jun 8, 2021

Fidget-Spinner approved these changes Jun 8, 2021

View reviewed changes

isidentical marked this pull request as ready for review June 8, 2021 16:54

isidentical merged commit e58d762 into python:main Jun 8, 2021

bedevere-bot removed the awaiting core review label Jun 8, 2021

isidentical added needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes labels Jun 8, 2021

miss-islington assigned isidentical Jun 8, 2021

isidentical added a commit to isidentical/cpython that referenced this pull request Jun 8, 2021

[3.9] bpo-11105: reduce the recursion limit for tests (pythonGH-26550).

1ba3e19

(cherry picked from commit e58d762) Co-authored-by: Batuhan Taskaya <[email protected]>

bedevere-bot removed the needs backport to 3.9 only security fixes label Jun 8, 2021

isidentical added a commit to isidentical/cpython that referenced this pull request Jun 8, 2021

[3.10] bpo-11105: reduce the recursion limit for tests (pythonGH-26550).

59c9615

(cherry picked from commit e58d762) Co-authored-by: Batuhan Taskaya <[email protected]>

bedevere-bot removed the needs backport to 3.10 only security fixes label Jun 8, 2021

vstinner mentioned this pull request Mar 12, 2024

[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type #83754

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-11105: use a lower recursion limit for infinite recursion tests #26550

bpo-11105: use a lower recursion limit for infinite recursion tests #26550

isidentical commented Jun 5, 2021 •

edited by bedevere-bot

Loading

isidentical commented Jun 5, 2021

Fidget-Spinner commented Jun 5, 2021 •

edited

Loading

isidentical commented Jun 5, 2021 •

edited

Loading

bedevere-bot commented Jun 5, 2021

isidentical commented Jun 5, 2021

Fidget-Spinner commented Jun 6, 2021

Fidget-Spinner commented Jun 6, 2021 •

edited by bedevere-bot

Loading

bedevere-bot commented Jun 8, 2021

Fidget-Spinner commented Jun 8, 2021

bedevere-bot commented Jun 8, 2021

Fidget-Spinner left a comment

Fidget-Spinner Jun 8, 2021 •

edited

Loading

isidentical Jun 8, 2021

miss-islington commented Jun 8, 2021

miss-islington commented Jun 8, 2021

miss-islington commented Jun 8, 2021

miss-islington commented Jun 8, 2021

bedevere-bot commented Jun 8, 2021

bedevere-bot commented Jun 8, 2021

bpo-11105: use a lower recursion limit for infinite recursion tests #26550

bpo-11105: use a lower recursion limit for infinite recursion tests #26550

Conversation

isidentical commented Jun 5, 2021 • edited by bedevere-bot Loading

isidentical commented Jun 5, 2021

Fidget-Spinner commented Jun 5, 2021 • edited Loading

isidentical commented Jun 5, 2021 • edited Loading

bedevere-bot commented Jun 5, 2021

isidentical commented Jun 5, 2021

Fidget-Spinner commented Jun 6, 2021

Fidget-Spinner commented Jun 6, 2021 • edited by bedevere-bot Loading

bedevere-bot commented Jun 8, 2021

Fidget-Spinner commented Jun 8, 2021

bedevere-bot commented Jun 8, 2021

Fidget-Spinner left a comment

Choose a reason for hiding this comment

Fidget-Spinner Jun 8, 2021 • edited Loading

Choose a reason for hiding this comment

isidentical Jun 8, 2021

Choose a reason for hiding this comment

miss-islington commented Jun 8, 2021

miss-islington commented Jun 8, 2021

miss-islington commented Jun 8, 2021

miss-islington commented Jun 8, 2021

bedevere-bot commented Jun 8, 2021

bedevere-bot commented Jun 8, 2021

isidentical commented Jun 5, 2021 •

edited by bedevere-bot

Loading

Fidget-Spinner commented Jun 5, 2021 •

edited

Loading

isidentical commented Jun 5, 2021 •

edited

Loading

Fidget-Spinner commented Jun 6, 2021 •

edited by bedevere-bot

Loading

Fidget-Spinner Jun 8, 2021 •

edited

Loading