patchcheck.py raises SystemError on main

[AlexWaygood]

Bug report

Following #105801, running python Tools/patchcheck/patchcheck.py now results in SystemError being raised.

To reproduce

1. Make any change to a .py file, e.g.:

--- a/Lib/test/test_fstring.py
+++ b/Lib/test/test_fstring.py
@@ -19,6 +19,7 @@

 a_global = 'global variable'

+

2. Run python Tools/patchcheck/patchcheck.py:

C:\Users\alexw\coding\cpython>python Tools/patchcheck/patchcheck.py
Running Release|x64 interpreter...
Getting base branch for PR ... upstream/main
Getting the list of files that have been added/changed ... 1 file
Fixing Python file whitespace ... Traceback (most recent call last):
  File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 325, in <module>
    main()
  File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 293, in main
    normalize_whitespace(python_files)
  File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 35, in call_fxn
    result = fxn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 188, in normalize_whitespace
    and reindent.check(os.path.join(SRCDIR, path))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Tools\patchcheck\reindent.py", line 138, in check
    if r.run():
       ^^^^^^^
  File "C:\Users\alexw\coding\cpython\Tools\patchcheck\reindent.py", line 203, in run
    for _token in tokens:
  File "C:\Users\alexw\coding\cpython\Lib\tokenize.py", line 537, in _generate_tokens_from_c_tokenizer
    for info in it:
SystemError: Negative size passed to PyUnicode_New

This failure was visible on the Azure Pipelines job on the PR FYI: https://dev.azure.com/Python/cpython/_build/results?buildId=130389&view=logs&j=256d7e09-002a-52d7-8661-29ee3960640e&t=3d7276d3-4e8d-5309-55ad-fb0b172d9925

Your environment

Python 3.13.0a0 (heads/main:c5111aec2b, Jun 15 2023, 10:53:37) [MSC v.1932 64 bit (AMD64)] on win32

Linked PRs

• gh-105820: Fix tok_mode expression buffer in file & readline tokenizer #105828
• [3.12] gh-105820: Fix tok_mode expression buffer in file & readline tokenizer (GH-105828) #105832

[pablogsal]

This failure was visible on the Azure Pipelines job on the PR FYI: https://dev.azure.com/Python/cpython/_build/results?buildId=130389&view=logs&j=256d7e09-002a-52d7-8661-29ee3960640e&t=3d7276d3-4e8d-5309-55ad-fb0b172d9925

Same comment as #105821 (comment)

[AlexWaygood]

Unlike the "hypothesis checks on Ubuntu" job, though, Azure Pipelines is incredibly flaky, it fails all the time for random reasons. So it's not realistic to make that one "required" in CI.

[pablogsal]

So it's not realistic to make that one "required" in CI.

Then we should move the parts we care about to some other check and just remove IMHO because nobody looks at it when merging because is constantly failing.

[AlexWaygood]

nobody looks at it when merging

"nobody" isn't accurate — I generally check failures to see if they're relevant before merging a PR. But I agree with:

Then we should move the parts we care about to some other check and just remove

And we've actually started doing that already; the new "lint" job in CI is basically meant to be a replacement for patchcheck.py, for example. Personally I don't care about patchcheck.py; I only opened this issue because I worried this might be indicative of other breakages :)

[pablogsal]

I only opened this issue because I worried this might be indicative of other breakages

Thanks a lot @AlexWaygood, this is certainly the case and we are investigating. Thanks for all the gray work 🖤

[hugovk]

Similarly, see #105823 to remove some duplicated docs building from Azure Pipelines.

[lysnikolaou]

I don't really know why we're seeng this just now, but git bisecting shows that this is the first bad commit:

9216e69a87d16d871625721ed5a8aa302511f367 is the first bad commit
commit 9216e69a87d16d871625721ed5a8aa302511f367
Author: Pablo Galindo Salgado <[email protected]>
Date:   Tue May 30 22:43:34 2023 +0100

    gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (#105070)

The reproducer I used was (the good case):

cpython (2ea34cf) (BISECTING) via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ cat tmp/t.py         
self.assertEqual(f'''{
3
=}''', '\n3\n=3')%                                                                                                                                                                                                                                                                                                                                                                                                                  

cpython (2ea34cf) (BISECTING) via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ ./python.exe -m tokenize tmp/t.py
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,4:            NAME           'self'         
1,4-1,5:            OP             '.'            
1,5-1,16:           NAME           'assertEqual'  
1,16-1,17:          OP             '('            
1,17-1,21:          FSTRING_START  "f'''"         
1,21-1,22:          OP             '{'            
1,22-1,23:          NL             '\n'           
2,0-2,1:            NUMBER         '3'            
2,1-2,2:            NL             '\n'           
3,0-3,1:            OP             '='            
3,1-3,2:            OP             '}'            
3,2-3,5:            FSTRING_END    "'''"          
3,5-3,6:            OP             ','            
3,7-3,16:           STRING         "'\\n3\\n=3'"  
3,16-3,17:          OP             ')'            
3,17-3,18:          NEWLINE        '\n'           
4,0-4,0:            ENDMARKER      ''             

I'm working on a fix.

[lysnikolaou]

Fixed.