gh-95991: Add some infrastructure for testing Limited API in _testcapi #95992

encukou · 2022-08-15T09:34:11Z

Limited API needs to be enabled per source file
Some builds don't support Limited API, so Limited API tests must be skipped on those builds
(currently this is Py_TRACE_REFS, but that may change.)
Py_LIMITED_API must be defined before <Python.h> is included.

This puts the hoop-jumping in testcapi/parts.h, so individual
test files can be relatively simple. (Currently that's only
vectorcall_limited.c, imagine more.)

Issue: Test Limited API in _testcapi module #95991

…estcapi - Limited API needs to be enabled per source file - Some builds don't support Limited API, so Limited API tests must be skipped on those builds (currently this is `Py_TRACE_REFS`, but that may change.) - `Py_LIMITED_API` must be defined before `<Python.h>` is included. This puts the hoop-jumping in `testcapi/parts.h`, so individual test files can be relatively simple. (Currently that's only `vectorcall_limited.c`, imagine more.)

encukou · 2022-08-15T09:36:12Z

I'm not sure about best practices for test/support. @vstinner or @erlend-aasland, any comments on the @requires_limited_api decorator?

encukou · 2022-08-15T09:40:05Z

I also plan to add some instructions for the devguide, mentioning the structure for a Modules/testcapi/*.c file:

#define Py_LIMITED_API 0x03...
#include "parts.h"
#ifdef LIMITED_API_AVAILABLE
(all content goes here)
#endif

and @requires_limited_api for any Python parts.

erlend-aasland

Looks good! You might want to document the test decorator in Doc/library/test.rst.

Lib/test/support/__init__.py

Modules/_testcapi/parts.h

Co-authored-by: Erlend E. Aasland <[email protected]>

encukou · 2022-08-16T08:13:02Z

This happens to move #undef NDEBUG and so it should fix assert warnings reported in #96017

bedevere-bot · 2022-08-16T08:13:30Z

🤖 New build scheduled with the buildbot fleet by @encukou for commit b3b50c0 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

encukou · 2022-08-16T08:45:58Z

I don't understand the failure on the Windows buildbot:

     6>PrepareForBuild:
         Creating directory "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\_testcapi.tlog\".
       InitializeBuildStatus:
         Creating "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\_testcapi.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
       ClCompile:
         C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\HostX86\arm64\CL.exe /c /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\Include" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\Include\internal" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PC" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\\" /Zi /nologo /W3 /WX- /diagnostics:column /MP5 /O2 /Oi /Oy- /GL /D WIN32 /D "PY3_DLLNAME=L\"python3\"" /D _WIN32 /D NDEBUG /D _ARM64_WINAPI_PARTITION_DESKTOP_SDK_AVAILABLE=1 /D _WINDLL /GF /Gm- /MD /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\\" /Fd"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\vc143.pdb" /external:W3 /Gd /TC /analyze- /FC /errorReport:queue /utf-8 ..\Modules\_testcapimodule.c ..\Modules\_testcapi\vectorcall.c ..\Modules\_testcapi\vectorcall_limited.c ..\Modules\_testcapi\heaptype.c ..\Modules\_testcapi\unicode.c
         _testcapimodule.c
         vectorcall.c
         vectorcall_limited.c
         heaptype.c
         unicode.c
       ResourceCompile:
         C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x86\rc.exe /D "ORIGINAL_FILENAME=\\\"_testcapi.pyd\\\"" /D FIELD3=100 /D NDEBUG /l"0x0409" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PC" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\Include" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\\" /nologo /fo"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\python_nt.res" ..\PC\python_nt.rc 
       Link:
         C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\HostX86\arm64\link.exe /ERRORREPORT:QUEUE /OUT:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\_testcapi.pyd" /INCREMENTAL:NO /NOLOGO /LIBPATH:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\\" advapi32.lib shell32.lib ole32.lib oleaut32.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /NODEFAULTLIB:LIBC /MANIFEST:NO /DEBUG /PDB:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\_testcapi.pdb" /SUBSYSTEM:WINDOWS /LTCG /LTCGOUT:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\_testcapi.iobj" /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\_testcapi.lib" /MACHINE:ARM64 /OPT:REF,NOICF /DLL "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\python_nt.res"
         "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\_testcapimodule.obj"
         "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\vectorcall.obj"
         "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\vectorcall_limited.obj"
         "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\heaptype.obj"
         "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\unicode.obj"
         "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\python312.lib"
     6>LINK : fatal error LNK1104: cannot open file 'python3.lib' [C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\_testcapi.vcxproj]
     6>Done Building Project "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\_testcapi.vcxproj" (Build target(s)) -- FAILED.
     1>Done Building Project "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\pcbuild.proj" (Build target(s)) -- FAILED.

Windows failures need to be resolved

encukou · 2022-08-16T11:56:22Z

@python/windows-team, do you know what's wrong here, by any chance?

zooba · 2022-08-16T17:24:50Z

It's probably getting built before the limited API DLL (python3.dll). We need to force a dependency into the project file to ensure they are built in the right order.

Add the below to _testcapi.vcxproj underneath the existing <ProjectReference Include="pythoncore.vcxproj"> that's in there:

    <ProjectReference Include="python3dll.vcxproj">
      <Project>{885d4898-d08d-4091-9c40-c700cfe3fc5a}</Project>
      <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
    </ProjectReference>

bedevere-bot · 2022-08-16T18:30:29Z

🤖 New build scheduled with the buildbot fleet by @encukou for commit ee59cb3 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

encukou · 2022-08-17T11:47:40Z

Woohoo! That worked! Thanks for the wincantation, Steve, and thanks Erlend for the review!

The other buildbot failures are unrelated, and being handled in #96006 and #95898

vstinner · 2022-08-17T13:12:18Z

When Py_TRACE_REFS is defined, it was discussed once to move the double linked list of objects out of PyObject, to keep the ABI compatiiblity. It shouldn't be hard to store it somewhere else. But so far, nobody proposed a concrete implementation of this idea.

bedevere-bot added the awaiting core review label Aug 15, 2022

encukou added the skip news label Aug 15, 2022

Remove repeated #undef NDEBU

df49186

erlend-aasland previously approved these changes Aug 15, 2022

View reviewed changes

Lib/test/support/__init__.py Outdated Show resolved Hide resolved

Modules/_testcapi/parts.h Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting core review labels Aug 15, 2022

encukou and others added 2 commits August 16, 2022 09:47

Update Lib/test/support/__init__.py

4b946d4

Co-authored-by: Erlend E. Aasland <[email protected]>

Document requires_limited_api

b3b50c0

encukou mentioned this pull request Aug 16, 2022

gh-96017: Fix some compiler warnings (GH-96018) #96018

Merged

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 16, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 16, 2022

tiran mentioned this pull request Aug 16, 2022

Compiler warnings in main / 3.11 #96017

Closed

encukou mentioned this pull request Aug 16, 2022

Support metaclasses in HPy. hpyproject/hpy#335

Merged

Force a dependency on python3.dll in the Windows build

ee59cb3

encukou requested a review from a team as a code owner August 16, 2022 18:03

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 16, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 16, 2022

erlend-aasland approved these changes Aug 16, 2022

View reviewed changes

encukou mentioned this pull request Aug 17, 2022

gh-93649: Split heaptype tests from _testcapimodule.c #95386

Merged

encukou merged commit 0f2b469 into python:main Aug 17, 2022

bedevere-bot removed the awaiting merge label Aug 17, 2022

encukou deleted the skip-limited-tests-for-tracerefs branch August 17, 2022 11:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-95991: Add some infrastructure for testing Limited API in _testcapi #95992

gh-95991: Add some infrastructure for testing Limited API in _testcapi #95992

encukou commented Aug 15, 2022 •

edited by bedevere-bot

Loading

encukou commented Aug 15, 2022

encukou commented Aug 15, 2022

erlend-aasland left a comment

encukou commented Aug 16, 2022

bedevere-bot commented Aug 16, 2022

encukou commented Aug 16, 2022

encukou commented Aug 16, 2022 •

edited

Loading

zooba commented Aug 16, 2022

bedevere-bot commented Aug 16, 2022

encukou commented Aug 17, 2022

vstinner commented Aug 17, 2022

gh-95991: Add some infrastructure for testing Limited API in _testcapi #95992

gh-95991: Add some infrastructure for testing Limited API in _testcapi #95992

Conversation

encukou commented Aug 15, 2022 • edited by bedevere-bot Loading

encukou commented Aug 15, 2022

encukou commented Aug 15, 2022

erlend-aasland left a comment

Choose a reason for hiding this comment

encukou commented Aug 16, 2022

bedevere-bot commented Aug 16, 2022

encukou commented Aug 16, 2022

encukou commented Aug 16, 2022 • edited Loading

zooba commented Aug 16, 2022

bedevere-bot commented Aug 16, 2022

encukou commented Aug 17, 2022

vstinner commented Aug 17, 2022

encukou commented Aug 15, 2022 •

edited by bedevere-bot

Loading

encukou commented Aug 16, 2022 •

edited

Loading