Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New python312dev.yml (to replace upstream.yml) #30006

Merged
merged 13 commits into from
Mar 9, 2023

Conversation

rwgk
Copy link
Contributor

@rwgk rwgk commented Mar 9, 2023

Description

Add new python312dev.yml, replacing obsolete upstream.yml. The new name is to make it completely obvious what the workflow is for. (We will have to update the name only once a year.)

Main purpose: testing builtins / state dict change under PR #30005

The python312dev workflow was working completely with python-3.12.0-alpha.1-linux-22.04-x64.tar.gz, but the C++ tests fail with alpha.2 and alpha.3. Debugging led deep into the Python sources. This is captured in a comment; further debugging is left for later (help appreciated). The main purpose (above) is sufficiently met via the Python tests + interactive testing of test_embed (ignoring the one failure).

Suggested changelog entry:

@rwgk rwgk added the gha_wf_python312dev_enable Enable python312dev.yml label Mar 9, 2023
@rwgk rwgk closed this Mar 9, 2023
@rwgk rwgk reopened this Mar 9, 2023
rwgk pushed a commit to rwgk/pybind11clif that referenced this pull request Mar 9, 2023
@rwgk rwgk added the gha_wf_ci_disable Disable ci.yml label Mar 9, 2023
Ralf W. Grosse-Kunstleve added 5 commits March 9, 2023 10:52
Root problem:

* Up to and including 3.12alpha3: `PyErr_Fetch()` produces the `FlakyException` type.

* With 3.12alpha6: `PyErr_Fetch()` produces the `ValueError` type instead.

Additional detail:

Up to and including 3.12alpha3: only `PyErr_NormalizeException()` hits the `ValueError` in `FlakyException._init__`, but not `PyErr_Fetch()`.
…aks between Python 3.12alpha3 (still working) and 3.12alpha6 (broken):

```
str(OrderedDict([(1, "a"), (2, "b")]))
```

Old:
```
OrderedDict([(1, 'a'), (2, 'b')])
```

New:
```
OrderedDict({1: 'a', 2: 'b'})
```
…with locally install 3.12alpha6, but the behavior is exactly the same as previously observed in the GH workflow).
@rwgk rwgk added gha_wf_python312dev_enable Enable python312dev.yml and removed gha_wf_python312dev_enable Enable python312dev.yml gha_wf_ci_disable Disable ci.yml labels Mar 9, 2023
@rwgk rwgk closed this Mar 9, 2023
@rwgk rwgk reopened this Mar 9, 2023
@rwgk rwgk marked this pull request as ready for review March 9, 2023 21:06
@rwgk rwgk requested a review from wangxf123456 March 9, 2023 21:06
@rwgk rwgk merged commit af5c653 into google:main Mar 9, 2023
@rwgk rwgk deleted the python312_yml_pywrapcc branch March 9, 2023 21:14
rwgk pushed a commit that referenced this pull request Mar 15, 2023
* Transfer PR #4329 from master to smart_holder branch, STEP 1.

The patch .rej below are resolved, but THIS STATE DOES NOT BUILD:

```
clang++ -o pybind11/tests/test_constants_and_functions.os -c -std=c++17 -fPIC -fvisibility=hidden -O0 -g -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wundef -Wnon-virtual-dtor -Wunused-result -Werror -isystem /usr/include/python3.10 -isystem /usr/include/eigen3 -DPYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX -DPYBIND11_ENABLE_TYPE_CASTER_ODR_GUARD_IF_AVAILABLE -DPYBIND11_TEST_BOOST -Ipybind11/include -I/usr/local/google/home/rwgk/forked/pybind11/include -I/usr/local/google/home/rwgk/clone/pybind11/include /usr/local/google/home/rwgk/forked/pybind11/tests/test_constants_and_functions.cpp
In file included from /usr/local/google/home/rwgk/forked/pybind11/tests/test_constants_and_functions.cpp:11:
In file included from /usr/local/google/home/rwgk/forked/pybind11/tests/pybind11_tests.h:3:
In file included from /usr/local/google/home/rwgk/forked/pybind11/include/pybind11/eval.h:14:
/usr/local/google/home/rwgk/forked/pybind11/include/pybind11/pybind11.h:1781:9: error: static_assert failed due to requirement '!holder_is_smart_holder == type_caster_type_is_type_caster_base_subtype' "py::class_ holder vs type_caster mismatch: missing PYBIND11_TYPE_CASTER_BASE_HOLDER(T, ...) or collision with custom py::detail::type_caster<T>?"
        static_assert(!holder_is_smart_holder == type_caster_type_is_type_caster_base_subtype,
        ^             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/google/home/rwgk/forked/pybind11/include/pybind11/pybind11.h:2460:11: note: in instantiation of function template specialization 'pybind11::class_<MyEnum>::class_<>' requested here
        : class_<Type>(scope, name, extra...), m_base(*this, scope) {
          ^
/usr/local/google/home/rwgk/forked/pybind11/tests/test_constants_and_functions.cpp:106:5: note: in instantiation of function template specialization 'pybind11::enum_<MyEnum>::enum_<>' requested here
    py::enum_<MyEnum>(m, "MyEnum")
    ^
1 error generated.
```

________

```
rwgk.c.googlers.com:~/forked/pybind11 $ patch -p 1 < ~/native_enum_git_diff_master_2022-11-19+131942.patch
patching file .github/workflows/python312.yml
patching file CMakeLists.txt
Hunk #1 FAILED at 111.
Hunk #2 succeeded at 138 (offset 5 lines).
1 out of 2 hunks FAILED -- saving rejects to file CMakeLists.txt.rej
patching file include/pybind11/cast.h
Hunk #1 FAILED at 12.
Hunk #2 succeeded at 78 (offset 30 lines).
Hunk #3 succeeded at 1173 (offset 41 lines).
1 out of 3 hunks FAILED -- saving rejects to file include/pybind11/cast.h.rej
patching file include/pybind11/detail/abi_platform_id.h
patching file include/pybind11/detail/cross_extension_shared_state.h
patching file include/pybind11/detail/internals.h
Hunk #1 FAILED at 16.
Hunk #2 succeeded at 109 (offset 1 line).
Hunk #3 FAILED at 203.
Hunk #4 succeeded at 398 (offset 11 lines).
Hunk #5 succeeded at 457 (offset 11 lines).
2 out of 5 hunks FAILED -- saving rejects to file include/pybind11/detail/internals.h.rej
patching file include/pybind11/detail/native_enum_data.h
patching file include/pybind11/detail/type_map.h
patching file include/pybind11/embed.h
patching file include/pybind11/native_enum.h
patching file include/pybind11/pybind11.h
Hunk #1 FAILED at 12.
Hunk #2 succeeded at 1269 (offset 1 line).
Hunk #3 succeeded at 2457 (offset 255 lines).
1 out of 3 hunks FAILED -- saving rejects to file include/pybind11/pybind11.h.rej
patching file include/pybind11/pytypes.h
patching file tests/CMakeLists.txt
Hunk #1 succeeded at 160 (offset 18 lines).
patching file tests/conftest.py
patching file tests/extra_python_package/test_files.py
Hunk #2 FAILED at 47.
1 out of 2 hunks FAILED -- saving rejects to file tests/extra_python_package/test_files.py.rej
patching file tests/test_embed/test_interpreter.cpp
patching file tests/test_enum.cpp
patching file tests/test_enum.py
patching file tests/test_native_enum.cpp
patching file tests/test_native_enum.py
```

* Make `smart_holder` code compatible with `type_caster_enum_type`

* Fix `if` condition guarding `Unable to cast native enum type to reference`

* WIP native_enum_add_to_parent

* PYBIND11_SILENCE_MSVC_C4127

* Transfer upstream.yml/python312.yml changes from PR #4397

* clang-tidy (clang 15) auto-fix

* Remove python312.yml: this is handled separately under PR #30006

* Fixes for ruff

* Replace `PyEval_GetBuiltins()` in `finalize_interpreter()` with `get_python_state_dict()`

* Bug fix: Ensure `state_dict` is destroyed before `Py_Finalize()`

* Restore tests/test_embed/test_interpreter.cpp from google_pywrapcc_main

* Restore include/pybind11/embed.h from google_pywrapcc_main

* Back out all native_enum changes, leaving only the cross_extension_shared_state changes.

* Undo unrelated one-line change (from `auto` to `internals`).
rwgk pushed a commit that referenced this pull request Mar 16, 2023
#30005)

* Transfer PR #4329 from master to smart_holder branch, STEP 1.

The patch .rej below are resolved, but THIS STATE DOES NOT BUILD:

```
clang++ -o pybind11/tests/test_constants_and_functions.os -c -std=c++17 -fPIC -fvisibility=hidden -O0 -g -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wundef -Wnon-virtual-dtor -Wunused-result -Werror -isystem /usr/include/python3.10 -isystem /usr/include/eigen3 -DPYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX -DPYBIND11_ENABLE_TYPE_CASTER_ODR_GUARD_IF_AVAILABLE -DPYBIND11_TEST_BOOST -Ipybind11/include -I/usr/local/google/home/rwgk/forked/pybind11/include -I/usr/local/google/home/rwgk/clone/pybind11/include /usr/local/google/home/rwgk/forked/pybind11/tests/test_constants_and_functions.cpp
In file included from /usr/local/google/home/rwgk/forked/pybind11/tests/test_constants_and_functions.cpp:11:
In file included from /usr/local/google/home/rwgk/forked/pybind11/tests/pybind11_tests.h:3:
In file included from /usr/local/google/home/rwgk/forked/pybind11/include/pybind11/eval.h:14:
/usr/local/google/home/rwgk/forked/pybind11/include/pybind11/pybind11.h:1781:9: error: static_assert failed due to requirement '!holder_is_smart_holder == type_caster_type_is_type_caster_base_subtype' "py::class_ holder vs type_caster mismatch: missing PYBIND11_TYPE_CASTER_BASE_HOLDER(T, ...) or collision with custom py::detail::type_caster<T>?"
        static_assert(!holder_is_smart_holder == type_caster_type_is_type_caster_base_subtype,
        ^             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/google/home/rwgk/forked/pybind11/include/pybind11/pybind11.h:2460:11: note: in instantiation of function template specialization 'pybind11::class_<MyEnum>::class_<>' requested here
        : class_<Type>(scope, name, extra...), m_base(*this, scope) {
          ^
/usr/local/google/home/rwgk/forked/pybind11/tests/test_constants_and_functions.cpp:106:5: note: in instantiation of function template specialization 'pybind11::enum_<MyEnum>::enum_<>' requested here
    py::enum_<MyEnum>(m, "MyEnum")
    ^
1 error generated.
```

________

```
rwgk.c.googlers.com:~/forked/pybind11 $ patch -p 1 < ~/native_enum_git_diff_master_2022-11-19+131942.patch
patching file .github/workflows/python312.yml
patching file CMakeLists.txt
Hunk #1 FAILED at 111.
Hunk #2 succeeded at 138 (offset 5 lines).
1 out of 2 hunks FAILED -- saving rejects to file CMakeLists.txt.rej
patching file include/pybind11/cast.h
Hunk #1 FAILED at 12.
Hunk #2 succeeded at 78 (offset 30 lines).
Hunk #3 succeeded at 1173 (offset 41 lines).
1 out of 3 hunks FAILED -- saving rejects to file include/pybind11/cast.h.rej
patching file include/pybind11/detail/abi_platform_id.h
patching file include/pybind11/detail/cross_extension_shared_state.h
patching file include/pybind11/detail/internals.h
Hunk #1 FAILED at 16.
Hunk #2 succeeded at 109 (offset 1 line).
Hunk #3 FAILED at 203.
Hunk #4 succeeded at 398 (offset 11 lines).
Hunk #5 succeeded at 457 (offset 11 lines).
2 out of 5 hunks FAILED -- saving rejects to file include/pybind11/detail/internals.h.rej
patching file include/pybind11/detail/native_enum_data.h
patching file include/pybind11/detail/type_map.h
patching file include/pybind11/embed.h
patching file include/pybind11/native_enum.h
patching file include/pybind11/pybind11.h
Hunk #1 FAILED at 12.
Hunk #2 succeeded at 1269 (offset 1 line).
Hunk #3 succeeded at 2457 (offset 255 lines).
1 out of 3 hunks FAILED -- saving rejects to file include/pybind11/pybind11.h.rej
patching file include/pybind11/pytypes.h
patching file tests/CMakeLists.txt
Hunk #1 succeeded at 160 (offset 18 lines).
patching file tests/conftest.py
patching file tests/extra_python_package/test_files.py
Hunk #2 FAILED at 47.
1 out of 2 hunks FAILED -- saving rejects to file tests/extra_python_package/test_files.py.rej
patching file tests/test_embed/test_interpreter.cpp
patching file tests/test_enum.cpp
patching file tests/test_enum.py
patching file tests/test_native_enum.cpp
patching file tests/test_native_enum.py
```

* Make `smart_holder` code compatible with `type_caster_enum_type`

* Fix `if` condition guarding `Unable to cast native enum type to reference`

* WIP native_enum_add_to_parent

* PYBIND11_SILENCE_MSVC_C4127

* Transfer upstream.yml/python312.yml changes from PR #4397

* clang-tidy (clang 15) auto-fix

* Remove python312.yml: this is handled separately under PR #30006

* Fixes for ruff

* Replace `PyEval_GetBuiltins()` in `finalize_interpreter()` with `get_python_state_dict()`

* Bug fix: Ensure `state_dict` is destroyed before `Py_Finalize()`

* Restore tests/test_embed/test_interpreter.cpp from google_pywrapcc_main

* Restore include/pybind11/embed.h from google_pywrapcc_main

* Undo unrelated one-line change (from `auto` to `internals`).

* Add missing `test_class_with_enum`

* Add note to docs/classes.rst pointing to PR #30005
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gha_wf_python312dev_enable Enable python312dev.yml
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants