Fix #323: Support Python 3.12 #327

mdboom · 2022-11-04T15:13:21Z

No description provided.

amotl

Thank you for adding support for Python 3.12. You may want to exercise my suggestions on how to improve CI/GHA setup, if you like them.

Also, it may make sense to add a corresponding line to https://github.com/python-greenlet/greenlet/blob/master/setup.py#L216, to indicate the package is ready for Python 3.12 already.

.github/workflows/tests.yml

mdboom · 2022-11-16T14:22:52Z

Thanks for the review, @amotl.

Having updated it, and rerunning, it seems we are now running into a new set of test failures. This may be due to recent changes in 3.12, I guess. I will have to investigate further.

======================================================================
FAIL: test_unhandled_exception_in_greenlet_aborts (greenlet.tests.test_cpp.CPPTests.test_unhandled_exception_in_greenlet_aborts)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line 316, in wrapper
    return _RefCountChecker(self, method)(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line [294](https://github.com/python-greenlet/greenlet/actions/runs/3480120383/jobs/5819513041#step:9:295), in __call__
    self._run_test(args, kwargs)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line 212, in _run_test
    self.function(self.testcase, *args, **kwargs)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_cpp.py", line 67, in test_unhandled_exception_in_greenlet_aborts
    self._do_test_unhandled_exception(run_unhandled_exception_in_greenlet_aborts)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_cpp.py", line 58, in _do_test_unhandled_exception
    self.assertEqual(p.exitcode, expected_exit)
AssertionError: -11 != -6

======================================================================
FAIL: test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_main (greenlet.tests.test_leaks.TestLeaks.test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_main)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 445, in test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_main
    self._check_untracked_memory_thread(deallocate_in_thread=False)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 422, in _check_untracked_memory_thread
    self.assertEqual(EXIT_COUNT[0], 0)
AssertionError: 398 != 0

======================================================================
FAIL: test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_thread (greenlet.tests.test_leaks.TestLeaks.test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_thread)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 439, in test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_thread
    self._check_untracked_memory_thread(deallocate_in_thread=True)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 417, in _check_untracked_memory_thread
    self.assertEqual(EXIT_COUNT[0], ITER * thread_func.ITER)
AssertionError: 398 != 20000

----------------------------------------------------------------------

mdboom · 2022-11-16T15:07:59Z

I can't seem to reproduce these errors locally, with either CPython main or 3.12.0a1 (which seems to be what the CI here is using). I'm not sure how to debug this further.

amotl · 2022-11-21T12:17:57Z

Thank you, Michael. Apologies that I probably can't help on this matter. However, maybe those observations add some additional insights?

At [1], there is also:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.1/x64/lib/python3.12/threading.py", line 1049, in _bootstrap_inner
  File "/opt/hostedtoolcache/Python/3.12.0-alpha.1/x64/lib/python3.12/threading.py", line 986, in run
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 392, in __call__
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 378, in run_it
RecursionError: maximum recursion depth exceeded

And at [2], there is:

terminate called after throwing an instance of 'exception_t'
Fatal Python error: Aborted

[1] https://github.com/python-greenlet/greenlet/actions/runs/3480120383/jobs/5819513041#step:9:237
[2] https://github.com/python-greenlet/greenlet/actions/runs/3480120383/jobs/5819513041#step:9:135

tacaswell · 2023-01-27T04:39:57Z

This has grown another compilation failure:

✔ 23:38:23 $ gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.12 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-cpython-312/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:20,
                 from src/greenlet/greenlet.cpp:19:
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:847:42: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘trash_delete_nesting’
  847 |     this->trash_delete_nesting = tstate->trash_delete_nesting;
      |                                          ^~~~~~~~~~~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:884:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘trash_delete_nesting’
  884 |     tstate->trash_delete_nesting = this->trash_delete_nesting;
      |             ^~~~~~~~~~~~~~~~~~~~
src/greenlet/greenlet.cpp: In function ‘PyObject* mod_get_tstate_trash_delete_nesting(PyObject*)’:
src/greenlet/greenlet.cpp:3084:36: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘trash_delete_nesting’
 3084 |     return PyLong_FromLong(tstate->trash_delete_nesting);
      |                                    ^~~~~~~~~~~~~~~~~~~~

raphaelauv · 2023-03-05T00:47:39Z

@mdboom thanks for the PR , you could try the CI with 3.12.0-alpha.5 ?

hugovk · 2023-03-05T06:03:19Z

3.12-dev points to the latest prerelease, which is currently 3.12.0-alpha.5 (with alpha 6 due next week: https://peps.python.org/pep-0693/), so how about keeping 3.12-dev and restarting the CI?

tacaswell · 2023-04-03T21:38:38Z

This is now failing with

$ gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -DCYTHON_FAST_THREAD_STATE=0 -fPIC -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding
/include/python3.12 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-cpython-312/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:20,
                 from src/greenlet/greenlet.cpp:19:
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:847:42: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘trash_delete_nesting’
  847 |     this->trash_delete_nesting = tstate->trash_delete_nesting;
      |                                          ^~~~~~~~~~~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:884:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘trash_delete_nesting’
  884 |     tstate->trash_delete_nesting = this->trash_delete_nesting;
      |             ^~~~~~~~~~~~~~~~~~~~
src/greenlet/greenlet.cpp: In function ‘PyObject* mod_get_tstate_trash_delete_nesting(PyObject*)’:
src/greenlet/greenlet.cpp:3084:36: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘trash_delete_nesting’
 3084 |     return PyLong_FromLong(tstate->trash_delete_nesting);
      |                                    ^~~~~~~~~~~~~~~~~~~~

which was removed in python/cpython#101209

The following patch restores compilation:

diff --git a/src/greenlet/greenlet.cpp b/src/greenlet/greenlet.cpp
index 5990323..cf3492e 100644
--- a/src/greenlet/greenlet.cpp
+++ b/src/greenlet/greenlet.cpp
@@ -3081,7 +3081,7 @@ static PyObject*
 mod_get_tstate_trash_delete_nesting(PyObject* UNUSED(module))
 {
     PyThreadState* tstate = PyThreadState_GET();
-    return PyLong_FromLong(tstate->trash_delete_nesting);
+    return PyLong_FromLong(tstate->trash.delete_nesting);
 }

 static PyMethodDef GreenMethods[] = {
diff --git a/src/greenlet/greenlet_greenlet.hpp b/src/greenlet/greenlet_greenlet.hpp
index 8e9228a..b08921c 100644
--- a/src/greenlet/greenlet_greenlet.hpp
+++ b/src/greenlet/greenlet_greenlet.hpp
@@ -844,7 +844,7 @@ void PythonState::operator<<(const PyThreadState *const tstate) G_NOEXCEPT
 #endif

     // All versions of Python.
-    this->trash_delete_nesting = tstate->trash_delete_nesting;
+    this->trash_delete_nesting = tstate->trash.delete_nesting;
 }

 void PythonState::operator>>(PyThreadState *const tstate) G_NOEXCEPT
@@ -881,7 +881,7 @@ void PythonState::operator>>(PyThreadState *const tstate) G_NOEXCEPT
     tstate->recursion_depth = this->recursion_depth;
 #endif
     // All versions of Python.
-    tstate->trash_delete_nesting = this->trash_delete_nesting;
+    tstate->trash.delete_nesting = this->trash_delete_nesting;
 }

 void PythonState::will_switch_from(PyThreadState *const origin_tstate) G_NOEXCEPT

but I have not tracked down if the upstream refactor is going to require a logical refactor on the greenlet side (as I do not actually understand what either side is doing!).

tacaswell · 2023-04-13T20:42:43Z

due to python/cpython#103083 the full patch set I need to get greenlet to compile is:

diff --git a/src/greenlet/greenlet.cpp b/src/greenlet/greenlet.cpp
index 5990323..cf3492e 100644
--- a/src/greenlet/greenlet.cpp
+++ b/src/greenlet/greenlet.cpp
@@ -3081,7 +3081,7 @@ static PyObject*
 mod_get_tstate_trash_delete_nesting(PyObject* UNUSED(module))
 {
     PyThreadState* tstate = PyThreadState_GET();
-    return PyLong_FromLong(tstate->trash_delete_nesting);
+    return PyLong_FromLong(tstate->trash.delete_nesting);
 }

 static PyMethodDef GreenMethods[] = {
diff --git a/src/greenlet/greenlet_greenlet.hpp b/src/greenlet/greenlet_greenlet.hpp
index 8e9228a..1213140 100644
--- a/src/greenlet/greenlet_greenlet.hpp
+++ b/src/greenlet/greenlet_greenlet.hpp
@@ -823,7 +823,6 @@ void PythonState::operator<<(const PyThreadState *const tstate) G_NOEXCEPT
       the switch, use `will_switch_from`.
     */
     this->cframe = tstate->cframe;
-    this->use_tracing = tstate->cframe->use_tracing;
 #endif
 #if GREENLET_PY311
     #if GREENLET_PY312
@@ -844,7 +843,7 @@ void PythonState::operator<<(const PyThreadState *const tstate) G_NOEXCEPT
 #endif

     // All versions of Python.
-    this->trash_delete_nesting = tstate->trash_delete_nesting;
+    this->trash_delete_nesting = tstate->trash.delete_nesting;
 }

 void PythonState::operator>>(PyThreadState *const tstate) G_NOEXCEPT
@@ -857,13 +856,6 @@ void PythonState::operator>>(PyThreadState *const tstate) G_NOEXCEPT
 #endif
 #if GREENLET_USE_CFRAME
     tstate->cframe = this->cframe;
-    /*
-      If we were tracing, we need to keep tracing.
-      There should never be the possibility of hitting the
-      root_cframe here. See note above about why we can't
-      just copy this from ``origin->cframe->use_tracing``.
-    */
-    tstate->cframe->use_tracing = this->use_tracing;
 #endif
 #if GREENLET_PY311
     #if GREENLET_PY312
@@ -881,18 +873,12 @@ void PythonState::operator>>(PyThreadState *const tstate) G_NOEXCEPT
     tstate->recursion_depth = this->recursion_depth;
 #endif
     // All versions of Python.
-    tstate->trash_delete_nesting = this->trash_delete_nesting;
+    tstate->trash.delete_nesting = this->trash_delete_nesting;
 }

 void PythonState::will_switch_from(PyThreadState *const origin_tstate) G_NOEXCEPT
 {
-#if GREENLET_USE_CFRAME
-    // The weird thing is, we don't actually save this for an
-    // effect on the current greenlet, it's saved for an
-    // effect on the target greenlet. That is, we want
-    // continuity of this setting across the greenlet switch.
-    this->use_tracing = origin_tstate->cframe->use_tracing;
-#endif
+
 }

 void PythonState::set_initial_state(const PyThreadState* const tstate) G_NOEXCEPT

There is probably more that can be pulled out of greenlet (as I think the use_tracking value is only used for restoring the the value to cframes).

Obviously this patch needs version gating....

mdboom · 2023-04-27T14:05:25Z

I have updated this to compile again (thanks @tacaswell for the pointers there).

There is now a different set of test failures that I'm working through (as well as a number of segfaults in the log):

Test failures with 3.12

======================================================================
FAIL: test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_main (greenlet.tests.test_leaks.TestLeaks.test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_main)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 445, in test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_main
    self._check_untracked_memory_thread(deallocate_in_thread=False)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 422, in _check_untracked_memory_thread
    self.assertEqual(EXIT_COUNT[0], 0)
AssertionError: 398 != 0

======================================================================
FAIL: test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_thread (greenlet.tests.test_leaks.TestLeaks.test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_thread)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 439, in test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_thread
    self._check_untracked_memory_thread(deallocate_in_thread=True)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_leaks.py", line 417, in _check_untracked_memory_thread
    self.assertEqual(EXIT_COUNT[0], ITER * thread_func.ITER)
AssertionError: 398 != 20000

======================================================================
FAIL: test_trace_events_from_greenlet_func_sets_profiler (greenlet.tests.test_tracing.TestPythonTracing.test_trace_events_from_greenlet_func_sets_profiler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line 316, in wrapper
    return _RefCountChecker(self, method)(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line [294](https://github.com/python-greenlet/greenlet/actions/runs/4820657398/jobs/8585461014?pr=327#step:9:295), in __call__
    self._run_test(args, kwargs)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line 212, in _run_test
    self.function(self.testcase, *args, **kwargs)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_tracing.py", line 181, in test_trace_events_from_greenlet_func_sets_profiler
    self._check_trace_events_from_greenlet_sets_profiler(greenlet.greenlet(run),
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_tracing.py", line 163, in _check_trace_events_from_greenlet_sets_profiler
    self.assertEqual(tracer.actions, [
AssertionError: Lists differ: [('re[87 chars]run')] != [('re[87 chars]run'), ('call', 'tpt_callback'), ('return', 't[55 chars]__')]

Second list contains 4 additional elements.
First extra element 4:
('call', 'tpt_callback')

  [('return', '__enter__'),
   ('call', 'tpt_callback'),
   ('return', 'tpt_callback'),
-  ('return', 'run')]
?                   ^

+  ('return', 'run'),
?                   ^

+  ('call', 'tpt_callback'),
+  ('return', 'tpt_callback'),
+  ('call', '__exit__'),
+  ('c_call', '__exit__')]

======================================================================
FAIL: test_trace_events_from_greenlet_subclass_sets_profiler (greenlet.tests.test_tracing.TestPythonTracing.test_trace_events_from_greenlet_subclass_sets_profiler)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line [316](https://github.com/python-greenlet/greenlet/actions/runs/4820657398/jobs/8585461014?pr=327#step:9:317), in wrapper
    return _RefCountChecker(self, method)(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line 294, in __call__
    self._run_test(args, kwargs)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line 212, in _run_test
    self.function(self.testcase, *args, **kwargs)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_tracing.py", line 191, in test_trace_events_from_greenlet_subclass_sets_profiler
    self._check_trace_events_from_greenlet_sets_profiler(X(), tracer)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_tracing.py", line 163, in _check_trace_events_from_greenlet_sets_profiler
    self.assertEqual(tracer.actions, [
AssertionError: Lists differ: [('re[87 chars]run')] != [('re[87 chars]run'), ('call', 'tpt_callback'), ('return', 't[55 chars]__')]

Second list contains 4 additional elements.
First extra element 4:
('call', 'tpt_callback')

  [('return', '__enter__'),
   ('call', 'tpt_callback'),
   ('return', 'tpt_callback'),
-  ('return', 'run')]
?                   ^

+  ('return', 'run'),
?                   ^

+  ('call', 'tpt_callback'),
+  ('return', 'tpt_callback'),
+  ('call', '__exit__'),
+  ('c_call', '__exit__')]

======================================================================
FAIL: test_trace_events_multiple_greenlets_switching_siblings (greenlet.tests.test_tracing.TestPythonTracing.test_trace_events_multiple_greenlets_switching_siblings)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line 316, in wrapper
    return _RefCountChecker(self, method)(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line 294, in __call__
    self._run_test(args, kwargs)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/leakcheck.py", line 212, in _run_test
    self.function(self.testcase, *args, **kwargs)
  File "/home/runner/work/greenlet/greenlet/src/greenlet/tests/test_tracing.py", line 269, in test_trace_events_multiple_greenlets_switching_siblings
    self.assertEqual(tracer.actions, [
AssertionError: Lists differ: [('re[90 chars]run')] != [('re[90 chars]run'), ('call', 'tpt_callback'), ('return', 't[55 chars]__')]

Second list contains 4 additional elements.
First extra element 4:
('call', 'tpt_callback')

  [('return', '__enter__'),
   ('call', 'tpt_callback'),
   ('return', 'tpt_callback'),
-  ('c_call', 'g1_run')]
?                      ^

+  ('c_call', 'g1_run'),
?                      ^

+  ('call', 'tpt_callback'),
+  ('return', 'tpt_callback'),
+  ('call', '__exit__'),
+  ('c_call', '__exit__')]

----------------------------------------------------------------------

mdboom · 2023-04-27T14:52:38Z

The root cause of two of the test failures (test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_main, test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_thread) appears to be a "maximum recursion depth exceeded" exception. I can make it pass by reducing the number of iterations from 10000 to 199 (200 fails). I don't understand the details of the 3.12 changes well enough, but either the recursion depth really just is smaller, or greenlet is trying to control the recursion limit and that technique no longer works.

@markshannon: I wonder if you have thoughts?

diff --git a/src/greenlet/tests/test_leaks.py b/src/greenlet/tests/test_leaks.py
index 0ed43b0..c0bfec5 100644
--- a/src/greenlet/tests/test_leaks.py
+++ b/src/greenlet/tests/test_leaks.py
@@ -366,7 +366,7 @@ class TestLeaks(TestCase):
                 raise
             return 1

-        ITER = 10000
+        ITER = 199
         def run_it():
             glets = []
             for _ in range(ITER):

mdboom · 2023-04-27T22:03:02Z

The other two failing tests (test_trace_events_multiple_greenlets_switching_siblings and test_trace_events_from_greenlet_subclass_sets_profiler) pass when using a Python with PEP669 merged in (i.e. a commit after this one). This makes sense, because prior to that, we would still need to be reading the use_tracing member, but after this we do not.

I don't know if it's worth the effort to get a newer 3.12 in CI now or just wait for the next alpha release to automatically fix these two failures.

hugovk · 2023-04-27T22:13:22Z

It's pretty easy to test 3.12 nightly with https://github.com/deadsnakes/action (let me know you'd like help setting it up).

But it's also only a week and a half until the next pre-release on 2023-05-08, which as it happens is the first beta (https://peps.python.org/pep-0693/#release-schedule).

mdboom · 2023-04-27T23:04:03Z

It's pretty easy to test 3.12 nightly with https://github.com/deadsnakes/action (let me know you'd like help setting it up).

An earlier revision of this PR did exactly that -- we can always go back to it. But longer term, 3.12-dev is probably the one we want anyway.

mdboom · 2023-05-02T14:53:56Z

The root cause of two of the test failures (test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_main, test_untracked_memory_doesnt_increase_unfinished_thread_dealloc_in_thread) appears to be a "maximum recursion depth exceeded" exception.

I have updated this PR to fix these failures by getting/restoring both the Python and C stack limits.

With that, the only remaining failures are related to pre-PEP669, and should be resolved when dev-3.12 updates to a version with PEP669.

markshannon · 2023-05-12T09:32:14Z

The change looks reasonable from a CPython perspective, but PEP 669 changes the way that tracing works internally and I'm not sure that greenlet's model matches. The failure of 3.12-dev, ubuntu-20.04 suggests something isn't quite right.
Might we worth retesting now that #104387 has fixed some minor issues with tracing support.

mdboom · 2023-05-30T16:17:55Z

The failure of 3.12-dev, ubuntu-20.04 suggests something isn't quite right.

That was expected, since when this ran 3.12-dev didn't yet have PEP 669 merged. I have pushed a new commit to get it to re-run now, and it is passing.

PEP 669 changes the way that tracing works internally and I'm not sure that greenlet's model matches

Can you elaborate? Do you see this as fixable by porting to the PEP 669 API?

encukou · 2023-06-14T11:46:54Z

src/greenlet/greenlet_greenlet.hpp

-#if GREENLET_PY311
+#if GREENLET_PY312
+    this->py_recursion_depth = tstate->py_recursion_limit - tstate->py_recursion_remaining;
+    this->c_recursion_depth = tstate->py_recursion_limit - tstate->py_recursion_remaining;


Should this be C_RECURSION_LIMIT - tstate->c_recursion_remaining?

Co-authored-by: Andreas Motl <[email protected]>

jamadden · 2023-06-20T16:50:15Z

Thank you everyone for all your work on this! It is greatly appreciated. Tests are passing with 3.12.0b2, so I'm going to get this merged so I can make a pre-release of greenlet ASAP.

Switch to wheel.mk. 3.0.1 (2023-10-25) ================== - Fix a potential crash on Python 3.8 at interpreter shutdown time. This was a regression from earlier 3.0.x releases. Reported by Matt Wozniski in `issue 376 <https://github.com/python-greenlet/greenlet/issues/376>`_. 3.0.0 (2023-10-02) ================== - No changes from 3.0rc3 aside from the version number. 3.0.0rc3 (2023-09-12) ===================== - Fix an intermittent error during process termination on some platforms (GCC/Linux/libstdc++). 3.0.0rc2 (2023-09-09) ===================== - Fix some potential bugs (assertion failures and memory leaks) in previously-untested error handling code. In some cases, this means that the process will execute a controlled ``abort()`` after severe trouble when previously the process might have continued for some time with a corrupt state. It is unlikely those errors occurred in practice. - Fix some assertion errors and potential bugs with re-entrant switches. - Fix a potential crash when certain compilers compile greenlet with high levels of optimization. The symptom would be that switching to a greenlet for the first time immediately crashes. - Fix a potential crash when the callable object passed to the greenlet constructor (or set as the ``greenlet.run`` attribute) has a destructor attached to it that switches. Typically, triggering this issue would require an unlikely subclass of ``greenlet.greenlet``. - Python 3.11+: Fix rare switching errors that could occur when a garbage collection was triggered during the middle of a switch, and Python-level code in ``__del__`` or weakref callbacks switched to a different greenlet and ultimately switched back to the original greenlet. This often manifested as a ``SystemError``: "switch returned NULL without an exception set." For context on the fixes, see `gevent issue #1985 <https://github.com/gevent/gevent/issues/1985>`_. 3.0.0rc1 (2023-09-01) ===================== - Windows wheels are linked statically to the C runtime in an effort to prevent import errors on systems without the correct C runtime installed. It's not clear if this will make the situation better or worse, so please share your experiences in `issue 346 <https://github.com/python-greenlet/greenlet/issues/346>`_. Note that this only applies to the binary wheels found on PyPI. Building greenlet from source defaults to the shared library. Set the environment variable ``GREENLET_STATIC_RUNTIME=1`` at build time to change that. - Build binary wheels for Python 3.12 on macOS. - Fix compiling greenlet on a debug build of CPython 3.12. There is `one known issue <https://github.com/python-greenlet/greenlet/issues/368>`_ that leads to an interpreter crash on debug builds. - Python 3.12: Fix walking the frame stack of suspended greenlets. Previously accessing ``glet.gr_frame.f_back`` would crash due to `changes in CPython's undocumented internal frame handling <https://github.com/python/cpython/commit/1e197e63e21f77b102ff2601a549dda4b6439455>`_. Platforms --------- - Now, greenlet *may* compile and work on Windows ARM64 using llvm-mingw, but this is untested and unsupported. See `PR <https://github.com/python-greenlet/greenlet/pull/224>`_ by Adrian Vladu. - Now, greenlet *may* compile and work on LoongArch64 Linux systems, but this is untested and unsupported. See `PR 257 <https://github.com/python-greenlet/greenlet/pull/257/files>`_ by merore. Known Issues ------------ - There may be (very) subtle issues with tracing on Python 3.12, which has redesigned the entire tracing infrastructure. 3.0.0a1 (2023-06-21) ==================== - Build binary wheels for S390x Linux. See `PR 358 <https://github.com/python-greenlet/greenlet/pull/358>`_ from Steven Silvester. - Fix a rare crash on shutdown seen in uWSGI deployments. See `issue 330 <https://github.com/python-greenlet/greenlet/issues/330>`_ and `PR 356 <https://github.com/python-greenlet/greenlet/pull/356>`_ from Andrew Wason. - Make the platform-specific low-level C/assembly snippets stop using the ``register`` storage class. Newer versions of standards remove this storage class, and it has been generally ignored by many compilers for some time. See `PR 347 <https://github.com/python-greenlet/greenlet/pull/347>`_ from Khem Raj. - Add initial support for Python 3.12. See `issue <https://github.com/python-greenlet/greenlet/issues/323>`_ and `PR <https://github.com/python-greenlet/greenlet/pull/327>`_; thanks go to (at least) Michael Droettboom, Andreas Motl, Thomas A Caswell, raphaelauv, Hugo van Kemenade, Mark Shannon, and Petr Viktorin. - Remove support for end-of-life Python versions, including Python 2.7, Python 3.5 and Python 3.6. - Require a compiler that supports ``noinline`` directives. See `issue 271 <https://github.com/python-greenlet/greenlet/issues/266>`_. - Require a compiler that supports C++11.

mdboom force-pushed the python-312 branch from e015bad to 244faa6 Compare November 4, 2022 16:45

amotl reviewed Nov 15, 2022

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

.github/workflows/tests.yml Outdated Show resolved Hide resolved

.github/workflows/tests.yml Outdated Show resolved Hide resolved

amotl mentioned this pull request Nov 15, 2022

support for python3.12 #323

Closed

mdboom mentioned this pull request Nov 16, 2022

CI fails with 3.12 python/pyperformance#245

Closed

hugovk mentioned this pull request Jan 11, 2023

Skip 3.12-dev pending Greenlet support python/pyperformance#259

Closed

jamadden force-pushed the master branch from 120ed21 to b55ea9e Compare January 28, 2023 13:55

IdoKendo mentioned this pull request Jan 30, 2023

Update linter and add type checker IdoKendo/mysql-context-manager#4

Merged

hugovk mentioned this pull request Feb 5, 2023

Skip tests failing on Python 3.12 pending greenlet support python/pyperformance#264

Closed

mdboom force-pushed the python-312 branch from d9b39dd to 387dee0 Compare April 27, 2023 13:52

miketheman mentioned this pull request May 25, 2023

Python 3.12 Readiness tracker pypi/warehouse#13757

Closed

10 tasks

arkamar mentioned this pull request May 27, 2023

greenlet fails to build with Python 3.12.0b1 #357

Closed

pquentin mentioned this pull request Jun 9, 2023

Test Python 3.12 support sumerc/yappi#139

Closed

encukou reviewed Jun 14, 2023

View reviewed changes

Fix python-greenlet#323: Support Python 3.12

d92c0bc

mdboom and others added 11 commits June 20, 2023 11:25

Fix 3.12 testing

e0cb042

Try a different way to specify version

4e11ba9

Fix syntax

fb660bd

Use 3.12-dev instead of deadsnakes

dc50545

Co-authored-by: Andreas Motl <[email protected]>

Updates for PEP669

0013135

Fix Python 3.11

86b096e

Revert ubuntu-latest change

fec6461

Fix legacy versions

af1061e

Print precise Python version

6ac5d10

Fix recursion depth updating

22a8434

Insert blank line

3262da9

jamadden force-pushed the python-312 branch from 6bba85f to 3262da9 Compare June 20, 2023 16:27

Add change note for 3.12 support.

1c7bfee

jamadden merged commit 1db0fc6 into python-greenlet:master Jun 20, 2023

JacobCoffee mentioned this pull request Jun 22, 2023

Enhancement: Python 3.12 litestar-org/litestar#1862

Closed

8 tasks

SnoopJ mentioned this pull request Aug 25, 2023

Official support for Python 3.12 sopel-irc/sopel#2457

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #323: Support Python 3.12 #327

Fix #323: Support Python 3.12 #327

mdboom commented Nov 4, 2022

amotl left a comment

mdboom commented Nov 16, 2022

mdboom commented Nov 16, 2022

amotl commented Nov 21, 2022

tacaswell commented Jan 27, 2023

raphaelauv commented Mar 5, 2023

hugovk commented Mar 5, 2023

tacaswell commented Apr 3, 2023

tacaswell commented Apr 13, 2023

mdboom commented Apr 27, 2023

mdboom commented Apr 27, 2023

mdboom commented Apr 27, 2023

hugovk commented Apr 27, 2023

mdboom commented Apr 27, 2023

mdboom commented May 2, 2023

markshannon commented May 12, 2023

mdboom commented May 30, 2023

encukou Jun 14, 2023

jamadden commented Jun 20, 2023

Fix #323: Support Python 3.12 #327

Fix #323: Support Python 3.12 #327

Conversation

mdboom commented Nov 4, 2022

amotl left a comment

Choose a reason for hiding this comment

mdboom commented Nov 16, 2022

mdboom commented Nov 16, 2022

amotl commented Nov 21, 2022

tacaswell commented Jan 27, 2023

raphaelauv commented Mar 5, 2023

hugovk commented Mar 5, 2023

tacaswell commented Apr 3, 2023

tacaswell commented Apr 13, 2023

mdboom commented Apr 27, 2023

mdboom commented Apr 27, 2023

mdboom commented Apr 27, 2023

hugovk commented Apr 27, 2023

mdboom commented Apr 27, 2023

mdboom commented May 2, 2023

markshannon commented May 12, 2023

mdboom commented May 30, 2023

encukou Jun 14, 2023

Choose a reason for hiding this comment

jamadden commented Jun 20, 2023