release: 0.23.0

[davidhewitt]

This is a branch to prep the 0.23 release. I think

• This release has a lot of stuff in here already which is going to push users to do migration work thanks to the freethreaded support and also IntoPyObject. I am nervous about adding much more.
• Python 3.13.0 has been live a couple of weeks and it would be great for us to just ship as soon as we have passable freethreading support so that the downstream ecosystem can be unblocked.

I hope that we might be able to get this shipped a week from now, 1st November. My list of what needs to go in:

• Derive macro for IntoPyObject #4495
• deprecate IntoPy in favor or IntoPyObject #4618
• Declare free-threaded support for PyModule #4588
• require #[pyclass] to be Sync #4566
• ... and a fix for "lazy" state of PyErr ... has challenges #4584 (at least the thread safety) - done in make PyErrState thread-safe #4671

I think the design scope of those is relatively clear, even if that's still a fair bit of typing to do!

Note that this means that not everything which we listed in #4554 is making it in; I am sorry for the pieces we missed. I'd rather ship this, get feedback and start working on fixes while we also prep those next pieces for 0.24. I'd love it if we can do some smaller, gentler releases for 0.24 and the few after that.

... that said, one of the things not making it into 0.23 will be FromPyObject changes, which I think will make sense to get on with ASAP 👀

[davidhewitt]

With all items done, I have rebased this and curated the (massive) CHANGELOG entry. I've provisionally marked this for release on Saturday (9th Nov).

In the interim, I will try to pass eyes over the docs and improve what I can before release.

[davidhewitt]

I think we are still missing docs for migration particularly of #[pyclass] requiring Sync. I have had a lot of family burden this past week so I've run out of energy to write these myself the past few days.

I might have more energy tomorrow; it is possible that instead of releasing I will push a docs PR, and perhaps as long as there are some docs we can release on Sunday. I think it'd be pretty unfair on users if we released without that doc explaining the Sync changes.

It is also possible that I won't get around to the docs writing tomorrow, family situation will be ongoing for the foreseeable future. Will make a judgement call what to do then.

[davidhewitt]

I didn't have time & energy to make progress yesterday, unclear if today will be any better. I have a train journey tomorrow so I can probably get the docs written at least.

If #4216 (reply in thread) reproduces on main, we should probably fix before release. It'll probably be a simple fix to make the enum code work more like #[pyo3(get)], but will be potentially breaking so not shippable in a post-release patch.

[Icxolu]

I attempted a fix in #4694. It makes it work, but we should probably overhaul complex enums in the future again to better reuse our code.

[davidhewitt]

Thanks!

With that fix now merged, and I think at least some basic docs in #4695, I think I will rebase and release this tomorrow evening (12th Nov).

Tomorrow in working hours I will try updating pydantic-core to use this branch, which will be a last-minute smoke test that nothing horribly broken is lying waiting to be released. Assuming that all goes fine, I see no reason to wait any further!

[ngoldbaum]

FWIW I was looking at a few downstream projects today as well. It looks like the rust parts of cryptography and all of rpds-py should be straightforward to update. The other projects are on older PyO3 versions or are using the FFI to define extensions. I haven't yet run into a downstream error about something not being Sync.

[davidhewitt]

Unfortunately the pydantic-core upgrade didn't go very well;

• hit some recursion errors with type inference trying to evaluate IntoPyObject for &&&&&&&&&&&&&&[...]&&Bound. Not clear to me why yet and if it will be a recurrent problem for users to deal with which might require some changes.
• my family and I are sick again, so I spent most the day sleeping

I will try to look again ASAP when I have some more energy, others are also welcome to investigate.

[alex]

Feel better!

…

On Tue, Nov 12, 2024, 1:49 PM David Hewitt ***@***.***> wrote: Unfortunately the pydantic-core upgrade didn't go very well; - hit some recursion errors with type inference trying to evaluate IntoPyObject for &&&&&&&&&&&&&&[...]&&Bound. Not clear to me why yet and if it will be a recurrent problem for users to deal with which might require some changes. - my family and I are sick again, so I spent most the day sleeping I will try to look again ASAP when I have some more energy, others are also welcome to investigate. — Reply to this email directly, view it on GitHub <#4651 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBB2DP4FDG2DF2D3ZV32AJEVPAVCNFSM6AAAAABQTRY646VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZRGMYTCNZWGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ngoldbaum]

rpds.py was pretty straightforward.

[ngoldbaum]

I noticed yesterday in rpds.py and today in cryptography that I'm making a lot of changes along the lines of this to migrate from ToPyObject to IntoPyObject:

@@ -60,16 +60,19 @@ fn decode_dss_signature(
 ) -> Result<pyo3::PyObject, CryptographyError> {
     let sig = asn1::parse_single::<DssSignature<'_>>(data)?;

-    Ok((
+    (
         big_byte_slice_to_py_int(py, sig.r.as_bytes())?,
         big_byte_slice_to_py_int(py, sig.s.as_bytes())?,
     )
-        .to_object(py))
+        .into_pyobject(py)
+        .map_err(Into::into)
+        .map(pyo3::BoundObject::into_any)
+        .map(pyo3::BoundObject::unbind)
 }

Maybe the docs and migration guide could point to BoundObject as a useful thing for this sort of operation? Or am I doing something weird here?

[Icxolu]

That seems reasonable to me. I believe in this specific case there are no generic involved, so you get a concrete type back (PyResult<Bound<'py, PyTuple>>). So I think you could also write this as

(...).into_pyobject(py)?.into_any().unbind()

here (or if we want to use map, directly referring to Bounds methods is also possible). BoundObject comes more into play it the type you're converting is a generic. Sometimes it can also be useful to propagate the target type up to get stronger typing throughout the code, but that's a decision to be taken on a case by case basis I would say.

[ngoldbaum]

Ah I see how I don't need to use map since I have a Result - there were a few places in rpds.py where it really was a map and BoundObject makes more sense. I'll try to write something up and ping you to review it.

[davidhewitt]

👍 I do think there is going to be some common helpers / trait we can put into PyO3 to make those kinds of patterns easier, but that can be a post-release QoL improvement.

I'm a bit brighter today so I'm trying to continue to push forward with final testing on pydantic-core. I am going to try to find a minimal reproduction for the recursion overflow. In the meanwhile I've also found #4701.

The WIP branch is at pydantic/pydantic-core#1450, if I silence deprecation warnings the upgrade is "easy", there was a couple of Sync fixups to classes. In particular it's ToPyObject -> IntoPyObject which seems to be the only challenging bit at least.

[ngoldbaum]

Should this work?

error[E0277]: can't compare `pyo3::Bound<'_, PyInt>` with `pyo3::Bound<'_, PyInt>`
   --> src/rust/src/backend/dh.rs:475:31
    |
475 |         Ok(self.y.bind(py).eq(other.y.bind(py))?
    |                            -- ^^^^^^^^^^^^^^^^ no implementation for `pyo3::Bound<'_, PyInt> == pyo3::Bound<'_, PyInt>`
    |                            |
    |                            required by a bound introduced by this call
    |
    = help: the trait `PartialEq<pyo3::Bound<'_, PyInt>>` is not implemented for `pyo3::Bound<'_, PyInt>`

I also see similar issues caused by PartialEq not being implemented between PyInt and PyAny.

[Icxolu]

I think you want use PyAnyMethods::eq and not PartialEq::eq? Normally PyAnyMethods should work through auto-deref of Bound, but in this case it seems to need an explicit deref (I assume because of the naming collision) (**self.y.bind(py)).eq(other.y.bind(py)

[davidhewitt]

I managed to capture a minimal repro for the bound overflow: #4702

[davidhewitt]

The discussion in #4702 has convinced me the current situation is ok; if users have trouble upgrading we can point them to it, and work on QoL improvements as patch releases / future releases.

Let's proceed to get docs merged and release 👍

[davidhewitt]

Ok, everything merged in, I will merge this into main now, assuming that succeeds I will then release!

[davidhewitt]

Plz hold, #4710

[davidhewitt]

https://github.com/PyO3/pyo3/releases/tag/v0.23.0 🎉