begin drafting Bound<T> migration guide

guide/src/migration.md

@@ -5,6 +5,40 @@ For a detailed list of all changes, see the [CHANGELOG](changelog.md).
 
 ## from 0.20.* to 0.21
 
+PyO3 0.21 introduces a new `Bound<'py, T>` smart pointer which replaces the existing "GIL Refs" API to interact with Python objects. For example, in PyO3 0.20 the reference `&'py PyAny` would be used to interact with Python objects. In PyO3 0.21 the updated type is `Bound<'py, PyAny>`. Making this change moves Rust ownership semantics out of PyO3's internals and into user code. This change fixes [a known soundness edge case of interaction with gevent](https://github.com/PyO3/pyo3/issues/3668) as well as improves CPU and [memory performance](https://github.com/PyO3/pyo3/issues/1056). For a full history of discussion see https://github.com/PyO3/pyo3/issues/3382.
+
+The "GIL Ref" `&'py PyAny` and similar types such as `&'py PyDict` continue to be available as a deprecated API. Due to the advantages of the new API it is advised that all users make the effort to upgrade as soon as possible.
+
+In addition to the major API type overhaul, PyO3 has needed to make a few small breaking adjustments to other APIs to close correctness and soundness gaps.
+
+The recommended steps to update to PyO3 0.21 is as follows:
+  1. Add compatibility code to convert new `Bound<T>` smart pointers back to GIL Refs
+  2. Fix all other PyO3 0.21 migration steps
+  3. Incrementally replace usage of the GIL Refs API by `Bound<T>` smart pointers
+
+The following sections are laid out in this order.
+
+### APIs which now return `Bound<T>` instead of GIL Refs
+
+To make usage of `Bound<T>` the canonical way to use PyO3, a number of APIs have been changed to return `Bound<T>` smart pointers. For example, Python type constructors such as `PyList::new` now return `Bound<'py, PyList>` instead of `&'py PyList`.
+
+As a temporary backwards-compatibility shim, it is recommended to use `Bound::into_gil_ref` to convert these types back into GIL Refs to fix compilation errors while updating PyO3.
+
+Before:
+
+```rust,ignore
+let list: &PyList = PyList::new(py, &[1, 2, 3]);
+```
+
+After:
+
+```rust
+let list: &PyList = PyList::new(py, &[1, 2, 3]).into_gil_ref();
+```
+
+The exhaustive list of APIs migrated is:
+ - `PyTzInfoAccess::get_tzinfo`
+
 ### `PyTypeInfo` and `PyTryFrom` have been adjusted
 
 The `PyTryFrom` trait has aged poorly, its [`try_from`] method now conflicts with `try_from` in the 2021 edition prelude. A lot of its functionality was also duplicated with `PyTypeInfo`.
@@ -196,6 +230,10 @@ impl PyClassAsyncIter {
 
 `PyType::name` has been renamed to `PyType::qualname` to indicate that it does indeed return the [qualified name](https://docs.python.org/3/glossary.html#term-qualified-name), matching the `__qualname__` attribute. The newly added `PyType::name` yields the full name including the module name now which corresponds to `__module__.__name__` on the level of attributes.
 
+### Migrating from the GIL-Refs API to `Bound<T>`
+
+TODO
+
 ## from 0.19.* to 0.20
 
 ### Drop support for older technologies

newsfragments/3692.added.md

@@ -0,0 +1 @@
+Add `PyNativeType::as_bound` to convert "GIL refs" to the new `Bound` smart pointer.

newsfragments/3692.changed.md

@@ -0,0 +1 @@
+Include `PyNativeType` in `pyo3::prelude`.

src/err/mod.rs

@@ -7,7 +7,7 @@ use crate::{
     exceptions::{self, PyBaseException},
     ffi,
 };
-use crate::{IntoPy, Py, PyAny, PyObject, Python, ToPyObject};
+use crate::{IntoPy, Py, PyAny, PyNativeType, PyObject, Python, ToPyObject};
 use std::borrow::Cow;
 use std::cell::UnsafeCell;
 use std::ffi::CString;
@@ -811,7 +811,7 @@ impl<'a> std::error::Error for PyDowncastError<'a> {}
 
 impl<'a> std::fmt::Display for PyDowncastError<'a> {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
-        display_downcast_error(f, Bound::borrowed_from_gil_ref(&self.from), &self.to)
+        display_downcast_error(f, self.from.as_bound(), &self.to)
     }
 }
 

src/instance.rs

@@ -27,6 +27,16 @@ pub unsafe trait PyNativeType: Sized {
     /// The form of this which is stored inside a `Py<T>` smart pointer.
     type AsRefSource: HasPyGilRef<AsRefTarget = Self>;
 
+    /// Cast `&self` to a `Bound` smart pointer.
+    ///
+    /// This is available as a migration tool to adjust code from the deprecated "GIL Refs"
+    /// API to the `Bound` smart pointer API.
+    fn as_bound<'a, 'py>(self: &'a &'py Self) -> &'a Bound<'py, Self::AsRefSource> {
+        // Safety: &'py Self is expected to be a Python pointer,
+        // so &'a &'py Self has the same layout as &'a Bound<'py, T>
+        unsafe { std::mem::transmute(self) }
+    }
+
     /// Returns a GIL marker constrained to the lifetime of this type.
     #[inline]
     fn py(&self) -> Python<'_> {
@@ -172,18 +182,6 @@ impl<'py, T> Bound<'py, T> {
         self.into_non_null().as_ptr()
     }
 
-    /// Internal helper to convert e.g. &'a &'py PyDict to &'a Bound<'py, PyDict> for
-    /// backwards-compatibility during migration to removal of pool.
-    #[doc(hidden)] // public and doc(hidden) to use in examples and tests for now
-    pub fn borrowed_from_gil_ref<'a, U>(gil_ref: &'a &'py U) -> &'a Self
-    where
-        U: PyNativeType<AsRefSource = T>,
-    {
-        // Safety: &'py T::AsRefTarget is expected to be a Python pointer,
-        // so &'a &'py T::AsRefTarget has the same layout as &'a Bound<'py, T>
-        unsafe { std::mem::transmute(gil_ref) }
-    }
-
     /// Internal helper to get to pool references for backwards compatibility
     #[doc(hidden)] // public and doc(hidden) to use in examples and tests for now
     pub fn as_gil_ref(&'py self) -> &'py T::AsRefTarget
@@ -1351,7 +1349,7 @@ where
 {
     /// Extracts `Self` from the source `PyObject`.
     fn extract(ob: &'a PyAny) -> PyResult<Self> {
-        Bound::borrowed_from_gil_ref(&ob)
+        ob.as_bound()
             .downcast()
             .map(Clone::clone)
             .map_err(Into::into)
@@ -1470,7 +1468,7 @@ impl PyObject {
 mod tests {
     use super::{Bound, Py, PyObject};
     use crate::types::{PyDict, PyString};
-    use crate::{PyAny, PyResult, Python, ToPyObject};
+    use crate::{PyAny, PyNativeType, PyResult, Python, ToPyObject};
 
     #[test]
     fn test_call0() {
@@ -1597,8 +1595,7 @@ a = A()
     #[test]
     fn test_py2_into_py_object() {
         Python::with_gil(|py| {
-            let instance: Bound<'_, PyAny> =
-                Bound::borrowed_from_gil_ref(&py.eval("object()", None, None).unwrap()).clone();
+            let instance = py.eval("object()", None, None).unwrap().as_bound().clone();
             let ptr = instance.as_ptr();
             let instance: PyObject = instance.clone().into();
             assert_eq!(instance.as_ptr(), ptr);

src/prelude.rs

@@ -17,6 +17,7 @@ pub use crate::marker::Python;
 pub use crate::pycell::{PyCell, PyRef, PyRefMut};
 pub use crate::pyclass_init::PyClassInitializer;
 pub use crate::types::{PyAny, PyModule};
+pub use crate::PyNativeType;
 
 #[cfg(feature = "macros")]
 pub use pyo3_macros::{pyclass, pyfunction, pymethods, pymodule, FromPyObject};