diff --git a/guide/src/SUMMARY.md b/guide/src/SUMMARY.md index 4c22c26f587..f1e3eccea8a 100644 --- a/guide/src/SUMMARY.md +++ b/guide/src/SUMMARY.md @@ -6,24 +6,25 @@ - [Getting started](getting-started.md) - [Using Rust from Python](rust-from-python.md) - - [Python modules](module.md) - - [Python functions](function.md) - - [Function signatures](function/signature.md) - - [Error handling](function/error-handling.md) - - [Python classes](class.md) - - [Class customizations](class/protocols.md) - - [Basic object customization](class/object.md) - - [Emulating numeric types](class/numeric.md) - - [Emulating callable objects](class/call.md) + - [Python modules](module.md) + - [Python functions](function.md) + - [Function signatures](function/signature.md) + - [Error handling](function/error-handling.md) + - [Python classes](class.md) + - [Class customizations](class/protocols.md) + - [Basic object customization](class/object.md) + - [Emulating numeric types](class/numeric.md) + - [Emulating callable objects](class/call.md) - [Calling Python from Rust](python-from-rust.md) - - [Python object types](types.md) - - [Python exceptions](exception.md) - - [Calling Python functions](python-from-rust/function-calls.md) - - [Executing existing Python code](python-from-rust/calling-existing-code.md) + - [Python object types](types.md) + - [Python exceptions](exception.md) + - [Calling Python functions](python-from-rust/function-calls.md) + - [Executing existing Python code](python-from-rust/calling-existing-code.md) - [Type conversions](conversions.md) - - [Mapping of Rust types to Python types](conversions/tables.md) - - [Conversion traits](conversions/traits.md) + - [Mapping of Rust types to Python types](conversions/tables.md) + - [Conversion traits](conversions/traits.md) - [Using `async` and `await`](async-await.md) + - [Awaiting Python awaitables](async-await/awaiting_python_awaitables) - [Parallelism](parallelism.md) - [Debugging](debugging.md) - [Features reference](features.md) @@ -31,10 +32,10 @@ - [Performance](performance.md) - [Advanced topics](advanced.md) - [Building and distribution](building-and-distribution.md) - - [Supporting multiple Python versions](building-and-distribution/multiple-python-versions.md) + - [Supporting multiple Python versions](building-and-distribution/multiple-python-versions.md) - [Useful crates](ecosystem.md) - - [Logging](ecosystem/logging.md) - - [Using `async` and `await`](ecosystem/async-await.md) + - [Logging](ecosystem/logging.md) + - [Using `async` and `await`](ecosystem/async-await.md) - [FAQ and troubleshooting](faq.md) --- diff --git a/guide/src/async-await/awaiting_python_awaitables.md b/guide/src/async-await/awaiting_python_awaitables.md new file mode 100644 index 00000000000..febecc6a286 --- /dev/null +++ b/guide/src/async-await/awaiting_python_awaitables.md @@ -0,0 +1,62 @@ +# Awaiting Python awaitables + +Python awaitable can be awaited on Rust side +using [`await_in_coroutine`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/function.await_in_coroutine). + +```rust +# # ![allow(dead_code)] +# #[cfg(feature = "experimental-async")] { +use pyo3::{prelude::*, coroutine::await_in_coroutine}; + +#[pyfunction] +async fn wrap_awaitable(awaitable: PyObject) -> PyResult { + Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?.await +} +# } +``` + +Behind the scene, `await_in_coroutine` calls the `__await__` method of the Python awaitable (or `__iter__` for +generator-based coroutine). + +## Restrictions + +As the name suggests, `await_in_coroutine` resulting future can only be awaited in coroutine context. Otherwise, it +panics. + +```rust +# # ![allow(dead_code)] +# #[cfg(feature = "experimental-async")] { +use pyo3::{prelude::*, coroutine::await_in_coroutine}; + +#[pyfunction] +fn block_on(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::executor::block_on(future) // ERROR: PyFuture must be awaited in coroutine context +} +# } +``` + +The future must also be the only one to be awaited at a time; it means that it's forbidden to await it in a `select!`. +Otherwise, it panics. + +```rust +# # ![allow(dead_code)] +# #[cfg(feature = "experimental-async")] { +use futures::FutureExt; +use pyo3::{prelude::*, coroutine::await_in_coroutine}; + +#[pyfunction] +async fn select(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::select_biased! { + _ = std::future::pending::<()>().fuse() => unreachable!(), + res = future.fuse() => res, // ERROR: Python awaitable mixed with Rust future + } +} +# } +``` + +These restrictions exist because awaiting a `await_in_coroutine` future strongly binds it to the +enclosing coroutine. The coroutine will then delegate its `send`/`throw`/`close` methods to the +awaited future. If it was awaited in a `select!`, `Coroutine::send` would no able to know if +the value passed would have to be delegated or not. diff --git a/newsfragments/3610.added.md b/newsfragments/3610.added.md new file mode 100644 index 00000000000..3b1493c29c0 --- /dev/null +++ b/newsfragments/3610.added.md @@ -0,0 +1 @@ +Add `#[pyo3(allow_threads)]` to release the GIL in (async) functions \ No newline at end of file diff --git a/newsfragments/3611.added.md b/newsfragments/3611.added.md new file mode 100644 index 00000000000..75aa9aee8fd --- /dev/null +++ b/newsfragments/3611.added.md @@ -0,0 +1 @@ +Add `coroutine::await_in_coroutine` to await awaitables in coroutine context diff --git a/pyo3-ffi/src/abstract_.rs b/pyo3-ffi/src/abstract_.rs index b5bf9cc3d35..3cc5cd346ad 100644 --- a/pyo3-ffi/src/abstract_.rs +++ b/pyo3-ffi/src/abstract_.rs @@ -129,7 +129,11 @@ extern "C" { pub fn PyIter_Next(arg1: *mut PyObject) -> *mut PyObject; #[cfg(all(not(PyPy), Py_3_10))] #[cfg_attr(PyPy, link_name = "PyPyIter_Send")] - pub fn PyIter_Send(iter: *mut PyObject, arg: *mut PyObject, presult: *mut *mut PyObject); + pub fn PyIter_Send( + iter: *mut PyObject, + arg: *mut PyObject, + presult: *mut *mut PyObject, + ) -> c_int; #[cfg_attr(PyPy, link_name = "PyPyNumber_Check")] pub fn PyNumber_Check(o: *mut PyObject) -> c_int; diff --git a/pyo3-macros-backend/src/attributes.rs b/pyo3-macros-backend/src/attributes.rs index e91b3b8d9a2..000ec4b5979 100644 --- a/pyo3-macros-backend/src/attributes.rs +++ b/pyo3-macros-backend/src/attributes.rs @@ -9,6 +9,7 @@ use syn::{ }; pub mod kw { + syn::custom_keyword!(allow_threads); syn::custom_keyword!(annotation); syn::custom_keyword!(attribute); syn::custom_keyword!(cancel_handle); diff --git a/pyo3-macros-backend/src/method.rs b/pyo3-macros-backend/src/method.rs index 982cf62946e..49448c017f1 100644 --- a/pyo3-macros-backend/src/method.rs +++ b/pyo3-macros-backend/src/method.rs @@ -6,6 +6,7 @@ use syn::{ext::IdentExt, spanned::Spanned, Ident, Result}; use crate::utils::Ctx; use crate::{ + attributes, attributes::{FromPyWithAttribute, TextSignatureAttribute, TextSignatureAttributeValue}, deprecations::{Deprecation, Deprecations}, params::{impl_arg_params, Holders}, @@ -379,6 +380,7 @@ pub struct FnSpec<'a> { pub asyncness: Option, pub unsafety: Option, pub deprecations: Deprecations<'a>, + pub allow_threads: Option, } pub fn parse_method_receiver(arg: &syn::FnArg) -> Result { @@ -416,6 +418,7 @@ impl<'a> FnSpec<'a> { text_signature, name, signature, + allow_threads, .. } = options; @@ -461,6 +464,7 @@ impl<'a> FnSpec<'a> { asyncness: sig.asyncness, unsafety: sig.unsafety, deprecations, + allow_threads, }) } @@ -603,6 +607,21 @@ impl<'a> FnSpec<'a> { bail_spanned!(name.span() => "`cancel_handle` may only be specified once"); } } + if let Some(FnArg::Py(py_arg)) = self + .signature + .arguments + .iter() + .find(|arg| matches!(arg, FnArg::Py(_))) + { + ensure_spanned!( + self.asyncness.is_none(), + py_arg.ty.span() => "GIL token cannot be passed to async function" + ); + ensure_spanned!( + self.allow_threads.is_none(), + py_arg.ty.span() => "GIL cannot be held in function annotated with `allow_threads`" + ); + } if self.asyncness.is_some() { ensure_spanned!( @@ -612,8 +631,21 @@ impl<'a> FnSpec<'a> { } let rust_call = |args: Vec, holders: &mut Holders| { - let mut self_arg = || self.tp.self_arg(cls, ExtractErrorMode::Raise, holders, ctx); - + let allow_threads = self.allow_threads.is_some(); + let mut self_arg = || { + let self_arg = self.tp.self_arg(cls, ExtractErrorMode::Raise, holders, ctx); + if self_arg.is_empty() { + self_arg + } else { + let self_checker = holders.push_gil_refs_checker(self_arg.span()); + quote! { + #pyo3_path::impl_::deprecations::inspect_type(#self_arg &#self_checker), + } + } + }; + let arg_names = (0..args.len()) + .map(|i| format_ident!("arg_{}", i)) + .collect::>(); let call = if self.asyncness.is_some() { let throw_callback = if cancel_handle.is_some() { quote! { Some(__throw_callback) } @@ -625,9 +657,6 @@ impl<'a> FnSpec<'a> { Some(cls) => quote!(Some(<#cls as #pyo3_path::PyTypeInfo>::NAME)), None => quote!(None), }; - let arg_names = (0..args.len()) - .map(|i| format_ident!("arg_{}", i)) - .collect::>(); let future = match self.tp { FnType::Fn(SelfType::Receiver { mutable: false, .. }) => { quote! {{ @@ -645,18 +674,7 @@ impl<'a> FnSpec<'a> { } _ => { let self_arg = self_arg(); - if self_arg.is_empty() { - quote! { function(#(#args),*) } - } else { - let self_checker = holders.push_gil_refs_checker(self_arg.span()); - quote! { - function( - // NB #self_arg includes a comma, so none inserted here - #pyo3_path::impl_::deprecations::inspect_type(#self_arg &#self_checker), - #(#args),* - ) - } - } + quote!(function(#self_arg #(#args),*)) } }; let mut call = quote! {{ @@ -665,6 +683,7 @@ impl<'a> FnSpec<'a> { #pyo3_path::intern!(py, stringify!(#python_name)), #qualname_prefix, #throw_callback, + #allow_threads, async move { #pyo3_path::impl_::wrap::OkWrap::wrap(future.await) }, ) }}; @@ -676,20 +695,21 @@ impl<'a> FnSpec<'a> { }}; } call - } else { + } else if allow_threads { let self_arg = self_arg(); - if self_arg.is_empty() { - quote! { function(#(#args),*) } + let (self_arg_name, self_arg_decl) = if self_arg.is_empty() { + (quote!(), quote!()) } else { - let self_checker = holders.push_gil_refs_checker(self_arg.span()); - quote! { - function( - // NB #self_arg includes a comma, so none inserted here - #pyo3_path::impl_::deprecations::inspect_type(#self_arg &#self_checker), - #(#args),* - ) - } - } + (quote!(__self,), quote! { let (__self,) = (#self_arg); }) + }; + quote! {{ + #self_arg_decl + #(let #arg_names = #args;)* + py.allow_threads(|| function(#self_arg_name #(#arg_names),*)) + }} + } else { + let self_arg = self_arg(); + quote!(function(#self_arg #(#args),*)) }; quotes::map_result_into_ptr(quotes::ok_wrap(call, ctx), ctx) }; diff --git a/pyo3-macros-backend/src/pyclass.rs b/pyo3-macros-backend/src/pyclass.rs index d9c84655b42..8ee1821e954 100644 --- a/pyo3-macros-backend/src/pyclass.rs +++ b/pyo3-macros-backend/src/pyclass.rs @@ -1174,6 +1174,7 @@ fn complex_enum_struct_variant_new<'a>( asyncness: None, unsafety: None, deprecations: Deprecations::new(ctx), + allow_threads: None, }; crate::pymethod::impl_py_method_def_new(&variant_cls_type, &spec, ctx) @@ -1199,6 +1200,7 @@ fn complex_enum_variant_field_getter<'a>( asyncness: None, unsafety: None, deprecations: Deprecations::new(ctx), + allow_threads: None, }; let property_type = crate::pymethod::PropertyType::Function { diff --git a/pyo3-macros-backend/src/pyfunction.rs b/pyo3-macros-backend/src/pyfunction.rs index 7c355533b83..f65a0597b58 100644 --- a/pyo3-macros-backend/src/pyfunction.rs +++ b/pyo3-macros-backend/src/pyfunction.rs @@ -91,6 +91,7 @@ pub struct PyFunctionOptions { pub signature: Option, pub text_signature: Option, pub krate: Option, + pub allow_threads: Option, } impl Parse for PyFunctionOptions { @@ -99,7 +100,8 @@ impl Parse for PyFunctionOptions { while !input.is_empty() { let lookahead = input.lookahead1(); - if lookahead.peek(attributes::kw::name) + if lookahead.peek(attributes::kw::allow_threads) + || lookahead.peek(attributes::kw::name) || lookahead.peek(attributes::kw::pass_module) || lookahead.peek(attributes::kw::signature) || lookahead.peek(attributes::kw::text_signature) @@ -121,6 +123,7 @@ impl Parse for PyFunctionOptions { } pub enum PyFunctionOption { + AllowThreads(attributes::kw::allow_threads), Name(NameAttribute), PassModule(attributes::kw::pass_module), Signature(SignatureAttribute), @@ -131,7 +134,9 @@ pub enum PyFunctionOption { impl Parse for PyFunctionOption { fn parse(input: ParseStream<'_>) -> Result { let lookahead = input.lookahead1(); - if lookahead.peek(attributes::kw::name) { + if lookahead.peek(attributes::kw::allow_threads) { + input.parse().map(PyFunctionOption::AllowThreads) + } else if lookahead.peek(attributes::kw::name) { input.parse().map(PyFunctionOption::Name) } else if lookahead.peek(attributes::kw::pass_module) { input.parse().map(PyFunctionOption::PassModule) @@ -171,6 +176,7 @@ impl PyFunctionOptions { } for attr in attrs { match attr { + PyFunctionOption::AllowThreads(allow_threads) => set_option!(allow_threads), PyFunctionOption::Name(name) => set_option!(name), PyFunctionOption::PassModule(pass_module) => set_option!(pass_module), PyFunctionOption::Signature(signature) => set_option!(signature), @@ -198,6 +204,7 @@ pub fn impl_wrap_pyfunction( ) -> syn::Result { check_generic(&func.sig)?; let PyFunctionOptions { + allow_threads, pass_module, name, signature, @@ -247,6 +254,7 @@ pub fn impl_wrap_pyfunction( python_name, signature, text_signature, + allow_threads, asyncness: func.sig.asyncness, unsafety: func.sig.unsafety, deprecations: Deprecations::new(ctx), diff --git a/pyo3-macros/src/lib.rs b/pyo3-macros/src/lib.rs index 64756a1c73b..65456334ac1 100644 --- a/pyo3-macros/src/lib.rs +++ b/pyo3-macros/src/lib.rs @@ -121,6 +121,7 @@ pub fn pymethods(attr: TokenStream, input: TokenStream) -> TokenStream { /// | `#[pyo3(name = "...")]` | Defines the name of the function in Python. | /// | `#[pyo3(text_signature = "...")]` | Defines the `__text_signature__` attribute of the function in Python. | /// | `#[pyo3(pass_module)]` | Passes the module containing the function as a `&PyModule` first argument to the function. | +/// | `#[pyo3(allow_threads)]` | Release the GIL in the function body, or each time the returned future is polled for `async fn` | /// /// For more on exposing functions see the [function section of the guide][1]. /// diff --git a/src/coroutine.rs b/src/coroutine.rs index f2feab4af16..47c0378e6e0 100644 --- a/src/coroutine.rs +++ b/src/coroutine.rs @@ -11,28 +11,74 @@ use std::{ use pyo3_macros::{pyclass, pymethods}; use crate::{ - coroutine::{cancel::ThrowCallback, waker::AsyncioWaker}, - exceptions::{PyAttributeError, PyRuntimeError, PyStopIteration}, + coroutine::waker::CoroutineWaker, + exceptions::{PyAttributeError, PyGeneratorExit, PyRuntimeError, PyStopIteration}, + marker::Ungil, panic::PanicException, - types::{string::PyStringMethods, PyIterator, PyString}, - Bound, IntoPy, Py, PyAny, PyErr, PyObject, PyResult, Python, + types::{string::PyStringMethods, PyString}, + IntoPy, Py, PyErr, PyObject, PyResult, Python, }; -pub(crate) mod cancel; +mod asyncio; +mod awaitable; +mod cancel; mod waker; -pub use cancel::CancelHandle; +pub use awaitable::await_in_coroutine; +pub use cancel::{CancelHandle, ThrowCallback}; const COROUTINE_REUSED_ERROR: &str = "cannot reuse already awaited coroutine"; +pub(crate) enum CoroOp { + Send(PyObject), + Throw(PyObject), + Close, +} + +trait CoroutineFuture: Send { + fn poll(self: Pin<&mut Self>, py: Python<'_>, waker: &Waker) -> Poll>; +} + +impl CoroutineFuture for F +where + F: Future> + Send, + T: IntoPy + Send, + E: Into + Send, +{ + fn poll(self: Pin<&mut Self>, py: Python<'_>, waker: &Waker) -> Poll> { + self.poll(&mut Context::from_waker(waker)) + .map_ok(|obj| obj.into_py(py)) + .map_err(Into::into) + } +} + +struct AllowThreads { + future: F, +} + +impl CoroutineFuture for AllowThreads +where + F: Future> + Send + Ungil, + T: IntoPy + Send + Ungil, + E: Into + Send + Ungil, +{ + fn poll(self: Pin<&mut Self>, py: Python<'_>, waker: &Waker) -> Poll> { + // SAFETY: future field is pinned when self is + let future = unsafe { self.map_unchecked_mut(|a| &mut a.future) }; + py.allow_threads(|| future.poll(&mut Context::from_waker(waker))) + .map_ok(|obj| obj.into_py(py)) + .map_err(Into::into) + } +} + /// Python coroutine wrapping a [`Future`]. #[pyclass(crate = "crate")] pub struct Coroutine { name: Option>, qualname_prefix: Option<&'static str>, throw_callback: Option, - future: Option> + Send>>>, - waker: Option>, + future: Option>>, + waker: Option>, } impl Coroutine { @@ -46,79 +92,76 @@ impl Coroutine { name: Option>, qualname_prefix: Option<&'static str>, throw_callback: Option, + allow_threads: bool, future: F, ) -> Self where - F: Future> + Send + 'static, - T: IntoPy, - E: Into, + F: Future> + Send + Ungil + 'static, + T: IntoPy + Send + Ungil, + E: Into + Send + Ungil, { - let wrap = async move { - let obj = future.await.map_err(Into::into)?; - // SAFETY: GIL is acquired when future is polled (see `Coroutine::poll`) - Ok(obj.into_py(unsafe { Python::assume_gil_acquired() })) - }; Self { name, qualname_prefix, throw_callback, - future: Some(Box::pin(wrap)), + future: Some(if allow_threads { + Box::pin(AllowThreads { future }) + } else { + Box::pin(future) + }), waker: None, } } - fn poll(&mut self, py: Python<'_>, throw: Option) -> PyResult { + fn poll_inner(&mut self, py: Python<'_>, mut op: CoroOp) -> PyResult { // raise if the coroutine has already been run to completion let future_rs = match self.future { Some(ref mut fut) => fut, None => return Err(PyRuntimeError::new_err(COROUTINE_REUSED_ERROR)), }; - // reraise thrown exception it - match (throw, &self.throw_callback) { - (Some(exc), Some(cb)) => cb.throw(exc), - (Some(exc), None) => { - self.close(); - return Err(PyErr::from_value_bound(exc.into_bound(py))); - } - (None, _) => {} + // if the future is not pending on a Python awaitable, + // execute throw callback or complete on close + if !matches!(self.waker, Some(ref w) if w.is_delegated(py)) { + match op { + send @ CoroOp::Send(_) => op = send, + CoroOp::Throw(exc) => match &self.throw_callback { + Some(cb) => { + cb.throw(exc.clone_ref(py)); + op = CoroOp::Send(py.None()); + } + None => return Err(PyErr::from_value_bound(exc.into_bound(py))), + }, + CoroOp::Close => return Err(PyGeneratorExit::new_err(py.None())), + }; } // create a new waker, or try to reset it in place if let Some(waker) = self.waker.as_mut().and_then(Arc::get_mut) { - waker.reset(); + waker.reset(op); } else { - self.waker = Some(Arc::new(AsyncioWaker::new())); + self.waker = Some(Arc::new(CoroutineWaker::new(op))); } - let waker = Waker::from(self.waker.clone().unwrap()); - // poll the Rust future and forward its results if ready + // poll the future and forward its results if ready; otherwise, yield from waker // polling is UnwindSafe because the future is dropped in case of panic - let poll = || future_rs.as_mut().poll(&mut Context::from_waker(&waker)); + let waker = Waker::from(self.waker.clone().unwrap()); + let poll = || future_rs.as_mut().poll(py, &waker); match panic::catch_unwind(panic::AssertUnwindSafe(poll)) { - Ok(Poll::Ready(res)) => { - self.close(); - return Err(PyStopIteration::new_err(res?)); - } - Err(err) => { - self.close(); - return Err(PanicException::from_panic_payload(err)); - } - _ => {} + Err(err) => Err(PanicException::from_panic_payload(err)), + Ok(Poll::Ready(res)) => Err(PyStopIteration::new_err(res?)), + Ok(Poll::Pending) => match self.waker.as_ref().unwrap().yield_(py) { + Ok(to_yield) => Ok(to_yield), + Err(err) => Err(err), + }, } - // otherwise, initialize the waker `asyncio.Future` - if let Some(future) = self.waker.as_ref().unwrap().initialize_future(py)? { - // `asyncio.Future` must be awaited; fortunately, it implements `__iter__ = __await__` - // and will yield itself if its result has not been set in polling above - if let Some(future) = PyIterator::from_bound_object(&future.as_borrowed()) - .unwrap() - .next() - { - // future has not been leaked into Python for now, and Rust code can only call - // `set_result(None)` in `Wake` implementation, so it's safe to unwrap - return Ok(future.unwrap().into()); - } + } + + fn poll(&mut self, py: Python<'_>, op: CoroOp) -> PyResult { + let result = self.poll_inner(py, op); + if result.is_err() { + // the Rust future is dropped, and the field set to `None` + // to indicate the coroutine has been run to completion + drop(self.future.take()); } - // if waker has been waken during future polling, this is roughly equivalent to - // `await asyncio.sleep(0)`, so just yield `None`. - Ok(py.None().into_py(py)) + result } } @@ -143,18 +186,20 @@ impl Coroutine { } } - fn send(&mut self, py: Python<'_>, _value: &Bound<'_, PyAny>) -> PyResult { - self.poll(py, None) + fn send(&mut self, py: Python<'_>, value: PyObject) -> PyResult { + self.poll(py, CoroOp::Send(value)) } fn throw(&mut self, py: Python<'_>, exc: PyObject) -> PyResult { - self.poll(py, Some(exc)) + self.poll(py, CoroOp::Throw(exc)) } - fn close(&mut self) { - // the Rust future is dropped, and the field set to `None` - // to indicate the coroutine has been run to completion - drop(self.future.take()); + fn close(&mut self, py: Python<'_>) -> PyResult<()> { + match self.poll(py, CoroOp::Close) { + Ok(_) => Ok(()), + Err(err) if err.is_instance_of::(py) => Ok(()), + Err(err) => Err(err), + } } fn __await__(self_: Py) -> Py { @@ -162,6 +207,6 @@ impl Coroutine { } fn __next__(&mut self, py: Python<'_>) -> PyResult { - self.poll(py, None) + self.poll(py, CoroOp::Send(py.None())) } } diff --git a/src/coroutine/asyncio.rs b/src/coroutine/asyncio.rs new file mode 100644 index 00000000000..0049b56c0b8 --- /dev/null +++ b/src/coroutine/asyncio.rs @@ -0,0 +1,96 @@ +//! Coroutine implementation compatible with asyncio. +use pyo3_macros::pyfunction; + +use crate::{ + intern, + sync::GILOnceCell, + types::{PyAnyMethods, PyCFunction, PyIterator}, + wrap_pyfunction_bound, Bound, IntoPy, Py, PyAny, PyObject, PyResult, Python, +}; + +/// `asyncio.get_running_loop` +fn get_running_loop(py: Python<'_>) -> PyResult> { + static GET_RUNNING_LOOP: GILOnceCell = GILOnceCell::new(); + let import = || -> PyResult<_> { + let module = py.import_bound("asyncio")?; + Ok(module.getattr("get_running_loop")?.into()) + }; + GET_RUNNING_LOOP + .get_or_try_init(py, import)? + .bind(py) + .call0() +} + +/// Asyncio-compatible coroutine waker. +/// +/// Polling a Rust future yields an `asyncio.Future`, whose `set_result` method is called +/// when `Waker::wake` is called. +pub(super) struct AsyncioWaker { + event_loop: PyObject, + future: PyObject, +} + +impl AsyncioWaker { + pub(super) fn new(py: Python<'_>) -> PyResult { + let event_loop = get_running_loop(py)?.into_py(py); + let future = event_loop.call_method0(py, "create_future")?; + Ok(Self { event_loop, future }) + } + + pub(super) fn yield_(&self, py: Python<'_>) -> PyResult { + let __await__; + // `asyncio.Future` must be awaited; in normal case, it implements `__iter__ = __await__`, + // but `create_future` may have been overriden + let mut iter = match PyIterator::from_bound_object(self.future.bind(py)) { + Ok(iter) => iter, + Err(_) => { + __await__ = self.future.call_method0(py, intern!(py, "__await__"))?; + PyIterator::from_bound_object(__await__.bind(py))? + } + }; + // future has not been wakened (because `yield_waken` would have been called + // otherwise), so it is expected to yield itself + Ok(iter.next().expect("future didn't yield")?.into_py(py)) + } + + #[allow(clippy::unnecessary_wraps)] + pub(super) fn yield_waken(py: Python<'_>) -> PyResult { + Ok(py.None()) + } + + pub(super) fn wake(&self, py: Python<'_>) -> PyResult<()> { + static RELEASE_WAITER: GILOnceCell> = GILOnceCell::new(); + let release_waiter = RELEASE_WAITER.get_or_try_init(py, || { + wrap_pyfunction_bound!(release_waiter, py).map(Into::into) + })?; + // `Future.set_result` must be called in event loop thread, + // so it requires `call_soon_threadsafe` + let call_soon_threadsafe = self.event_loop.call_method1( + py, + intern!(py, "call_soon_threadsafe"), + (release_waiter, &self.future), + ); + if let Err(err) = call_soon_threadsafe { + // `call_soon_threadsafe` will raise if the event loop is closed; + // instead of catching an unspecific `RuntimeError`, check directly if it's closed. + let is_closed = self.event_loop.call_method0(py, "is_closed")?; + if !is_closed.extract(py)? { + return Err(err); + } + } + Ok(()) + } +} + +/// Call `future.set_result` if the future is not done. +/// +/// Future can be cancelled by the event loop before being wakened. +/// See +#[pyfunction(crate = "crate")] +fn release_waiter(future: &Bound<'_, PyAny>) -> PyResult<()> { + let done = future.call_method0(intern!(future.py(), "done"))?; + if !done.extract::()? { + future.call_method1(intern!(future.py(), "set_result"), (future.py().None(),))?; + } + Ok(()) +} diff --git a/src/coroutine/awaitable.rs b/src/coroutine/awaitable.rs new file mode 100644 index 00000000000..9eee03a0f07 --- /dev/null +++ b/src/coroutine/awaitable.rs @@ -0,0 +1,161 @@ +use std::{ + future::Future, + pin::Pin, + task::{Context, Poll}, +}; + +use super::waker::try_delegate; +use crate::{ + coroutine::CoroOp, + exceptions::{PyAttributeError, PyTypeError}, + intern, + sync::GILOnceCell, + types::{PyAnyMethods, PyIterator, PyTypeMethods}, + Bound, PyAny, PyErr, PyObject, PyResult, Python, +}; + +const NOT_IN_COROUTINE_CONTEXT: &str = "PyFuture must be awaited in coroutine context"; + +fn is_awaitable(obj: &Bound<'_, PyAny>) -> PyResult { + static IS_AWAITABLE: GILOnceCell = GILOnceCell::new(); + let import = || { + PyResult::Ok( + obj.py() + .import_bound("inspect")? + .getattr("isawaitable")? + .into(), + ) + }; + IS_AWAITABLE + .get_or_try_init(obj.py(), import)? + .call1(obj.py(), (obj,))? + .extract(obj.py()) +} + +pub(crate) enum YieldOrReturn { + Return(PyObject), + Yield(PyObject), +} + +pub(crate) fn delegate( + py: Python<'_>, + await_impl: PyObject, + op: &CoroOp, +) -> PyResult { + match op { + CoroOp::Send(obj) => { + cfg_if::cfg_if! { + if #[cfg(all(Py_3_10, not(PyPy), not(Py_LIMITED_API)))] { + let mut result = std::ptr::null_mut(); + match unsafe { crate::ffi::PyIter_Send(await_impl.as_ptr(), obj.as_ptr(), &mut result) } + { + -1 => Err(PyErr::take(py).unwrap()), + 0 => Ok(YieldOrReturn::Return(unsafe { + PyObject::from_owned_ptr(py, result) + })), + 1 => Ok(YieldOrReturn::Yield(unsafe { + PyObject::from_owned_ptr(py, result) + })), + _ => unreachable!(), + } + } else { + let send = intern!(py, "send"); + if obj.is_none(py) || !await_impl.bind(py).hasattr(send).unwrap_or(false) { + await_impl.call_method0(py, intern!(py, "__next__")) + } else { + await_impl.call_method1(py, send, (obj,)) + } + .map(YieldOrReturn::Yield) + } + } + } + CoroOp::Throw(exc) => { + let throw = intern!(py, "throw"); + if await_impl.bind(py).hasattr(throw).unwrap_or(false) { + await_impl + .call_method1(py, throw, (exc,)) + .map(YieldOrReturn::Yield) + } else { + Err(PyErr::from_value_bound(exc.bind(py).clone())) + } + } + CoroOp::Close => { + let close = intern!(py, "close"); + if await_impl.bind(py).hasattr(close).unwrap_or(false) { + await_impl + .call_method0(py, close) + .map(YieldOrReturn::Return) + } else { + Ok(YieldOrReturn::Return(py.None())) + } + } + } +} + +struct AwaitImpl(PyObject); + +impl AwaitImpl { + fn new(obj: &Bound<'_, PyAny>) -> PyResult { + let __await__ = intern!(obj.py(), "__await__"); + match obj.call_method0(__await__) { + Ok(iter) => Ok(Self(iter.unbind())), + Err(err) if err.is_instance_of::(obj.py()) => { + if obj.hasattr(__await__)? { + Err(err) + } else if is_awaitable(obj)? { + Ok(Self( + PyIterator::from_bound_object(obj)?.unbind().into_any(), + )) + } else { + Err(PyTypeError::new_err(format!( + "object {tp} can't be used in 'await' expression", + tp = obj.get_type().name()? + ))) + } + } + Err(err) => Err(err), + } + } +} + +impl Future for AwaitImpl { + type Output = PyResult; + + fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + match try_delegate(cx.waker(), self.0.clone()) { + Some(poll) => poll, + None => panic!("{}", NOT_IN_COROUTINE_CONTEXT), + } + } +} + +/// Allows awaiting arbitrary Python awaitable inside PyO3 coroutine context, e.g. async pyfunction. +/// +/// Awaiting the resulting future will panic if it's not done in coroutine context. +/// However, the future can be instantiated outside of coroutine context. +/// +/// ```rust +/// use pyo3::{coroutine::await_in_coroutine, prelude::*, py_run, wrap_pyfunction_bound}; +/// +/// # fn main() { +/// #[pyfunction] +/// async fn wrap_awaitable(awaitable: PyObject) -> PyResult { +/// let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; +/// future.await +/// } +/// Python::with_gil(|py| { +/// let wrap_awaitable = wrap_pyfunction_bound!(wrap_awaitable, py).unwrap(); +/// let test = r#" +/// import asyncio +/// assert asyncio.run(wrap_awaitable(asyncio.sleep(1, result=42))) == 42 +/// "#; +/// py_run!(py, wrap_awaitable, test); +/// }) +/// # } +/// ``` +/// ```rust +pub fn await_in_coroutine( + obj: &Bound<'_, PyAny>, +) -> PyResult> + Send + Sync + 'static> { + AwaitImpl::new(obj) +} diff --git a/src/coroutine/cancel.rs b/src/coroutine/cancel.rs index 47f5d69430a..2b968941caa 100644 --- a/src/coroutine/cancel.rs +++ b/src/coroutine/cancel.rs @@ -1,8 +1,11 @@ -use crate::{Py, PyAny, PyObject}; -use std::future::Future; -use std::pin::Pin; -use std::sync::{Arc, Mutex}; -use std::task::{Context, Poll, Waker}; +use std::{ + future::Future, + pin::Pin, + sync::{Arc, Mutex}, + task::{Context, Poll, Waker}, +}; + +use crate::PyObject; #[derive(Debug, Default)] struct Inner { @@ -44,6 +47,15 @@ impl CancelHandle { /// Retrieve the exception thrown in the associated coroutine. pub async fn cancelled(&mut self) -> PyObject { + // TODO use `std::future::poll_fn` with MSRV 1.64+ + struct Cancelled<'a>(&'a mut CancelHandle); + + impl Future for Cancelled<'_> { + type Output = PyObject; + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + self.0.poll_cancelled(cx) + } + } Cancelled(self).await } @@ -53,21 +65,11 @@ impl CancelHandle { } } -// Because `poll_fn` is not available in MSRV -struct Cancelled<'a>(&'a mut CancelHandle); - -impl Future for Cancelled<'_> { - type Output = PyObject; - fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { - self.0.poll_cancelled(cx) - } -} - #[doc(hidden)] pub struct ThrowCallback(Arc>); impl ThrowCallback { - pub(super) fn throw(&self, exc: Py) { + pub(super) fn throw(&self, exc: PyObject) { let mut inner = self.0.lock().unwrap(); inner.exception = Some(exc); if let Some(waker) = inner.waker.take() { diff --git a/src/coroutine/waker.rs b/src/coroutine/waker.rs index fc7c54e1f5a..4e9592c25c3 100644 --- a/src/coroutine/waker.rs +++ b/src/coroutine/waker.rs @@ -1,106 +1,119 @@ -use crate::sync::GILOnceCell; -use crate::types::any::PyAnyMethods; -use crate::types::PyCFunction; -use crate::{intern, wrap_pyfunction_bound, Bound, Py, PyAny, PyObject, PyResult, Python}; -use pyo3_macros::pyfunction; -use std::sync::Arc; -use std::task::Wake; +use std::{ + cell::Cell, + sync::Arc, + task::{Poll, Wake, Waker}, +}; -/// Lazy `asyncio.Future` wrapper, implementing [`Wake`] by calling `Future.set_result`. -/// -/// asyncio future is let uninitialized until [`initialize_future`][1] is called. -/// If [`wake`][2] is called before future initialization (during Rust future polling), -/// [`initialize_future`][1] will return `None` (it is roughly equivalent to `asyncio.sleep(0)`) -/// -/// [1]: AsyncioWaker::initialize_future -/// [2]: AsyncioWaker::wake -pub struct AsyncioWaker(GILOnceCell>); +use crate::{ + coroutine::{ + asyncio::AsyncioWaker, + awaitable::{delegate, YieldOrReturn}, + CoroOp, + }, + exceptions::PyStopIteration, + intern, + sync::GILOnceCell, + types::PyAnyMethods, + Bound, PyObject, PyResult, Python, +}; -impl AsyncioWaker { - pub(super) fn new() -> Self { - Self(GILOnceCell::new()) +const MIXED_AWAITABLE_AND_FUTURE_ERROR: &str = "Python awaitable mixed with Rust future"; + +enum State { + Pending(AsyncioWaker), + Waken, + Delegated(PyObject), +} + +pub(super) struct CoroutineWaker { + state: GILOnceCell, + op: CoroOp, +} + +impl CoroutineWaker { + pub(super) fn new(op: CoroOp) -> Self { + Self { + state: GILOnceCell::new(), + op, + } } - pub(super) fn reset(&mut self) { - self.0.take(); + pub(super) fn reset(&mut self, op: CoroOp) { + self.state.take(); + self.op = op; } - pub(super) fn initialize_future<'py>( - &self, - py: Python<'py>, - ) -> PyResult>> { - let init = || LoopAndFuture::new(py).map(Some); - let loop_and_future = self.0.get_or_try_init(py, init)?.as_ref(); - Ok(loop_and_future.map(|LoopAndFuture { future, .. }| future.bind(py))) + pub(super) fn is_delegated(&self, py: Python<'_>) -> bool { + matches!(self.state.get(py), Some(State::Delegated(_))) + } + + pub(super) fn yield_(&self, py: Python<'_>) -> PyResult { + let init = || PyResult::Ok(State::Pending(AsyncioWaker::new(py)?)); + let state = self.state.get_or_try_init(py, init)?; + match state { + State::Pending(waker) => waker.yield_(py), + State::Waken => AsyncioWaker::yield_waken(py), + State::Delegated(obj) => Ok(obj.clone_ref(py)), + } + } + + fn delegate(&self, py: Python<'_>, await_impl: PyObject) -> Poll> { + match delegate(py, await_impl, &self.op) { + Ok(YieldOrReturn::Yield(obj)) => { + let delegated = self.state.set(py, State::Delegated(obj)); + assert!(delegated.is_ok(), "{}", MIXED_AWAITABLE_AND_FUTURE_ERROR); + Poll::Pending + } + Ok(YieldOrReturn::Return(obj)) => Poll::Ready(Ok(obj)), + Err(err) if err.is_instance_of::(py) => Poll::Ready( + err.value_bound(py) + .getattr(intern!(py, "value")) + .map(Bound::unbind), + ), + Err(err) => Poll::Ready(Err(err)), + } } } -impl Wake for AsyncioWaker { +impl Wake for CoroutineWaker { fn wake(self: Arc) { self.wake_by_ref() } fn wake_by_ref(self: &Arc) { - Python::with_gil(|gil| { - if let Some(loop_and_future) = self.0.get_or_init(gil, || None) { - loop_and_future - .set_result(gil) - .expect("unexpected error in coroutine waker"); - } - }); + Python::with_gil(|gil| match WAKER_HACK.with(|cell| cell.take()) { + Some(WakerHack::Argument(await_impl)) => WAKER_HACK.with(|cell| { + let res = self.delegate(gil, await_impl); + cell.set(Some(WakerHack::Result(res))) + }), + Some(WakerHack::Result(_)) => unreachable!(), + None => match self.state.get_or_init(gil, || State::Waken) { + State::Pending(waker) => waker.wake(gil).expect("wake error"), + State::Waken => {} + State::Delegated(_) => panic!("{}", MIXED_AWAITABLE_AND_FUTURE_ERROR), + }, + }) } } -struct LoopAndFuture { - event_loop: PyObject, - future: PyObject, +enum WakerHack { + Argument(PyObject), + Result(Poll>), } -impl LoopAndFuture { - fn new(py: Python<'_>) -> PyResult { - static GET_RUNNING_LOOP: GILOnceCell = GILOnceCell::new(); - let import = || -> PyResult<_> { - let module = py.import_bound("asyncio")?; - Ok(module.getattr("get_running_loop")?.into()) - }; - let event_loop = GET_RUNNING_LOOP.get_or_try_init(py, import)?.call0(py)?; - let future = event_loop.call_method0(py, "create_future")?; - Ok(Self { event_loop, future }) - } - - fn set_result(&self, py: Python<'_>) -> PyResult<()> { - static RELEASE_WAITER: GILOnceCell> = GILOnceCell::new(); - let release_waiter = RELEASE_WAITER.get_or_try_init(py, || { - wrap_pyfunction_bound!(release_waiter, py).map(Bound::unbind) - })?; - // `Future.set_result` must be called in event loop thread, - // so it requires `call_soon_threadsafe` - let call_soon_threadsafe = self.event_loop.call_method1( - py, - intern!(py, "call_soon_threadsafe"), - (release_waiter, self.future.bind(py)), - ); - if let Err(err) = call_soon_threadsafe { - // `call_soon_threadsafe` will raise if the event loop is closed; - // instead of catching an unspecific `RuntimeError`, check directly if it's closed. - let is_closed = self.event_loop.call_method0(py, "is_closed")?; - if !is_closed.extract(py)? { - return Err(err); - } - } - Ok(()) - } +thread_local! { + static WAKER_HACK: Cell> = Cell::new(None); } -/// Call `future.set_result` if the future is not done. -/// -/// Future can be cancelled by the event loop before being waken. -/// See -#[pyfunction(crate = "crate")] -fn release_waiter(future: &Bound<'_, PyAny>) -> PyResult<()> { - let done = future.call_method0(intern!(future.py(), "done"))?; - if !done.extract::()? { - future.call_method1(intern!(future.py(), "set_result"), (future.py().None(),))?; +pub(crate) fn try_delegate( + waker: &Waker, + await_impl: PyObject, +) -> Option>> { + WAKER_HACK.with(|cell| cell.set(Some(WakerHack::Argument(await_impl)))); + waker.wake_by_ref(); + match WAKER_HACK.with(|cell| cell.take()) { + Some(WakerHack::Result(poll)) => Some(poll), + Some(WakerHack::Argument(_)) => None, + None => unreachable!(), } - Ok(()) } diff --git a/src/gil.rs b/src/gil.rs index 0bcb8c086c0..dd14097d801 100644 --- a/src/gil.rs +++ b/src/gil.rs @@ -1,13 +1,16 @@ //! Interaction with Python's global interpreter lock -use crate::impl_::not_send::{NotSend, NOT_SEND}; -use crate::{ffi, Python}; -use std::cell::Cell; #[cfg(debug_assertions)] use std::cell::RefCell; #[cfg(not(debug_assertions))] use std::cell::UnsafeCell; -use std::{mem, ptr::NonNull, sync}; +use std::{cell::Cell, mem, ptr::NonNull, sync}; + +use crate::{ + ffi, + impl_::not_send::{NotSend, NOT_SEND}, + Python, +}; static START: sync::Once = sync::Once::new(); @@ -505,15 +508,15 @@ fn decrement_gil_count() { #[cfg(test)] mod tests { - #[allow(deprecated)] - use super::GILPool; - use super::{gil_is_acquired, GIL_COUNT, OWNED_OBJECTS, POOL}; - use crate::types::any::PyAnyMethods; - use crate::{ffi, gil, PyObject, Python}; use std::ptr::NonNull; #[cfg(not(target_arch = "wasm32"))] use std::sync; + #[allow(deprecated)] + use super::GILPool; + use super::{gil_is_acquired, GIL_COUNT, OWNED_OBJECTS, POOL}; + use crate::{ffi, gil, types::any::PyAnyMethods, PyObject, Python}; + fn get_object(py: Python<'_>) -> PyObject { py.eval_bound("object()", None, None).unwrap().unbind() } @@ -790,9 +793,10 @@ mod tests { #[test] #[cfg(not(target_arch = "wasm32"))] // We are building wasm Python with pthreads disabled fn test_clone_without_gil() { - use crate::{Py, PyAny}; use std::{sync::Arc, thread}; + use crate::{Py, PyAny}; + // Some events for synchronizing static GIL_ACQUIRED: Event = Event::new(); static OBJECT_CLONED: Event = Event::new(); @@ -855,9 +859,10 @@ mod tests { #[test] #[cfg(not(target_arch = "wasm32"))] // We are building wasm Python with pthreads disabled fn test_clone_in_other_thread() { - use crate::Py; use std::{sync::Arc, thread}; + use crate::Py; + // Some events for synchronizing static OBJECT_CLONED: Event = Event::new(); @@ -930,4 +935,47 @@ mod tests { POOL.update_counts(py); }) } + + #[cfg(feature = "macros")] + #[test] + fn allow_threads_fn() { + #[crate::pyfunction(allow_threads, crate = "crate")] + fn without_gil(_arg1: PyObject, _arg2: PyObject) { + GIL_COUNT.with(|c| assert_eq!(c.get(), 0)); + } + Python::with_gil(|gil| { + let without_gil = crate::wrap_pyfunction_bound!(without_gil, gil).unwrap(); + crate::py_run!(gil, without_gil, "without_gil(..., ...)"); + }) + } + + #[cfg(feature = "experimental-async")] + #[cfg(not(target_arch = "wasm32"))] + #[test] + fn allow_threads_async_fn() { + #[crate::pyfunction(allow_threads, crate = "crate")] + async fn without_gil(_arg1: PyObject, _arg2: PyObject) { + use std::task::Poll; + GIL_COUNT.with(|c| assert_eq!(c.get(), 0)); + let mut ready = false; + futures::future::poll_fn(|cx| { + if ready { + return Poll::Ready(()); + } + ready = true; + cx.waker().wake_by_ref(); + Poll::Pending + }) + .await; + GIL_COUNT.with(|c| assert_eq!(c.get(), 0)); + } + Python::with_gil(|gil| { + let without_gil = crate::wrap_pyfunction_bound!(without_gil, gil).unwrap(); + crate::py_run!( + gil, + without_gil, + "import asyncio; asyncio.run(without_gil(..., ...))" + ); + }) + } } diff --git a/src/impl_/coroutine.rs b/src/impl_/coroutine.rs index 1d3119400a0..830bcfa644c 100644 --- a/src/impl_/coroutine.rs +++ b/src/impl_/coroutine.rs @@ -4,8 +4,9 @@ use std::{ }; use crate::{ - coroutine::{cancel::ThrowCallback, Coroutine}, + coroutine::{Coroutine, ThrowCallback}, instance::Bound, + marker::Ungil, pycell::impl_::PyClassBorrowChecker, pyclass::boolean_struct::False, types::{PyAnyMethods, PyString}, @@ -16,17 +17,19 @@ pub fn new_coroutine( name: &Bound<'_, PyString>, qualname_prefix: Option<&'static str>, throw_callback: Option, + allow_threads: bool, future: F, ) -> Coroutine where - F: Future> + Send + 'static, - T: IntoPy, - E: Into, + F: Future> + Send + Ungil + 'static, + T: IntoPy + Send + Ungil, + E: Into + Send + Ungil, { Coroutine::new( Some(name.clone().into()), qualname_prefix, throw_callback, + allow_threads, future, ) } diff --git a/src/lib.rs b/src/lib.rs index b400f143f5a..fa6e1b0f0ff 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -312,27 +312,29 @@ //! [Rust from Python]: https://github.com/PyO3/pyo3#using-rust-from-python //! [Features chapter of the guide]: https://pyo3.rs/latest/features.html#features-reference "Features Reference - PyO3 user guide" //! [`Ungil`]: crate::marker::Ungil -pub use crate::class::*; -pub use crate::conversion::{AsPyPointer, FromPyObject, IntoPy, ToPyObject}; #[allow(deprecated)] pub use crate::conversion::{FromPyPointer, PyTryFrom, PyTryInto}; -pub use crate::err::{ - DowncastError, DowncastIntoError, PyDowncastError, PyErr, PyErrArguments, PyResult, ToPyErr, -}; #[allow(deprecated)] pub use crate::gil::GILPool; #[cfg(not(any(PyPy, GraalPy)))] pub use crate::gil::{prepare_freethreaded_python, with_embedded_python_interpreter}; -pub use crate::instance::{Borrowed, Bound, Py, PyNativeType, PyObject}; -pub use crate::marker::Python; #[allow(deprecated)] pub use crate::pycell::PyCell; -pub use crate::pycell::{PyRef, PyRefMut}; -pub use crate::pyclass::PyClass; -pub use crate::pyclass_init::PyClassInitializer; -pub use crate::type_object::{PyTypeCheck, PyTypeInfo}; -pub use crate::types::PyAny; -pub use crate::version::PythonVersionInfo; +pub use crate::{ + class::*, + conversion::{AsPyPointer, FromPyObject, IntoPy, ToPyObject}, + err::{ + DowncastError, DowncastIntoError, PyDowncastError, PyErr, PyErrArguments, PyResult, ToPyErr, + }, + instance::{Borrowed, Bound, Py, PyNativeType, PyObject}, + marker::Python, + pycell::{PyRef, PyRefMut}, + pyclass::PyClass, + pyclass_init::PyClassInitializer, + type_object::{PyTypeCheck, PyTypeInfo}, + types::PyAny, + version::PythonVersionInfo, +}; pub(crate) mod ffi_ptr_ext; pub(crate) mod py_result_ext; @@ -347,7 +349,6 @@ pub(crate) mod sealed; /// once is resolved. pub mod class { pub use self::gc::{PyTraverseError, PyVisit}; - #[doc(hidden)] pub use self::methods::{ PyClassAttributeDef, PyGetterDef, PyMethodDef, PyMethodDefType, PyMethodType, PySetterDef, @@ -410,16 +411,15 @@ pub mod class { } } +#[cfg(all(feature = "macros", feature = "multiple-pymethods"))] +#[doc(hidden)] +pub use inventory; #[cfg(feature = "macros")] #[doc(hidden)] pub use { indoc, // Re-exported for py_run unindent, // Re-exported for py_run -}; - -#[cfg(all(feature = "macros", feature = "multiple-pymethods"))] -#[doc(hidden)] -pub use inventory; // Re-exported for `#[pyclass]` and `#[pymethods]` with `multiple-pymethods`. +}; // Re-exported for `#[pyclass]` and `#[pymethods]` with `multiple-pymethods`. /// Tests and helpers which reside inside PyO3's main library. Declared first so that macros /// are available in unit tests. @@ -462,14 +462,7 @@ pub mod type_object; pub mod types; mod version; -#[allow(unused_imports)] // with no features enabled this module has no public exports -pub use crate::conversions::*; - -#[cfg(feature = "macros")] -pub use pyo3_macros::{pyfunction, pymethods, pymodule, FromPyObject}; - /// A proc macro used to expose Rust structs and fieldless enums as Python objects. -/// #[doc = include_str!("../guide/pyclass-parameters.md")] /// /// For more on creating Python classes, @@ -478,6 +471,11 @@ pub use pyo3_macros::{pyfunction, pymethods, pymodule, FromPyObject}; /// [1]: https://pyo3.rs/latest/class.html #[cfg(feature = "macros")] pub use pyo3_macros::pyclass; +#[cfg(feature = "macros")] +pub use pyo3_macros::{pyfunction, pymethods, pymodule, FromPyObject}; + +#[allow(unused_imports)] // with no features enabled this module has no public exports +pub use crate::conversions::*; #[cfg(feature = "macros")] #[macro_use] @@ -512,6 +510,7 @@ pub mod doc_test { "README.md" => readme_md, "guide/src/advanced.md" => guide_advanced_md, "guide/src/async-await.md" => guide_async_await_md, + "guide/src/async-await/awaiting_python_awaitables.md" => guide_async_await_awaiting_python_awaitable_md, "guide/src/building-and-distribution.md" => guide_building_and_distribution_md, "guide/src/building-and-distribution/multiple-python-versions.md" => guide_bnd_multiple_python_versions_md, "guide/src/class.md" => guide_class_md, diff --git a/src/tests/common.rs b/src/tests/common.rs index 854d73e4d7b..78962d68cd0 100644 --- a/src/tests/common.rs +++ b/src/tests/common.rs @@ -6,13 +6,14 @@ #[macro_use] mod inner { + use pyo3::{ + prelude::*, + types::{IntoPyDict, PyList}, + }; + #[allow(unused_imports)] // pulls in `use crate as pyo3` in `test_utils.rs` use super::*; - use pyo3::prelude::*; - - use pyo3::types::{IntoPyDict, PyList}; - #[macro_export] macro_rules! py_assert { ($py:expr, $($val:ident)+, $assertion:literal) => { @@ -156,6 +157,17 @@ mod inner { .unwrap(); }}; } + + // see https://stackoverflow.com/questions/60359157/valueerror-set-wakeup-fd-only-works-in-main-thread-on-windows-on-python-3-8-wit + #[cfg(feature = "macros")] + pub fn asyncio_windows(test: &str) -> String { + let set_event_loop_policy = r#" + import asyncio, sys + if sys.platform == "win32": + asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) + "#; + pyo3::unindent::unindent(set_event_loop_policy) + &pyo3::unindent::unindent(test) + } } #[allow(unused_imports)] // some tests use just the macros and none of the other functionality diff --git a/tests/test_await_in_coroutine.rs b/tests/test_await_in_coroutine.rs new file mode 100644 index 00000000000..3f835c27588 --- /dev/null +++ b/tests/test_await_in_coroutine.rs @@ -0,0 +1,177 @@ +#![cfg(feature = "experimental-async")] + +use std::task::Poll; + +use futures::{future::poll_fn, FutureExt}; +use pyo3::{ + coroutine::{await_in_coroutine, CancelHandle}, + exceptions::{PyAttributeError, PyTypeError}, + prelude::*, + py_run, +}; + +#[path = "../src/tests/common.rs"] +mod common; + +#[pyfunction] +async fn wrap_awaitable(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + future.await +} + +#[test] +fn awaitable() { + Python::with_gil(|gil| { + let wrap_awaitable = wrap_pyfunction_bound!(wrap_awaitable, gil).unwrap(); + let test = r#" + import types + import asyncio; + + class BadAwaitable: + def __await__(self): + raise AttributeError("__await__") + + @types.coroutine + def gen_coro(): + yield None + + async def main(): + await wrap_awaitable(...) + asyncio.run(main()) + "#; + let globals = gil.import_bound("__main__").unwrap().dict(); + globals.set_item("wrap_awaitable", wrap_awaitable).unwrap(); + let run = |awaitable| { + gil.run_bound( + &common::asyncio_windows(test).replace("...", awaitable), + Some(&globals), + None, + ) + }; + run("asyncio.sleep(0.001)").unwrap(); + run("gen_coro()").unwrap(); + assert!(run("None").unwrap_err().is_instance_of::(gil)); + assert!(run("BadAwaitable()") + .unwrap_err() + .is_instance_of::(gil)); + }) +} + +#[test] +fn cancel_delegation() { + #[pyfunction] + async fn wrap_cancellable(awaitable: PyObject, #[pyo3(cancel_handle)] cancel: CancelHandle) { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil))).unwrap(); + let result = future.await; + Python::with_gil(|gil| { + assert_eq!( + result.unwrap_err().get_type_bound(gil).name().unwrap(), + "CancelledError" + ) + }); + assert!(!cancel.is_cancelled()); + } + Python::with_gil(|gil| { + let wrap_cancellable = wrap_pyfunction_bound!(wrap_cancellable, gil).unwrap(); + let test = r#" + import asyncio; + + async def main(): + task = asyncio.create_task(wrap_cancellable(asyncio.sleep(0.001))) + await asyncio.sleep(0) + task.cancel() + await task + asyncio.run(main()) + "#; + let globals = gil.import_bound("__main__").unwrap().dict(); + globals + .set_item("wrap_cancellable", wrap_cancellable) + .unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); + }) +} + +#[test] +#[should_panic(expected = "PyFuture must be awaited in coroutine context")] +fn pyfuture_without_coroutine() { + #[pyfunction] + fn block_on(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::executor::block_on(future) + } + Python::with_gil(|gil| { + let block_on = wrap_pyfunction_bound!(block_on, gil).unwrap(); + let test = r#" + async def coro(): + ... + block_on(coro()) + "#; + py_run!(gil, block_on, &common::asyncio_windows(test)); + }) +} + +async fn checkpoint() { + let mut ready = false; + poll_fn(|cx| { + if ready { + return Poll::Ready(()); + } + ready = true; + cx.waker().wake_by_ref(); + Poll::Pending + }) + .await +} + +#[test] +#[should_panic(expected = "Python awaitable mixed with Rust future")] +fn pyfuture_in_select() { + #[pyfunction] + async fn select(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::select_biased! { + _ = checkpoint().fuse() => unreachable!(), + res = future.fuse() => res, + } + } + Python::with_gil(|gil| { + let select = wrap_pyfunction_bound!(select, gil).unwrap(); + let test = r#" + import asyncio; + async def main(): + return await select(asyncio.sleep(1)) + asyncio.run(main()) + "#; + let globals = gil.import_bound("__main__").unwrap().dict(); + globals.set_item("select", select).unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); + }) +} + +#[test] +#[should_panic(expected = "Python awaitable mixed with Rust future")] +fn pyfuture_in_select2() { + #[pyfunction] + async fn select2(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::select_biased! { + res = future.fuse() => res, + _ = checkpoint().fuse() => unreachable!(), + } + } + Python::with_gil(|gil| { + let select2 = wrap_pyfunction_bound!(select2, gil).unwrap(); + let test = r#" + import asyncio; + async def main(): + return await select2(asyncio.sleep(1)) + asyncio.run(main()) + "#; + let globals = gil.import_bound("__main__").unwrap().dict(); + globals.set_item("select2", select2).unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); + }) +} diff --git a/tests/test_compile_error.rs b/tests/test_compile_error.rs index 30e77888cfd..bb438eb7343 100644 --- a/tests/test_compile_error.rs +++ b/tests/test_compile_error.rs @@ -54,6 +54,6 @@ fn test_compile_errors() { t.compile_fail("tests/ui/invalid_pymodule_two_pymodule_init.rs"); #[cfg(feature = "experimental-async")] #[cfg(any(not(Py_LIMITED_API), Py_3_10))] // to avoid PyFunctionArgument for &str - t.compile_fail("tests/ui/invalid_cancel_handle.rs"); + t.compile_fail("tests/ui/invalid_async_pyfunction.rs"); t.pass("tests/ui/pymodule_missing_docs.rs"); } diff --git a/tests/test_coroutine.rs b/tests/test_coroutine.rs index 75b524edf78..4ab13733ecf 100644 --- a/tests/test_coroutine.rs +++ b/tests/test_coroutine.rs @@ -17,15 +17,6 @@ use std::sync::atomic::{AtomicBool, Ordering}; #[path = "../src/tests/common.rs"] mod common; -fn handle_windows(test: &str) -> String { - let set_event_loop_policy = r#" - import asyncio, sys - if sys.platform == "win32": - asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) - "#; - pyo3::unindent::unindent(set_event_loop_policy) + &pyo3::unindent::unindent(test) -} - #[test] fn noop_coroutine() { #[pyfunction] @@ -35,7 +26,7 @@ fn noop_coroutine() { Python::with_gil(|gil| { let noop = wrap_pyfunction_bound!(noop, gil).unwrap(); let test = "import asyncio; assert asyncio.run(noop()) == 42"; - py_run!(gil, noop, &handle_windows(test)); + py_run!(gil, noop, &common::asyncio_windows(test)); }) } @@ -79,7 +70,7 @@ fn test_coroutine_qualname() { ("MyClass", gil.get_type_bound::().as_any()), ] .into_py_dict_bound(gil); - py_run!(gil, *locals, &handle_windows(test)); + py_run!(gil, *locals, &common::asyncio_windows(test)); }) } @@ -101,7 +92,7 @@ fn sleep_0_like_coroutine() { Python::with_gil(|gil| { let sleep_0 = wrap_pyfunction_bound!(sleep_0, gil).unwrap(); let test = "import asyncio; assert asyncio.run(sleep_0()) == 42"; - py_run!(gil, sleep_0, &handle_windows(test)); + py_run!(gil, sleep_0, &common::asyncio_windows(test)); }) } @@ -120,7 +111,7 @@ fn sleep_coroutine() { Python::with_gil(|gil| { let sleep = wrap_pyfunction_bound!(sleep, gil).unwrap(); let test = r#"import asyncio; assert asyncio.run(sleep(0.1)) == 42"#; - py_run!(gil, sleep, &handle_windows(test)); + py_run!(gil, sleep, &common::asyncio_windows(test)); }) } @@ -140,11 +131,7 @@ fn cancelled_coroutine() { let globals = gil.import_bound("__main__").unwrap().dict(); globals.set_item("sleep", sleep).unwrap(); let err = gil - .run_bound( - &pyo3::unindent::unindent(&handle_windows(test)), - Some(&globals), - None, - ) + .run_bound(&common::asyncio_windows(test), Some(&globals), None) .unwrap_err(); assert_eq!( err.value_bound(gil).get_type().qualname().unwrap(), @@ -180,12 +167,8 @@ fn coroutine_cancel_handle() { globals .set_item("cancellable_sleep", cancellable_sleep) .unwrap(); - gil.run_bound( - &pyo3::unindent::unindent(&handle_windows(test)), - Some(&globals), - None, - ) - .unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); }) } @@ -210,12 +193,8 @@ fn coroutine_is_cancelled() { "#; let globals = gil.import_bound("__main__").unwrap().dict(); globals.set_item("sleep_loop", sleep_loop).unwrap(); - gil.run_bound( - &pyo3::unindent::unindent(&handle_windows(test)), - Some(&globals), - None, - ) - .unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); }) } @@ -244,7 +223,7 @@ fn coroutine_panic() { else: assert False; "#; - py_run!(gil, panic, &handle_windows(test)); + py_run!(gil, panic, &common::asyncio_windows(test)); }) } @@ -341,6 +320,6 @@ fn test_async_method_receiver_with_other_args() { assert asyncio.run(v.get_value_plus_with(1, 1)) == 12 "#; let locals = [("Value", gil.get_type_bound::())].into_py_dict_bound(gil); - py_run!(gil, *locals, test); + py_run!(gil, *locals, &common::asyncio_windows(test)); }); } diff --git a/tests/test_methods.rs b/tests/test_methods.rs index 2b5396e9ee4..f343dd3eb98 100644 --- a/tests/test_methods.rs +++ b/tests/test_methods.rs @@ -1158,3 +1158,23 @@ fn test_issue_2988() { ) { } } + +#[pyclass] +struct NoGILCounter(usize); + +#[pymethods] +impl NoGILCounter { + #[pyo3(allow_threads, signature = (other = 1))] + fn inc_no_gil(&mut self, other: usize) -> usize { + self.0 += other; + self.0 + } +} + +#[test] +fn test_method_allow_threads() { + Python::with_gil(|py| { + let counter = Py::new(py, NoGILCounter(42)).unwrap(); + py_run!(py, counter, "assert counter.inc_no_gil() == 43") + }) +} diff --git a/tests/ui/invalid_cancel_handle.rs b/tests/ui/invalid_async_pyfunction.rs similarity index 60% rename from tests/ui/invalid_cancel_handle.rs rename to tests/ui/invalid_async_pyfunction.rs index cff6c5dcbad..d39c017c0e7 100644 --- a/tests/ui/invalid_cancel_handle.rs +++ b/tests/ui/invalid_async_pyfunction.rs @@ -1,20 +1,26 @@ use pyo3::prelude::*; +#[pyfunction(allow_threads)] +async fn async_with_gil(_py: Python<'_>) {} + +#[pyfunction(allow_threads)] +async fn async_with_bound(_obj: &Bound<'_, PyAny>) {} + #[pyfunction] -async fn cancel_handle_repeated(#[pyo3(cancel_handle, cancel_handle)] _param: String) {} +async fn cancel_handle_repeated(#[pyo3(cancel_handle, cancel_handle)] _param: i32) {} #[pyfunction] async fn cancel_handle_repeated2( - #[pyo3(cancel_handle)] _param: String, - #[pyo3(cancel_handle)] _param2: String, + #[pyo3(cancel_handle)] _param: i32, + #[pyo3(cancel_handle)] _param2: i32, ) { } #[pyfunction] -fn cancel_handle_synchronous(#[pyo3(cancel_handle)] _param: String) {} +fn cancel_handle_synchronous(#[pyo3(cancel_handle)] _param: i32) {} #[pyfunction] -async fn cancel_handle_wrong_type(#[pyo3(cancel_handle)] _param: String) {} +async fn cancel_handle_wrong_type(#[pyo3(cancel_handle)] _param: i32) {} #[pyfunction] async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} diff --git a/tests/ui/invalid_async_pyfunction.stderr b/tests/ui/invalid_async_pyfunction.stderr new file mode 100644 index 00000000000..0e17c1c5824 --- /dev/null +++ b/tests/ui/invalid_async_pyfunction.stderr @@ -0,0 +1,145 @@ +error: GIL token cannot be passed to async function + --> tests/ui/invalid_async_pyfunction.rs:4:30 + | +4 | async fn async_with_gil(_py: Python<'_>) {} + | ^^^^^^ + +error: `cancel_handle` may only be specified once per argument + --> tests/ui/invalid_async_pyfunction.rs:10:55 + | +10 | async fn cancel_handle_repeated(#[pyo3(cancel_handle, cancel_handle)] _param: i32) {} + | ^^^^^^^^^^^^^ + +error: `cancel_handle` may only be specified once + --> tests/ui/invalid_async_pyfunction.rs:15:28 + | +15 | #[pyo3(cancel_handle)] _param2: i32, + | ^^^^^^^ + +error: `cancel_handle` attribute can only be used with `async fn` + --> tests/ui/invalid_async_pyfunction.rs:20:53 + | +20 | fn cancel_handle_synchronous(#[pyo3(cancel_handle)] _param: i32) {} + | ^^^^^^ + +error: `from_py_with` and `cancel_handle` cannot be specified together + --> tests/ui/invalid_async_pyfunction.rs:30:12 + | +30 | #[pyo3(cancel_handle, from_py_with = "cancel_handle")] _param: pyo3::coroutine::CancelHandle, + | ^^^^^^^^^^^^^ + +error[E0277]: `*mut pyo3::Python<'static>` cannot be shared between threads safely + --> tests/ui/invalid_async_pyfunction.rs:6:1 + | +6 | #[pyfunction(allow_threads)] + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `*mut pyo3::Python<'static>` cannot be shared between threads safely + | + = help: within `pyo3::Bound<'_, pyo3::PyAny>`, the trait `Sync` is not implemented for `*mut pyo3::Python<'static>`, which is required by `{async block@$DIR/tests/ui/invalid_async_pyfunction.rs:6:1: 6:29}: Send` +note: required because it appears within the type `PhantomData<*mut pyo3::Python<'static>>` + --> $RUST/core/src/marker.rs + | + | pub struct PhantomData; + | ^^^^^^^^^^^ +note: required because it appears within the type `impl_::not_send::NotSend` + --> src/impl_/not_send.rs + | + | pub(crate) struct NotSend(PhantomData<*mut Python<'static>>); + | ^^^^^^^ + = note: required because it appears within the type `(&pyo3::gil::GILGuard, impl_::not_send::NotSend)` +note: required because it appears within the type `PhantomData<(&pyo3::gil::GILGuard, impl_::not_send::NotSend)>` + --> $RUST/core/src/marker.rs + | + | pub struct PhantomData; + | ^^^^^^^^^^^ +note: required because it appears within the type `pyo3::Python<'_>` + --> src/marker.rs + | + | pub struct Python<'py>(PhantomData<(&'py GILGuard, NotSend)>); + | ^^^^^^ +note: required because it appears within the type `pyo3::Bound<'_, pyo3::PyAny>` + --> src/instance.rs + | + | pub struct Bound<'py, T>(Python<'py>, ManuallyDrop>); + | ^^^^^ + = note: required for `&pyo3::Bound<'_, pyo3::PyAny>` to implement `Send` +note: required because it's used within this `async` fn body + --> tests/ui/invalid_async_pyfunction.rs:7:52 + | +7 | async fn async_with_bound(_obj: &Bound<'_, PyAny>) {} + | ^^ +note: required because it's used within this `async` block + --> tests/ui/invalid_async_pyfunction.rs:6:1 + | +6 | #[pyfunction(allow_threads)] + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +note: required by a bound in `new_coroutine` + --> src/impl_/coroutine.rs + | + | pub fn new_coroutine( + | ------------- required by a bound in this function +... + | F: Future> + Send + Ungil + 'static, + | ^^^^ required by this bound in `new_coroutine` + = note: this error originates in the attribute macro `pyfunction` (in Nightly builds, run with -Z macro-backtrace for more info) + +error[E0308]: mismatched types + --> tests/ui/invalid_async_pyfunction.rs:22:1 + | +22 | #[pyfunction] + | ^^^^^^^^^^^^^ + | | + | expected `i32`, found `CancelHandle` + | arguments to this function are incorrect + | +note: function defined here + --> tests/ui/invalid_async_pyfunction.rs:23:10 + | +23 | async fn cancel_handle_wrong_type(#[pyo3(cancel_handle)] _param: i32) {} + | ^^^^^^^^^^^^^^^^^^^^^^^^ ----------- + = note: this error originates in the attribute macro `pyfunction` (in Nightly builds, run with -Z macro-backtrace for more info) + +error[E0277]: the trait bound `CancelHandle: PyFunctionArgument<'_, '_>` is not satisfied + --> tests/ui/invalid_async_pyfunction.rs:26:50 + | +26 | async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} + | ^^^^ the trait `PyClass` is not implemented for `CancelHandle`, which is required by `CancelHandle: PyFunctionArgument<'_, '_>` + | + = help: the following other types implement trait `PyFunctionArgument<'a, 'py>`: + Option<&'a pyo3::Bound<'py, T>> + &'a pyo3::Bound<'py, T> + &'a pyo3::coroutine::Coroutine + &'a mut pyo3::coroutine::Coroutine + = note: required for `CancelHandle` to implement `FromPyObject<'_>` + = note: required for `CancelHandle` to implement `FromPyObjectBound<'_, '_>` + = note: required for `CancelHandle` to implement `PyFunctionArgument<'_, '_>` +note: required by a bound in `extract_argument` + --> src/impl_/extract_argument.rs + | + | pub fn extract_argument<'a, 'py, T>( + | ---------------- required by a bound in this function +... + | T: PyFunctionArgument<'a, 'py>, + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `extract_argument` + +error[E0277]: the trait bound `CancelHandle: PyFunctionArgument<'_, '_>` is not satisfied + --> tests/ui/invalid_async_pyfunction.rs:26:50 + | +26 | async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} + | ^^^^ the trait `Clone` is not implemented for `CancelHandle`, which is required by `CancelHandle: PyFunctionArgument<'_, '_>` + | + = help: the following other types implement trait `PyFunctionArgument<'a, 'py>`: + Option<&'a pyo3::Bound<'py, T>> + &'a pyo3::Bound<'py, T> + &'a pyo3::coroutine::Coroutine + &'a mut pyo3::coroutine::Coroutine + = note: required for `CancelHandle` to implement `FromPyObject<'_>` + = note: required for `CancelHandle` to implement `FromPyObjectBound<'_, '_>` + = note: required for `CancelHandle` to implement `PyFunctionArgument<'_, '_>` +note: required by a bound in `extract_argument` + --> src/impl_/extract_argument.rs + | + | pub fn extract_argument<'a, 'py, T>( + | ---------------- required by a bound in this function +... + | T: PyFunctionArgument<'a, 'py>, + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `extract_argument` diff --git a/tests/ui/invalid_cancel_handle.stderr b/tests/ui/invalid_cancel_handle.stderr deleted file mode 100644 index f6452611679..00000000000 --- a/tests/ui/invalid_cancel_handle.stderr +++ /dev/null @@ -1,85 +0,0 @@ -error: `cancel_handle` may only be specified once per argument - --> tests/ui/invalid_cancel_handle.rs:4:55 - | -4 | async fn cancel_handle_repeated(#[pyo3(cancel_handle, cancel_handle)] _param: String) {} - | ^^^^^^^^^^^^^ - -error: `cancel_handle` may only be specified once - --> tests/ui/invalid_cancel_handle.rs:9:28 - | -9 | #[pyo3(cancel_handle)] _param2: String, - | ^^^^^^^ - -error: `cancel_handle` attribute can only be used with `async fn` - --> tests/ui/invalid_cancel_handle.rs:14:53 - | -14 | fn cancel_handle_synchronous(#[pyo3(cancel_handle)] _param: String) {} - | ^^^^^^ - -error: `from_py_with` and `cancel_handle` cannot be specified together - --> tests/ui/invalid_cancel_handle.rs:24:12 - | -24 | #[pyo3(cancel_handle, from_py_with = "cancel_handle")] _param: pyo3::coroutine::CancelHandle, - | ^^^^^^^^^^^^^ - -error[E0308]: mismatched types - --> tests/ui/invalid_cancel_handle.rs:16:1 - | -16 | #[pyfunction] - | ^^^^^^^^^^^^^ - | | - | expected `String`, found `CancelHandle` - | arguments to this function are incorrect - | -note: function defined here - --> tests/ui/invalid_cancel_handle.rs:17:10 - | -17 | async fn cancel_handle_wrong_type(#[pyo3(cancel_handle)] _param: String) {} - | ^^^^^^^^^^^^^^^^^^^^^^^^ -------------- - = note: this error originates in the attribute macro `pyfunction` (in Nightly builds, run with -Z macro-backtrace for more info) - -error[E0277]: the trait bound `CancelHandle: PyFunctionArgument<'_, '_>` is not satisfied - --> tests/ui/invalid_cancel_handle.rs:20:50 - | -20 | async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} - | ^^^^ the trait `PyClass` is not implemented for `CancelHandle`, which is required by `CancelHandle: PyFunctionArgument<'_, '_>` - | - = help: the following other types implement trait `PyFunctionArgument<'a, 'py>`: - Option<&'a pyo3::Bound<'py, T>> - &'a pyo3::Bound<'py, T> - &'a pyo3::coroutine::Coroutine - &'a mut pyo3::coroutine::Coroutine - = note: required for `CancelHandle` to implement `FromPyObject<'_>` - = note: required for `CancelHandle` to implement `FromPyObjectBound<'_, '_>` - = note: required for `CancelHandle` to implement `PyFunctionArgument<'_, '_>` -note: required by a bound in `extract_argument` - --> src/impl_/extract_argument.rs - | - | pub fn extract_argument<'a, 'py, T>( - | ---------------- required by a bound in this function -... - | T: PyFunctionArgument<'a, 'py>, - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `extract_argument` - -error[E0277]: the trait bound `CancelHandle: PyFunctionArgument<'_, '_>` is not satisfied - --> tests/ui/invalid_cancel_handle.rs:20:50 - | -20 | async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} - | ^^^^ the trait `Clone` is not implemented for `CancelHandle`, which is required by `CancelHandle: PyFunctionArgument<'_, '_>` - | - = help: the following other types implement trait `PyFunctionArgument<'a, 'py>`: - Option<&'a pyo3::Bound<'py, T>> - &'a pyo3::Bound<'py, T> - &'a pyo3::coroutine::Coroutine - &'a mut pyo3::coroutine::Coroutine - = note: required for `CancelHandle` to implement `FromPyObject<'_>` - = note: required for `CancelHandle` to implement `FromPyObjectBound<'_, '_>` - = note: required for `CancelHandle` to implement `PyFunctionArgument<'_, '_>` -note: required by a bound in `extract_argument` - --> src/impl_/extract_argument.rs - | - | pub fn extract_argument<'a, 'py, T>( - | ---------------- required by a bound in this function -... - | T: PyFunctionArgument<'a, 'py>, - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `extract_argument` diff --git a/tests/ui/invalid_pyfunction_signatures.stderr b/tests/ui/invalid_pyfunction_signatures.stderr index 97d0fd3b4af..cd8e14f911c 100644 --- a/tests/ui/invalid_pyfunction_signatures.stderr +++ b/tests/ui/invalid_pyfunction_signatures.stderr @@ -16,7 +16,7 @@ error: expected argument from function definition `y` but got argument `x` 13 | #[pyo3(signature = (x))] | ^ -error: expected one of: `name`, `pass_module`, `signature`, `text_signature`, `crate` +error: expected one of: `allow_threads`, `name`, `pass_module`, `signature`, `text_signature`, `crate` --> tests/ui/invalid_pyfunction_signatures.rs:18:14 | 18 | #[pyfunction(x)] diff --git a/tests/ui/invalid_pyfunctions.rs b/tests/ui/invalid_pyfunctions.rs index 1c0c45d6b95..919c6f84a73 100644 --- a/tests/ui/invalid_pyfunctions.rs +++ b/tests/ui/invalid_pyfunctions.rs @@ -39,4 +39,10 @@ fn first_argument_not_module<'a, 'py>( module.name() } +#[pyfunction(allow_threads)] +fn allow_threads_with_gil(_py: Python<'_>) {} + +#[pyfunction(allow_threads)] +fn allow_threads_with_bound(_obj: &Bound<'_, PyAny>) {} + fn main() {} diff --git a/tests/ui/invalid_pyfunctions.stderr b/tests/ui/invalid_pyfunctions.stderr index 830f17ee877..a3c4f52873a 100644 --- a/tests/ui/invalid_pyfunctions.stderr +++ b/tests/ui/invalid_pyfunctions.stderr @@ -47,6 +47,12 @@ error: expected `&PyModule` or `Py` as first argument with `pass_modul 32 | fn pass_module_but_no_arguments<'py>() {} | ^^ +error: GIL cannot be held in function annotated with `allow_threads` + --> tests/ui/invalid_pyfunctions.rs:43:32 + | +43 | fn allow_threads_with_gil(_py: Python<'_>) {} + | ^^^^^^ + error[E0277]: the trait bound `&str: From>` is not satisfied --> tests/ui/invalid_pyfunctions.rs:36:14 | @@ -61,3 +67,54 @@ error[E0277]: the trait bound `&str: From> > = note: required for `BoundRef<'_, '_, pyo3::prelude::PyModule>` to implement `Into<&str>` + +error[E0277]: `*mut pyo3::Python<'static>` cannot be shared between threads safely + --> tests/ui/invalid_pyfunctions.rs:45:1 + | +45 | #[pyfunction(allow_threads)] + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `*mut pyo3::Python<'static>` cannot be shared between threads safely + | + = help: within `&pyo3::Bound<'_, pyo3::PyAny>`, the trait `Sync` is not implemented for `*mut pyo3::Python<'static>`, which is required by `{closure@$DIR/tests/ui/invalid_pyfunctions.rs:45:1: 45:29}: Ungil` +note: required because it appears within the type `PhantomData<*mut pyo3::Python<'static>>` + --> $RUST/core/src/marker.rs + | + | pub struct PhantomData; + | ^^^^^^^^^^^ +note: required because it appears within the type `impl_::not_send::NotSend` + --> src/impl_/not_send.rs + | + | pub(crate) struct NotSend(PhantomData<*mut Python<'static>>); + | ^^^^^^^ + = note: required because it appears within the type `(&pyo3::gil::GILGuard, impl_::not_send::NotSend)` +note: required because it appears within the type `PhantomData<(&pyo3::gil::GILGuard, impl_::not_send::NotSend)>` + --> $RUST/core/src/marker.rs + | + | pub struct PhantomData; + | ^^^^^^^^^^^ +note: required because it appears within the type `pyo3::Python<'_>` + --> src/marker.rs + | + | pub struct Python<'py>(PhantomData<(&'py GILGuard, NotSend)>); + | ^^^^^^ +note: required because it appears within the type `pyo3::Bound<'_, pyo3::PyAny>` + --> src/instance.rs + | + | pub struct Bound<'py, T>(Python<'py>, ManuallyDrop>); + | ^^^^^ + = note: required because it appears within the type `&pyo3::Bound<'_, pyo3::PyAny>` + = note: required for `&&pyo3::Bound<'_, pyo3::PyAny>` to implement `Send` +note: required because it's used within this closure + --> tests/ui/invalid_pyfunctions.rs:45:1 + | +45 | #[pyfunction(allow_threads)] + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + = note: required for `{closure@$DIR/tests/ui/invalid_pyfunctions.rs:45:1: 45:29}` to implement `Ungil` +note: required by a bound in `pyo3::Python::<'py>::allow_threads` + --> src/marker.rs + | + | pub fn allow_threads(self, f: F) -> T + | ------------- required by a bound in this associated function + | where + | F: Ungil + FnOnce() -> T, + | ^^^^^ required by this bound in `Python::<'py>::allow_threads` + = note: this error originates in the attribute macro `pyfunction` (in Nightly builds, run with -Z macro-backtrace for more info)