From 718f8871743b8a831eade7a672d40385fbc75edc Mon Sep 17 00:00:00 2001 From: David Hewitt Date: Mon, 26 Feb 2024 23:17:39 +0000 Subject: [PATCH] wip bound docs --- guide/src/SUMMARY.md | 27 +- guide/src/conversions/tables.md | 71 +-- guide/src/function-calls.md | 1 + guide/src/index.md | 16 + guide/src/python_from_rust.md | 506 ------------------ .../python_from_rust/calling-existing-code.md | 390 ++++++++++++++ guide/src/python_from_rust/function-calls.md | 114 ++++ guide/src/rust_from_python.md | 13 + guide/src/types.md | 55 +- 9 files changed, 629 insertions(+), 564 deletions(-) create mode 100644 guide/src/function-calls.md create mode 100644 guide/src/python_from_rust/calling-existing-code.md create mode 100644 guide/src/python_from_rust/function-calls.md create mode 100644 guide/src/rust_from_python.md diff --git a/guide/src/SUMMARY.md b/guide/src/SUMMARY.md index 34775a0d185..b71229efac3 100644 --- a/guide/src/SUMMARY.md +++ b/guide/src/SUMMARY.md @@ -5,22 +5,25 @@ --- - [Getting started](getting_started.md) -- [Python modules](module.md) -- [Python functions](function.md) - - [Function signatures](function/signature.md) - - [Error handling](function/error_handling.md) -- [Python classes](class.md) - - [Class customizations](class/protocols.md) - - [Basic object customization](class/object.md) - - [Emulating numeric types](class/numeric.md) - - [Emulating callable objects](class/call.md) +- [Using Rust from Python](rust_from_python.md) + - [Python modules](module.md) + - [Python functions](function.md) + - [Function signatures](function/signature.md) + - [Error handling](function/error_handling.md) + - [Python classes](class.md) + - [Class customizations](class/protocols.md) + - [Basic object customization](class/object.md) + - [Emulating numeric types](class/numeric.md) + - [Emulating callable objects](class/call.md) +- [Using Python from Rust](python_from_rust.md) + - [Python object types](types.md) + - [Python exceptions](exception.md) + - [Calling Python functions](python_from_rust/function-calls.md) + - [Executing existing Python code](python_from_rust/calling-existing-code.md) - [Type conversions](conversions.md) - [Mapping of Rust types to Python types](conversions/tables.md) - [Conversion traits](conversions/traits.md) -- [Python exceptions](exception.md) -- [Calling Python from Rust](python_from_rust.md) - [Using `async` and `await`](async-await.md) -- [GIL, mutability and object types](types.md) - [Parallelism](parallelism.md) - [Debugging](debugging.md) - [Features reference](features.md) diff --git a/guide/src/conversions/tables.md b/guide/src/conversions/tables.md index b0582431156..3996798c344 100644 --- a/guide/src/conversions/tables.md +++ b/guide/src/conversions/tables.md @@ -12,49 +12,49 @@ The table below contains the Python type and the corresponding function argument | Python | Rust | Rust (Python-native) | | ------------- |:-------------------------------:|:--------------------:| -| `object` | - | `&PyAny` | -| `str` | `String`, `Cow`, `&str`, `char`, `OsString`, `PathBuf`, `Path` | `&PyString`, `&PyUnicode` | -| `bytes` | `Vec`, `&[u8]`, `Cow<[u8]>` | `&PyBytes` | -| `bool` | `bool` | `&PyBool` | -| `int` | `i8`, `u8`, `i16`, `u16`, `i32`, `u32`, `i64`, `u64`, `i128`, `u128`, `isize`, `usize`, `num_bigint::BigInt`[^1], `num_bigint::BigUint`[^1] | `&PyLong` | -| `float` | `f32`, `f64` | `&PyFloat` | -| `complex` | `num_complex::Complex`[^2] | `&PyComplex` | -| `list[T]` | `Vec` | `&PyList` | -| `dict[K, V]` | `HashMap`, `BTreeMap`, `hashbrown::HashMap`[^3], `indexmap::IndexMap`[^4] | `&PyDict` | -| `tuple[T, U]` | `(T, U)`, `Vec` | `&PyTuple` | -| `set[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^3] | `&PySet` | -| `frozenset[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^3] | `&PyFrozenSet` | -| `bytearray` | `Vec`, `Cow<[u8]>` | `&PyByteArray` | -| `slice` | - | `&PySlice` | -| `type` | - | `&PyType` | -| `module` | - | `&PyModule` | +| `object` | - | `PyAny` | +| `str` | `String`, `Cow`, `&str`, `char`, `OsString`, `PathBuf`, `Path` | `PyString`, `PyUnicode` | +| `bytes` | `Vec`, `&[u8]`, `Cow<[u8]>` | `PyBytes` | +| `bool` | `bool` | `PyBool` | +| `int` | `i8`, `u8`, `i16`, `u16`, `i32`, `u32`, `i64`, `u64`, `i128`, `u128`, `isize`, `usize`, `num_bigint::BigInt`[^1], `num_bigint::BigUint`[^1] | `PyLong` | +| `float` | `f32`, `f64` | `PyFloat` | +| `complex` | `num_complex::Complex`[^2] | `PyComplex` | +| `list[T]` | `Vec` | `PyList` | +| `dict[K, V]` | `HashMap`, `BTreeMap`, `hashbrown::HashMap`[^3], `indexmap::IndexMap`[^4] | `PyDict` | +| `tuple[T, U]` | `(T, U)`, `Vec` | `PyTuple` | +| `set[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^3] | `PySet` | +| `frozenset[T]` | `HashSet`, `BTreeSet`, `hashbrown::HashSet`[^3] | `PyFrozenSet` | +| `bytearray` | `Vec`, `Cow<[u8]>` | `PyByteArray` | +| `slice` | - | `PySlice` | +| `type` | - | `PyType` | +| `module` | - | `PyModule` | | `collections.abc.Buffer` | - | `PyBuffer` | -| `datetime.datetime` | `SystemTime`, `chrono::DateTime`[^5], `chrono::NaiveDateTime`[^5] | `&PyDateTime` | -| `datetime.date` | `chrono::NaiveDate`[^5] | `&PyDate` | -| `datetime.time` | `chrono::NaiveTime`[^5] | `&PyTime` | -| `datetime.tzinfo` | `chrono::FixedOffset`[^5], `chrono::Utc`[^5], `chrono_tz::TimeZone`[^6] | `&PyTzInfo` | -| `datetime.timedelta` | `Duration`, `chrono::Duration`[^5] | `&PyDelta` | +| `datetime.datetime` | `SystemTime`, `chrono::DateTime`[^5], `chrono::NaiveDateTime`[^5] | `PyDateTime` | +| `datetime.date` | `chrono::NaiveDate`[^5] | `PyDate` | +| `datetime.time` | `chrono::NaiveTime`[^5] | `PyTime` | +| `datetime.tzinfo` | `chrono::FixedOffset`[^5], `chrono::Utc`[^5], `chrono_tz::TimeZone`[^6] | `PyTzInfo` | +| `datetime.timedelta` | `Duration`, `chrono::Duration`[^5] | `PyDelta` | | `decimal.Decimal` | `rust_decimal::Decimal`[^7] | - | | `ipaddress.IPv4Address` | `std::net::IpAddr`, `std::net::IpV4Addr` | - | | `ipaddress.IPv6Address` | `std::net::IpAddr`, `std::net::IpV6Addr` | - | -| `os.PathLike ` | `PathBuf`, `Path` | `&PyString`, `&PyUnicode` | -| `pathlib.Path` | `PathBuf`, `Path` | `&PyString`, `&PyUnicode` | +| `os.PathLike ` | `PathBuf`, `Path` | `PyString`, `PyUnicode` | +| `pathlib.Path` | `PathBuf`, `Path` | `PyString`, `PyUnicode` | | `typing.Optional[T]` | `Option` | - | -| `typing.Sequence[T]` | `Vec` | `&PySequence` | +| `typing.Sequence[T]` | `Vec` | `PySequence` | | `typing.Mapping[K, V]` | `HashMap`, `BTreeMap`, `hashbrown::HashMap`[^3], `indexmap::IndexMap`[^4] | `&PyMapping` | -| `typing.Iterator[Any]` | - | `&PyIterator` | +| `typing.Iterator[Any]` | - | `PyIterator` | | `typing.Union[...]` | See [`#[derive(FromPyObject)]`](traits.html#deriving-a-hrefhttpsdocsrspyo3latestpyo3conversiontraitfrompyobjecthtmlfrompyobjecta-for-enums) | - | -There are also a few special types related to the GIL and Rust-defined `#[pyclass]`es which may come in useful: +It is also worth remembering the following special types: -| What | Description | -| ------------- | ------------------------------------- | -| `Python` | A GIL token, used to pass to PyO3 constructors to prove ownership of the GIL | -| `Py` | A Python object isolated from the GIL lifetime. This can be sent to other threads. | -| `PyObject` | An alias for `Py` | -| `&PyCell` | A `#[pyclass]` value owned by Python. | -| `PyRef` | A `#[pyclass]` borrowed immutably. | -| `PyRefMut` | A `#[pyclass]` borrowed mutably. | +| What | Description | +| ---------------- | ------------------------------------- | +| `Python<'py>` | A GIL token, used to pass to PyO3 constructors to prove ownership of the GIL. | +| `Bound<'py, T>` | A Python object connected to the GIL lifetime. This provides access to most of PyO3's APIs. | +| `Py` | A Python object isolated from the GIL lifetime. This can be sent to other threads. | +| `PyObject` | An alias for `Py` | +| `PyRef` | A `#[pyclass]` borrowed immutably. | +| `PyRefMut` | A `#[pyclass]` borrowed mutably. | For more detail on accepting `#[pyclass]` values as function arguments, see [the section of this guide on Python Classes](../class.md). @@ -95,7 +95,8 @@ Finally, the following Rust types are also able to convert to Python as return v | `BTreeMap` | `Dict[K, V]` | | `HashSet` | `Set[T]` | | `BTreeSet` | `Set[T]` | -| `&PyCell` | `T` | +| `Py` | `T` | +| `Bound` | `T` | | `PyRef` | `T` | | `PyRefMut` | `T` | diff --git a/guide/src/function-calls.md b/guide/src/function-calls.md new file mode 100644 index 00000000000..e5c9c1ed9b3 --- /dev/null +++ b/guide/src/function-calls.md @@ -0,0 +1 @@ +# Calling Python functions diff --git a/guide/src/index.md b/guide/src/index.md index 80534caea5f..f47334e1d28 100644 --- a/guide/src/index.md +++ b/guide/src/index.md @@ -2,6 +2,22 @@ Welcome to the PyO3 user guide! This book is a companion to [PyO3's API docs](https://docs.rs/pyo3). It contains examples and documentation to explain all of PyO3's use cases in detail. +The rough order of material in this user guide is as follows: + 1. Getting started + 2. Wrapping Rust code for use from Python + 3. How to use Python code from Rust + 4. Remaining topics which go into advanced concepts in detail + Please choose from the chapters on the left to jump to individual topics, or continue below to start with PyO3's README. +
+⚠️ Warning: API update in progress 🛠️ + +PyO3 0.21 has introduced a significant new API, termed the "Bound" API after the new smart pointer `Bound`. + +While much of the guide has been updated to the new API, it is possible some stray references to the older "GIL Refs" API such as `&PyAny` may remain. +
+ +
+ {{#include ../../README.md}} diff --git a/guide/src/python_from_rust.md b/guide/src/python_from_rust.md index d8f3214f58b..9cc647ffa7c 100644 --- a/guide/src/python_from_rust.md +++ b/guide/src/python_from_rust.md @@ -3,509 +3,3 @@ This chapter of the guide documents some ways to interact with Python code from Rust: - How to call Python functions - How to execute existing Python code - -## Calling Python functions - -Any Python-native object reference (such as `&PyAny`, `&PyList`, or `&PyCell`) can be used to call Python functions. - -PyO3 offers two APIs to make function calls: - -* [`call`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call) - call any callable Python object. -* [`call_method`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call_method) - call a method on the Python object. - -Both of these APIs take `args` and `kwargs` arguments (for positional and keyword arguments respectively). There are variants for less complex calls: - -* [`call1`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call1) and [`call_method1`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call_method1) to call only with positional `args`. -* [`call0`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call0) and [`call_method0`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call_method0) to call with no arguments. - -For convenience the [`Py`](types.html#pyt-and-pyobject) smart pointer also exposes these same six API methods, but needs a `Python` token as an additional first argument to prove the GIL is held. - -The example below calls a Python function behind a `PyObject` (aka `Py`) reference: - -```rust -use pyo3::prelude::*; -use pyo3::types::PyTuple; - -fn main() -> PyResult<()> { - let arg1 = "arg1"; - let arg2 = "arg2"; - let arg3 = "arg3"; - - Python::with_gil(|py| { - let fun: Py = PyModule::from_code_bound( - py, - "def example(*args, **kwargs): - if args != (): - print('called with args', args) - if kwargs != {}: - print('called with kwargs', kwargs) - if args == () and kwargs == {}: - print('called with no arguments')", - "", - "", - )? - .getattr("example")? - .into(); - - // call object without any arguments - fun.call0(py)?; - - // call object with PyTuple - let args = PyTuple::new_bound(py, &[arg1, arg2, arg3]); - fun.call1(py, args)?; - - // pass arguments as rust tuple - let args = (arg1, arg2, arg3); - fun.call1(py, args)?; - Ok(()) - }) -} -``` - -### Creating keyword arguments - -For the `call` and `call_method` APIs, `kwargs` can be `None` or `Some(&PyDict)`. You can use the [`IntoPyDict`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.IntoPyDict.html) trait to convert other dict-like containers, e.g. `HashMap` or `BTreeMap`, as well as tuples with up to 10 elements and `Vec`s where each element is a two-element tuple. - -```rust -use pyo3::prelude::*; -use pyo3::types::IntoPyDict; -use std::collections::HashMap; - -fn main() -> PyResult<()> { - let key1 = "key1"; - let val1 = 1; - let key2 = "key2"; - let val2 = 2; - - Python::with_gil(|py| { - let fun: Py = PyModule::from_code_bound( - py, - "def example(*args, **kwargs): - if args != (): - print('called with args', args) - if kwargs != {}: - print('called with kwargs', kwargs) - if args == () and kwargs == {}: - print('called with no arguments')", - "", - "", - )? - .getattr("example")? - .into(); - - // call object with PyDict - let kwargs = [(key1, val1)].into_py_dict_bound(py); - fun.call_bound(py, (), Some(&kwargs))?; - - // pass arguments as Vec - let kwargs = vec![(key1, val1), (key2, val2)]; - fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?; - - // pass arguments as HashMap - let mut kwargs = HashMap::<&str, i32>::new(); - kwargs.insert(key1, 1); - fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?; - - Ok(()) - }) -} -``` - -
- -During PyO3's [migration from "GIL Refs" to the `Bound` smart pointer](./migration.md#migrating-from-the-gil-refs-api-to-boundt), [`Py::call`]({{#PYO3_DOCS_URL}}/pyo3/struct.py#method.call) is temporarily named `call_bound` (and `call_method` is temporarily `call_method_bound`). - -(This temporary naming is only the case for the `Py` smart pointer. The methods on the `&PyAny` GIL Ref such as `call` have not been given replacements, and the methods on the `Bound` smart pointer such as [`Bound::call`]({#PYO3_DOCS_URL}}/pyo3/prelude/trait.pyanymethods#tymethod.call) already use follow the newest API conventions.) - -
- -## Executing existing Python code - -If you already have some existing Python code that you need to execute from Rust, the following FAQs can help you select the right PyO3 functionality for your situation: - -### Want to access Python APIs? Then use `PyModule::import`. - -[`Pymodule::import`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.import) can -be used to get handle to a Python module from Rust. You can use this to import and use any Python -module available in your environment. - -```rust -use pyo3::prelude::*; - -fn main() -> PyResult<()> { - Python::with_gil(|py| { - let builtins = PyModule::import_bound(py, "builtins")?; - let total: i32 = builtins - .getattr("sum")? - .call1((vec![1, 2, 3],))? - .extract()?; - assert_eq!(total, 6); - Ok(()) - }) -} -``` - -### Want to run just an expression? Then use `eval`. - -[`Python::eval`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.eval) is -a method to execute a [Python expression](https://docs.python.org/3.7/reference/expressions.html) -and return the evaluated value as a `&PyAny` object. - -```rust -use pyo3::prelude::*; - -# fn main() -> Result<(), ()> { -Python::with_gil(|py| { - let result = py - .eval_bound("[i * 10 for i in range(5)]", None, None) - .map_err(|e| { - e.print_and_set_sys_last_vars(py); - })?; - let res: Vec = result.extract().unwrap(); - assert_eq!(res, vec![0, 10, 20, 30, 40]); - Ok(()) -}) -# } -``` - -### Want to run statements? Then use `run`. - -[`Python::run`] is a method to execute one or more -[Python statements](https://docs.python.org/3.7/reference/simple_stmts.html). -This method returns nothing (like any Python statement), but you can get -access to manipulated objects via the `locals` dict. - -You can also use the [`py_run!`] macro, which is a shorthand for [`Python::run`]. -Since [`py_run!`] panics on exceptions, we recommend you use this macro only for -quickly testing your Python extensions. - -```rust -use pyo3::prelude::*; -use pyo3::py_run; - -# fn main() { -#[pyclass] -struct UserData { - id: u32, - name: String, -} - -#[pymethods] -impl UserData { - fn as_tuple(&self) -> (u32, String) { - (self.id, self.name.clone()) - } - - fn __repr__(&self) -> PyResult { - Ok(format!("User {}(id: {})", self.name, self.id)) - } -} - -Python::with_gil(|py| { - let userdata = UserData { - id: 34, - name: "Yu".to_string(), - }; - let userdata = Py::new(py, userdata).unwrap(); - let userdata_as_tuple = (34, "Yu"); - py_run!(py, userdata userdata_as_tuple, r#" -assert repr(userdata) == "User Yu(id: 34)" -assert userdata.as_tuple() == userdata_as_tuple - "#); -}) -# } -``` - -## You have a Python file or code snippet? Then use `PyModule::from_code`. - -[`PyModule::from_code`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.from_code) -can be used to generate a Python module which can then be used just as if it was imported with -`PyModule::import`. - -**Warning**: This will compile and execute code. **Never** pass untrusted code -to this function! - -```rust -use pyo3::{ - prelude::*, - types::IntoPyDict, -}; - -# fn main() -> PyResult<()> { -Python::with_gil(|py| { - let activators = PyModule::from_code_bound( - py, - r#" -def relu(x): - """see https://en.wikipedia.org/wiki/Rectifier_(neural_networks)""" - return max(0.0, x) - -def leaky_relu(x, slope=0.01): - return x if x >= 0 else x * slope - "#, - "activators.py", - "activators", - )?; - - let relu_result: f64 = activators.getattr("relu")?.call1((-1.0,))?.extract()?; - assert_eq!(relu_result, 0.0); - - let kwargs = [("slope", 0.2)].into_py_dict_bound(py); - let lrelu_result: f64 = activators - .getattr("leaky_relu")? - .call((-1.0,), Some(&kwargs))? - .extract()?; - assert_eq!(lrelu_result, -0.2); -# Ok(()) -}) -# } -``` - -### Want to embed Python in Rust with additional modules? - -Python maintains the `sys.modules` dict as a cache of all imported modules. -An import in Python will first attempt to lookup the module from this dict, -and if not present will use various strategies to attempt to locate and load -the module. - -The [`append_to_inittab`]({{#PYO3_DOCS_URL}}/pyo3/macro.append_to_inittab.html) -macro can be used to add additional `#[pymodule]` modules to an embedded -Python interpreter. The macro **must** be invoked _before_ initializing Python. - -As an example, the below adds the module `foo` to the embedded interpreter: - -```rust -use pyo3::prelude::*; - -#[pyfunction] -fn add_one(x: i64) -> i64 { - x + 1 -} - -#[pymodule] -fn foo(foo_module: &Bound<'_, PyModule>) -> PyResult<()> { - foo_module.add_function(wrap_pyfunction!(add_one, foo_module)?)?; - Ok(()) -} - -fn main() -> PyResult<()> { - pyo3::append_to_inittab!(foo); - Python::with_gil(|py| Python::run_bound(py, "import foo; foo.add_one(6)", None, None)) -} -``` - -If `append_to_inittab` cannot be used due to constraints in the program, -an alternative is to create a module using [`PyModule::new`] -and insert it manually into `sys.modules`: - -```rust -use pyo3::prelude::*; -use pyo3::types::PyDict; - -#[pyfunction] -pub fn add_one(x: i64) -> i64 { - x + 1 -} - -fn main() -> PyResult<()> { - Python::with_gil(|py| { - // Create new module - let foo_module = PyModule::new_bound(py, "foo")?; - foo_module.add_function(wrap_pyfunction!(add_one, &foo_module)?)?; - - // Import and get sys.modules - let sys = PyModule::import_bound(py, "sys")?; - let py_modules: Bound<'_, PyDict> = sys.getattr("modules")?.downcast_into()?; - - // Insert foo into sys.modules - py_modules.set_item("foo", foo_module)?; - - // Now we can import + run our python code - Python::run_bound(py, "import foo; foo.add_one(6)", None, None) - }) -} -``` - -### Include multiple Python files - -You can include a file at compile time by using -[`std::include_str`](https://doc.rust-lang.org/std/macro.include_str.html) macro. - -Or you can load a file at runtime by using -[`std::fs::read_to_string`](https://doc.rust-lang.org/std/fs/fn.read_to_string.html) function. - -Many Python files can be included and loaded as modules. If one file depends on -another you must preserve correct order while declaring `PyModule`. - -Example directory structure: -```text -. -├── Cargo.lock -├── Cargo.toml -├── python_app -│ ├── app.py -│ └── utils -│ └── foo.py -└── src - └── main.rs -``` - -`python_app/app.py`: -```python -from utils.foo import bar - - -def run(): - return bar() -``` - -`python_app/utils/foo.py`: -```python -def bar(): - return "baz" -``` - -The example below shows: -* how to include content of `app.py` and `utils/foo.py` into your rust binary -* how to call function `run()` (declared in `app.py`) that needs function - imported from `utils/foo.py` - -`src/main.rs`: -```rust,ignore -use pyo3::prelude::*; - -fn main() -> PyResult<()> { - let py_foo = include_str!(concat!( - env!("CARGO_MANIFEST_DIR"), - "/python_app/utils/foo.py" - )); - let py_app = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/python_app/app.py")); - let from_python = Python::with_gil(|py| -> PyResult> { - PyModule::from_code_bound(py, py_foo, "utils.foo", "utils.foo")?; - let app: Py = PyModule::from_code_bound(py, py_app, "", "")? - .getattr("run")? - .into(); - app.call0(py) - }); - - println!("py: {}", from_python?); - Ok(()) -} -``` - -The example below shows: -* how to load content of `app.py` at runtime so that it sees its dependencies - automatically -* how to call function `run()` (declared in `app.py`) that needs function - imported from `utils/foo.py` - -It is recommended to use absolute paths because then your binary can be run -from anywhere as long as your `app.py` is in the expected directory (in this example -that directory is `/usr/share/python_app`). - -`src/main.rs`: -```rust,no_run -use pyo3::prelude::*; -use pyo3::types::PyList; -use std::fs; -use std::path::Path; - -fn main() -> PyResult<()> { - let path = Path::new("/usr/share/python_app"); - let py_app = fs::read_to_string(path.join("app.py"))?; - let from_python = Python::with_gil(|py| -> PyResult> { - let syspath = py.import_bound("sys")?.getattr("path")?.downcast_into::()?; - syspath.insert(0, &path)?; - let app: Py = PyModule::from_code_bound(py, &py_app, "", "")? - .getattr("run")? - .into(); - app.call0(py) - }); - - println!("py: {}", from_python?); - Ok(()) -} -``` - - -[`Python::run`]: {{#PYO3_DOCS_URL}}/pyo3/struct.Python.html#method.run -[`py_run!`]: {{#PYO3_DOCS_URL}}/pyo3/macro.py_run.html - -## Need to use a context manager from Rust? - -Use context managers by directly invoking `__enter__` and `__exit__`. - -```rust -use pyo3::prelude::*; - -fn main() { - Python::with_gil(|py| { - let custom_manager = PyModule::from_code_bound( - py, - r#" -class House(object): - def __init__(self, address): - self.address = address - def __enter__(self): - print(f"Welcome to {self.address}!") - def __exit__(self, type, value, traceback): - if type: - print(f"Sorry you had {type} trouble at {self.address}") - else: - print(f"Thank you for visiting {self.address}, come again soon!") - - "#, - "house.py", - "house", - ) - .unwrap(); - - let house_class = custom_manager.getattr("House").unwrap(); - let house = house_class.call1(("123 Main Street",)).unwrap(); - - house.call_method0("__enter__").unwrap(); - - let result = py.eval_bound("undefined_variable + 1", None, None); - - // If the eval threw an exception we'll pass it through to the context manager. - // Otherwise, __exit__ is called with empty arguments (Python "None"). - match result { - Ok(_) => { - let none = py.None(); - house - .call_method1("__exit__", (&none, &none, &none)) - .unwrap(); - } - Err(e) => { - house - .call_method1("__exit__", (e.get_type_bound(py), e.value_bound(py), e.traceback_bound(py))) - .unwrap(); - } - } - }) -} -``` - -## Handling system signals/interrupts (Ctrl-C) - -The best way to handle system signals when running Rust code is to periodically call `Python::check_signals` to handle any signals captured by Python's signal handler. See also [the FAQ entry](./faq.md#ctrl-c-doesnt-do-anything-while-my-rust-code-is-executing). - -Alternatively, set Python's `signal` module to take the default action for a signal: - -```rust -use pyo3::prelude::*; - -# fn main() -> PyResult<()> { -Python::with_gil(|py| -> PyResult<()> { - let signal = py.import_bound("signal")?; - // Set SIGINT to have the default action - signal - .getattr("signal")? - .call1((signal.getattr("SIGINT")?, signal.getattr("SIG_DFL")?))?; - Ok(()) -}) -# } -``` - - -[`PyModule::new`]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.new diff --git a/guide/src/python_from_rust/calling-existing-code.md b/guide/src/python_from_rust/calling-existing-code.md new file mode 100644 index 00000000000..1c2ee3bcc3a --- /dev/null +++ b/guide/src/python_from_rust/calling-existing-code.md @@ -0,0 +1,390 @@ +# Executing existing Python code + +If you already have some existing Python code that you need to execute from Rust, the following FAQs can help you select the right PyO3 functionality for your situation: + +## Want to access Python APIs? Then use `PyModule::import`. + +[`Pymodule::import`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.import) can +be used to get handle to a Python module from Rust. You can use this to import and use any Python +module available in your environment. + +```rust +use pyo3::prelude::*; + +fn main() -> PyResult<()> { + Python::with_gil(|py| { + let builtins = PyModule::import_bound(py, "builtins")?; + let total: i32 = builtins + .getattr("sum")? + .call1((vec![1, 2, 3],))? + .extract()?; + assert_eq!(total, 6); + Ok(()) + }) +} +``` + +## Want to run just an expression? Then use `eval`. + +[`Python::eval`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.eval) is +a method to execute a [Python expression](https://docs.python.org/3.7/reference/expressions.html) +and return the evaluated value as a `&PyAny` object. + +```rust +use pyo3::prelude::*; + +# fn main() -> Result<(), ()> { +Python::with_gil(|py| { + let result = py + .eval_bound("[i * 10 for i in range(5)]", None, None) + .map_err(|e| { + e.print_and_set_sys_last_vars(py); + })?; + let res: Vec = result.extract().unwrap(); + assert_eq!(res, vec![0, 10, 20, 30, 40]); + Ok(()) +}) +# } +``` + +## Want to run statements? Then use `run`. + +[`Python::run`] is a method to execute one or more +[Python statements](https://docs.python.org/3.7/reference/simple_stmts.html). +This method returns nothing (like any Python statement), but you can get +access to manipulated objects via the `locals` dict. + +You can also use the [`py_run!`] macro, which is a shorthand for [`Python::run`]. +Since [`py_run!`] panics on exceptions, we recommend you use this macro only for +quickly testing your Python extensions. + +```rust +use pyo3::prelude::*; +use pyo3::py_run; + +# fn main() { +#[pyclass] +struct UserData { + id: u32, + name: String, +} + +#[pymethods] +impl UserData { + fn as_tuple(&self) -> (u32, String) { + (self.id, self.name.clone()) + } + + fn __repr__(&self) -> PyResult { + Ok(format!("User {}(id: {})", self.name, self.id)) + } +} + +Python::with_gil(|py| { + let userdata = UserData { + id: 34, + name: "Yu".to_string(), + }; + let userdata = Py::new(py, userdata).unwrap(); + let userdata_as_tuple = (34, "Yu"); + py_run!(py, userdata userdata_as_tuple, r#" +assert repr(userdata) == "User Yu(id: 34)" +assert userdata.as_tuple() == userdata_as_tuple + "#); +}) +# } +``` + +## You have a Python file or code snippet? Then use `PyModule::from_code`. + +[`PyModule::from_code`]({{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.from_code) +can be used to generate a Python module which can then be used just as if it was imported with +`PyModule::import`. + +**Warning**: This will compile and execute code. **Never** pass untrusted code +to this function! + +```rust +use pyo3::{ + prelude::*, + types::IntoPyDict, +}; + +# fn main() -> PyResult<()> { +Python::with_gil(|py| { + let activators = PyModule::from_code_bound( + py, + r#" +def relu(x): + """see https://en.wikipedia.org/wiki/Rectifier_(neural_networks)""" + return max(0.0, x) + +def leaky_relu(x, slope=0.01): + return x if x >= 0 else x * slope + "#, + "activators.py", + "activators", + )?; + + let relu_result: f64 = activators.getattr("relu")?.call1((-1.0,))?.extract()?; + assert_eq!(relu_result, 0.0); + + let kwargs = [("slope", 0.2)].into_py_dict_bound(py); + let lrelu_result: f64 = activators + .getattr("leaky_relu")? + .call((-1.0,), Some(&kwargs))? + .extract()?; + assert_eq!(lrelu_result, -0.2); +# Ok(()) +}) +# } +``` + +## Want to embed Python in Rust with additional modules? + +Python maintains the `sys.modules` dict as a cache of all imported modules. +An import in Python will first attempt to lookup the module from this dict, +and if not present will use various strategies to attempt to locate and load +the module. + +The [`append_to_inittab`]({{#PYO3_DOCS_URL}}/pyo3/macro.append_to_inittab.html) +macro can be used to add additional `#[pymodule]` modules to an embedded +Python interpreter. The macro **must** be invoked _before_ initializing Python. + +As an example, the below adds the module `foo` to the embedded interpreter: + +```rust +use pyo3::prelude::*; + +#[pyfunction] +fn add_one(x: i64) -> i64 { + x + 1 +} + +#[pymodule] +fn foo(foo_module: &Bound<'_, PyModule>) -> PyResult<()> { + foo_module.add_function(wrap_pyfunction!(add_one, foo_module)?)?; + Ok(()) +} + +fn main() -> PyResult<()> { + pyo3::append_to_inittab!(foo); + Python::with_gil(|py| Python::run_bound(py, "import foo; foo.add_one(6)", None, None)) +} +``` + +If `append_to_inittab` cannot be used due to constraints in the program, +an alternative is to create a module using [`PyModule::new`] +and insert it manually into `sys.modules`: + +```rust +use pyo3::prelude::*; +use pyo3::types::PyDict; + +#[pyfunction] +pub fn add_one(x: i64) -> i64 { + x + 1 +} + +fn main() -> PyResult<()> { + Python::with_gil(|py| { + // Create new module + let foo_module = PyModule::new_bound(py, "foo")?; + foo_module.add_function(wrap_pyfunction!(add_one, &foo_module)?)?; + + // Import and get sys.modules + let sys = PyModule::import_bound(py, "sys")?; + let py_modules: Bound<'_, PyDict> = sys.getattr("modules")?.downcast_into()?; + + // Insert foo into sys.modules + py_modules.set_item("foo", foo_module)?; + + // Now we can import + run our python code + Python::run_bound(py, "import foo; foo.add_one(6)", None, None) + }) +} +``` + +## Include multiple Python files + +You can include a file at compile time by using +[`std::include_str`](https://doc.rust-lang.org/std/macro.include_str.html) macro. + +Or you can load a file at runtime by using +[`std::fs::read_to_string`](https://doc.rust-lang.org/std/fs/fn.read_to_string.html) function. + +Many Python files can be included and loaded as modules. If one file depends on +another you must preserve correct order while declaring `PyModule`. + +Example directory structure: +```text +. +├── Cargo.lock +├── Cargo.toml +├── python_app +│ ├── app.py +│ └── utils +│ └── foo.py +└── src + └── main.rs +``` + +`python_app/app.py`: +```python +from utils.foo import bar + + +def run(): + return bar() +``` + +`python_app/utils/foo.py`: +```python +def bar(): + return "baz" +``` + +The example below shows: +* how to include content of `app.py` and `utils/foo.py` into your rust binary +* how to call function `run()` (declared in `app.py`) that needs function + imported from `utils/foo.py` + +`src/main.rs`: +```rust,ignore +use pyo3::prelude::*; + +fn main() -> PyResult<()> { + let py_foo = include_str!(concat!( + env!("CARGO_MANIFEST_DIR"), + "/python_app/utils/foo.py" + )); + let py_app = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/python_app/app.py")); + let from_python = Python::with_gil(|py| -> PyResult> { + PyModule::from_code_bound(py, py_foo, "utils.foo", "utils.foo")?; + let app: Py = PyModule::from_code_bound(py, py_app, "", "")? + .getattr("run")? + .into(); + app.call0(py) + }); + + println!("py: {}", from_python?); + Ok(()) +} +``` + +The example below shows: +* how to load content of `app.py` at runtime so that it sees its dependencies + automatically +* how to call function `run()` (declared in `app.py`) that needs function + imported from `utils/foo.py` + +It is recommended to use absolute paths because then your binary can be run +from anywhere as long as your `app.py` is in the expected directory (in this example +that directory is `/usr/share/python_app`). + +`src/main.rs`: +```rust,no_run +use pyo3::prelude::*; +use pyo3::types::PyList; +use std::fs; +use std::path::Path; + +fn main() -> PyResult<()> { + let path = Path::new("/usr/share/python_app"); + let py_app = fs::read_to_string(path.join("app.py"))?; + let from_python = Python::with_gil(|py| -> PyResult> { + let syspath = py.import_bound("sys")?.getattr("path")?.downcast_into::()?; + syspath.insert(0, &path)?; + let app: Py = PyModule::from_code_bound(py, &py_app, "", "")? + .getattr("run")? + .into(); + app.call0(py) + }); + + println!("py: {}", from_python?); + Ok(()) +} +``` + + +[`Python::run`]: {{#PYO3_DOCS_URL}}/pyo3/struct.Python.html#method.run +[`py_run!`]: {{#PYO3_DOCS_URL}}/pyo3/macro.py_run.html + +#£ Need to use a context manager from Rust? + +Use context managers by directly invoking `__enter__` and `__exit__`. + +```rust +use pyo3::prelude::*; + +fn main() { + Python::with_gil(|py| { + let custom_manager = PyModule::from_code_bound( + py, + r#" +class House(object): + def __init__(self, address): + self.address = address + def __enter__(self): + print(f"Welcome to {self.address}!") + def __exit__(self, type, value, traceback): + if type: + print(f"Sorry you had {type} trouble at {self.address}") + else: + print(f"Thank you for visiting {self.address}, come again soon!") + + "#, + "house.py", + "house", + ) + .unwrap(); + + let house_class = custom_manager.getattr("House").unwrap(); + let house = house_class.call1(("123 Main Street",)).unwrap(); + + house.call_method0("__enter__").unwrap(); + + let result = py.eval_bound("undefined_variable + 1", None, None); + + // If the eval threw an exception we'll pass it through to the context manager. + // Otherwise, __exit__ is called with empty arguments (Python "None"). + match result { + Ok(_) => { + let none = py.None(); + house + .call_method1("__exit__", (&none, &none, &none)) + .unwrap(); + } + Err(e) => { + house + .call_method1("__exit__", (e.get_type_bound(py), e.value_bound(py), e.traceback_bound(py))) + .unwrap(); + } + } + }) +} +``` + +## Handling system signals/interrupts (Ctrl-C) + +The best way to handle system signals when running Rust code is to periodically call `Python::check_signals` to handle any signals captured by Python's signal handler. See also [the FAQ entry](./faq.md#ctrl-c-doesnt-do-anything-while-my-rust-code-is-executing). + +Alternatively, set Python's `signal` module to take the default action for a signal: + +```rust +use pyo3::prelude::*; + +# fn main() -> PyResult<()> { +Python::with_gil(|py| -> PyResult<()> { + let signal = py.import_bound("signal")?; + // Set SIGINT to have the default action + signal + .getattr("signal")? + .call1((signal.getattr("SIGINT")?, signal.getattr("SIG_DFL")?))?; + Ok(()) +}) +# } +``` + + +[`PyModule::new`]: {{#PYO3_DOCS_URL}}/pyo3/types/struct.PyModule.html#method.new diff --git a/guide/src/python_from_rust/function-calls.md b/guide/src/python_from_rust/function-calls.md new file mode 100644 index 00000000000..ad9a19419ce --- /dev/null +++ b/guide/src/python_from_rust/function-calls.md @@ -0,0 +1,114 @@ +# Calling Python functions + +The `Bound<'py, T>` smart pointer (such as `Bound<'py, PyAny>`, `Bound<'py, PyList>`, or `Bound<'py, MyClass>`) can be used to call Python functions. + +PyO3 offers two APIs to make function calls: + +* [`call`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call) - call any callable Python object. +* [`call_method`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call_method) - call a method on the Python object. + +Both of these APIs take `args` and `kwargs` arguments (for positional and keyword arguments respectively). There are variants for less complex calls: + +* [`call1`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call1) and [`call_method1`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call_method1) to call only with positional `args`. +* [`call0`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call0) and [`call_method0`]({{#PYO3_DOCS_URL}}/pyo3/prelude/trait.PyAnyMethods#tymethod.call_method0) to call with no arguments. + +For convenience the [`Py`](types.html#pyt-and-pyobject) smart pointer also exposes these same six API methods, but needs a `Python` token as an additional first argument to prove the GIL is held. + +The example below calls a Python function behind a `PyObject` (aka `Py`) reference: + +```rust +use pyo3::prelude::*; +use pyo3::types::PyTuple; + +fn main() -> PyResult<()> { + let arg1 = "arg1"; + let arg2 = "arg2"; + let arg3 = "arg3"; + + Python::with_gil(|py| { + let fun: Py = PyModule::from_code_bound( + py, + "def example(*args, **kwargs): + if args != (): + print('called with args', args) + if kwargs != {}: + print('called with kwargs', kwargs) + if args == () and kwargs == {}: + print('called with no arguments')", + "", + "", + )? + .getattr("example")? + .into(); + + // call object without any arguments + fun.call0(py)?; + + // pass object with Rust tuple of positional arguments + let args = (arg1, arg2, arg3); + fun.call1(py, args)?; + + // call object with Python tuple of positional arguments + let args = PyTuple::new_bound(py, &[arg1, arg2, arg3]); + fun.call1(py, args)?; + Ok(()) + }) +} +``` + +## Creating keyword arguments + +For the `call` and `call_method` APIs, `kwargs` are `Option<&Bound<'py, PyDict>>`, so can either be `None` or `Some(&dict)`. You can use the [`IntoPyDict`]({{#PYO3_DOCS_URL}}/pyo3/types/trait.IntoPyDict.html) trait to convert other dict-like containers, e.g. `HashMap` or `BTreeMap`, as well as tuples with up to 10 elements and `Vec`s where each element is a two-element tuple. + +```rust +use pyo3::prelude::*; +use pyo3::types::IntoPyDict; +use std::collections::HashMap; + +fn main() -> PyResult<()> { + let key1 = "key1"; + let val1 = 1; + let key2 = "key2"; + let val2 = 2; + + Python::with_gil(|py| { + let fun: Py = PyModule::from_code_bound( + py, + "def example(*args, **kwargs): + if args != (): + print('called with args', args) + if kwargs != {}: + print('called with kwargs', kwargs) + if args == () and kwargs == {}: + print('called with no arguments')", + "", + "", + )? + .getattr("example")? + .into(); + + // call object with PyDict + let kwargs = [(key1, val1)].into_py_dict_bound(py); + fun.call_bound(py, (), Some(&kwargs))?; + + // pass arguments as Vec + let kwargs = vec![(key1, val1), (key2, val2)]; + fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?; + + // pass arguments as HashMap + let mut kwargs = HashMap::<&str, i32>::new(); + kwargs.insert(key1, 1); + fun.call_bound(py, (), Some(&kwargs.into_py_dict_bound(py)))?; + + Ok(()) + }) +} +``` + +
+ +During PyO3's [migration from "GIL Refs" to the `Bound` smart pointer](./migration.md#migrating-from-the-gil-refs-api-to-boundt), [`Py::call`]({{#PYO3_DOCS_URL}}/pyo3/struct.py#method.call) is temporarily named `call_bound` (and `call_method` is temporarily `call_method_bound`). + +(This temporary naming is only the case for the `Py` smart pointer. The methods on the `&PyAny` GIL Ref such as `call` have not been given replacements, and the methods on the `Bound` smart pointer such as [`Bound::call`]({#PYO3_DOCS_URL}}/pyo3/prelude/trait.pyanymethods#tymethod.call) already use follow the newest API conventions.) + +
diff --git a/guide/src/rust_from_python.md b/guide/src/rust_from_python.md new file mode 100644 index 00000000000..470d5719098 --- /dev/null +++ b/guide/src/rust_from_python.md @@ -0,0 +1,13 @@ +# Using Rust from Python + +This chapter of the guide is dedicated to explaining how to wrap Rust code into Python objects. + +PyO3 uses Rust's "procedural macros" to provide a powerful yet simple API to denote what Rust code should map into Python objects. + +The three types of Python objects which PyO3 can produce are: + +- Python modules, via the `#[pymodule]` macro +- Python functions, via the `#[pyfunction]` macro +- Python classes, via the `#[pyclass]` macro (plus `#[pymethods]` to define methods for those clases) + +The following subchapters go through each of these in turn. diff --git a/guide/src/types.md b/guide/src/types.md index 372c6c8632f..c2d5988409d 100644 --- a/guide/src/types.md +++ b/guide/src/types.md @@ -1,22 +1,55 @@ -# GIL lifetimes, mutability and Python object types +# Python object types -On first glance, PyO3 provides a huge number of different types that can be used +PyO3 offers two main sets of types to interact with Python objects. This section of the guide expands into detail about these types and how to choose which to use. + +The first set of types is are the "smart pointers" which all Python objects are wrapped in. These are `Py`, `Bound<'py, T>`, and `Borrowed<'a, 'py, T>`. The [first section below](#pyo3s-smart-pointers) expands on each of these in detail and why there are three of them. + +The second set of types are types which fill in the generic parameter `T` of the smart pointers. The most common is `PyAny`, which represents any Python object (similar to Python's `typing.Any`). There are also concrete types for many Python built-in types, such as `PyList`, `PyDict`, and `PyTuple`. User defined `#[pyclass]` types also fit this category. The [second section below](#concrete-python-types) expands on how to use these types. + +Before PyO3 0.21, PyO3's main API to interact with Python objects was a deprecated API known as the "GIL Refs" API, containing reference types such as `&PyAny`, `&PyList`, and `&PyCell` for user-defined `#[pyclass]` types. The [third section below](#the-gil-refs-api) details this deprecated API. + +## PyO3's smart pointers + +PyO3's API offers three generic smart pointers: `Py`, `Bound<'py, T>` and `Borrowed<'a, 'py, T>`. For each of these the type parameter `T` will be filled by a [concrete Python type](#concrete-python-types). For example, a Python list object can be represented by `Py`, `Bound<'py, PyList>`, and `Borrowed<'a, 'py, PyList>`. + +These smart pointers have different numbers of lifetime parameters, which defines how they behave. `Py` has no lifetime parameters, `Bound<'py, T>` has a lifetime parameter `'py`, and `Borrowed<'a, 'py, T>` has two lifetime parameters `'a` and `'py`. + +Python objects are reference counted, like [`std::sync::Arc`](https://doc.rust-lang.org/stable/std/sync/struct.Arc.html). A major reason for these smart pointers is to bring Python's reference counting to a Rust API. + +The recommendation of when to use each of these smart pointers is as follows: + +- Use `Bound<'py, T>` for as much as possible, as it offers the most efficient and complete API. +- Use `Py` mostly just for storage inside Rust `struct`s which do not want to add a lifetime parameter for `Bound<'py, T>`. +- `Borrowed<'a, 'py, T>` is almost never used. It is occasionally present at the boundary between Rust and the Python interpreter, for example when borrowing data from Python tuples (which is safe because they are immutable). + +The sections below also explain these smart pointers in a little more detail. + +### `Py` (and `PyObject`) + +### `Bound<'py, T>` + +### `Borrowed<'a, 'py, T>` + +GIL lifetimes, mutability and Python object types + +At first glance, PyO3 provides a huge number of different types that can be used to wrap or refer to Python objects. This page delves into the details and gives an overview of their intended meaning, with examples when each type is best used. +## Concrete Python types + + ## The Python GIL, mutability, and Rust types -Since Python has no concept of ownership, and works solely with boxed objects, -any Python object can be referenced any number of times, and mutation is allowed -from any reference. +Python code differs from Rust in two key ways: +- There is no concept of ownership; all Python objects are reference counted +- There is no concept of exclusive (`&mut`) references; any reference can mutate a Python object + +PyO3's API for interacting with Python objects is built with this in mind. There are two generic "smart pointers" to Python objects, `Py` and `Bound<'py, T>`, which both use Python reference counting as their memory management. Almost all methods on these smart pointers use `&self` receivers, that is, a `&mut Py` or `&mut Bound<'py, T>` are almost never used in PyO3 code. -The situation is helped a little by the Global Interpreter Lock (GIL), which -ensures that only one thread can use the Python interpreter and its API at the -same time, while non-Python operations (system calls and extension code) can -unlock the GIL. (See [the section on parallelism](parallelism.md) for how to do -that in PyO3.) +The situation is helped by the Global Interpreter Lock (GIL), which ensures that only one thread can use the Python interpreter and its API at the same time, while non-Python operations (system calls and extension code) can unlock the GIL. (See [the section on parallelism](parallelism.md) for how to do that in PyO3.) In PyO3, holding the GIL is modeled by acquiring a token of the type `Python<'py>`, which serves three purposes: @@ -47,7 +80,7 @@ references is done at runtime using `PyCell`, a scheme very similar to To get hold of a `Python<'py>` token to prove the GIL is held, consult [PyO3's documentation][obtaining-py]. -## Object types +## The GIL Refs API ### [`PyAny`][PyAny]