deprecate implicit default for trailing optional arguments #4078

Icxolu · 2024-04-13T22:12:27Z

This deprecates the implicit default for trailing optional (Option<...>) arguments.

Motivation

Loosely summarized from #2935,

The current behavior is unexpected from both Rust and Python.
Rust does not have a concept of default values for arguments, so it's surprising that pyo3 just adds one without explicit request

Python does have default values for argument, but they have to be set explicitly.

def fun_1(a, b):
    ...
def fun_2(a=None, b=None):
    ...

fun_1() # <- This is errors instead of using `None` as default
fun_2() # <- This is fine

This makes porting Python functions to Rust more surprising that it needs to be, especially if an Option<...> argument is not in trailing position.

Expecting functions without #[pyo3(signature=(...)] to have a trivial signature (without defaults or other modifiers) is more intuitive. To even know about the current behavior, users have to carefully read the docs.

Migration

This PR adds a deprecation warning to Option<...> arguments on functions that do not currently have a
#[pyo3(signature=(...)] attribute and a big warning in the guide. This should push users to explicitly spell out their desired behavior during the migration period. After that we should probably make this a hard error for one release cycle, before we allow Option<...> arguments without #[pyo3(signature=(...)] again, (but without special handling).

Xref: #2935, #3735

~~Stacked on top of #4033, so I leave this as a draft for now.~~

codspeed-hq · 2024-04-14T16:10:11Z

CodSpeed Performance Report

Merging #4078 will not alter performance

_{Comparing Icxolu:fix-2935 (6cf6532) with main (104328c)}

🎉 Hooray! `codspeed-rust` just leveled up to 2.6.0!

A heads-up, this is a breaking change and it might affect your current performance baseline a bit. But here's the exciting part - it's packed with new, cool features and promises improved result stability 🥳!
Curious about what's new? Visit our releases page to delve into all the awesome details about this new version.

Summary

✅ 69 untouched benchmarks

davidhewitt · 2024-04-14T20:33:16Z

This gets a big 👍 from me! I do occasionally find the trailing optional arguments a convenient default, but just as often I've had to deliberately disable them. Having the special case also increases cognitive overhead of working out the Python signature.

Anyone got strong feelings that we shouldn't be doing this?

davidhewitt

Given there doesn't seem to be much resistance, shall we move ahead with this for 0.22? I think the simplification of this edge case is a worthwhile improvement!

Just one idea regarding the deprecation message, which might be interesting to play with...

guide/src/function/signature.md

davidhewitt · 2024-05-09T07:50:11Z

tests/ui/deprecations.stderr

+error: use of deprecated function `pyo3::deprecations::deprecate_implicit_option`: Implicit default for trailing optional arguments is phased out. Add an explicit `#[pyo3(signature = (...))]` attribute a to silence this warning. In a future pyo3 version `Option<..>` arguments will be treated the same as any other argument.
+   --> tests/ui/deprecations.rs:119:39
+    |
+119 | fn pyfunction_option_2(_i: u32, _any: Option<i32>) {}


It would be awesome if the macro could emit the right signature here instead of (...).

I wonder, what happens if we made the macro deprecation expand to something like:

#[deprecated(note = "this function has implicit defaults for the trailing `Option<T>` argument `_any`. These implicit defaults are being phased out. Add `#[pyo3(signature = (_i, _any = None))]` to this function")] const PYFUNCTION_OPTION_2_SIGNATURE: () = (); // immediately use the deprecated symbol to trigger the deprecation warning const _: () = PYFUNCTION_OPTION_2_SIGNATURE;

I think if we attached that to the span of the whole function, rather than just an individual argument, it'd be super clear to users how to upgrade.

That would be really nice! I'll give it a try

Icxolu · 2024-05-09T10:49:08Z

I don't really have a clue why #![allow(deprecated)] makes this trigger as dead_code. Should I just add #[allow(dead_code)] to the generated const? Or is there a better way?

davidhewitt · 2024-05-09T10:52:37Z

tests/ui/deprecations.stderr

+error: use of deprecated constant `MyClass::__pymethod_set_set_option__::SIGNATURE`: This function has implicit defaults for the trailing `Option<T>` arguments. These implicit defaults are being phased out. Add `#[pyo3(signature = (_value=None)]` to this function to silence this warning and keep the current behavior
+  --> tests/ui/deprecations.rs:43:8
+   |
+43 |     fn set_option(&self, _value: Option<i32>) {}
+   |        ^^^^^^^^^^
+
+error: use of deprecated constant `__pyfunction_pyfunction_option_2::SIGNATURE`: This function has implicit defaults for the trailing `Option<T>` arguments. These implicit defaults are being phased out. Add `#[pyo3(signature = (_i, _any=None)]` to this function to silence this warning and keep the current behavior
+   --> tests/ui/deprecations.rs:132:4
+    |
+132 | fn pyfunction_option_2(_i: u32, _any: Option<i32>) {}
+    |    ^^^^^^^^^^^^^^^^^^^
+
+error: use of deprecated constant `__pyfunction_pyfunction_option_3::SIGNATURE`: This function has implicit defaults for the trailing `Option<T>` arguments. These implicit defaults are being phased out. Add `#[pyo3(signature = (_i, _any=None, _foo=None)]` to this function to silence this warning and keep the current behavior
+   --> tests/ui/deprecations.rs:135:4
+    |
+135 | fn pyfunction_option_3(_i: u32, _any: Option<i32>, _foo: Option<String>) {}
+    |    ^^^^^^^^^^^^^^^^^^^
+
+error: use of deprecated constant `__pyfunction_pyfunction_option_4::SIGNATURE`: This function has implicit defaults for the trailing `Option<T>` arguments. These implicit defaults are being phased out. Add `#[pyo3(signature = (_i, _any=None, _foo=None)]` to this function to silence this warning and keep the current behavior
+   --> tests/ui/deprecations.rs:138:4
+    |
+138 | fn pyfunction_option_4(
+    |    ^^^^^^^^^^^^^^^^^^^


I think this is great 👍 💯

pyo3-macros-backend/src/deprecations.rs

davidhewitt · 2024-05-09T10:56:12Z

Think it might be unrelated to the #![allow] and just the compiler finding the const isn't ending up in the binary.

https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=88c10b1e7f4ae498da94f3eae712178d

So I guess just add #[allow(dead_code)]? It is still having the effect of producing an awesome message 🚀

pyo3-macros-backend/src/deprecations.rs

davidhewitt

Just a final thought here - we should document this in the migration guide, I think.

Icxolu · 2024-05-09T21:55:56Z

Good idea!

davidhewitt

Great!

guide/src/migration.md

Implicit defaults for trailing optional arguments have been deprecated in pyo3 v0.22.0 PyO3/pyo3#4078

* update dependencies * update get_logical_plan signature * remove row_number() function row_number was converted to a UDF in datafusion v42 apache/datafusion#12030 This specific functionality needs to be added back in. * remove unneeded dependency * fix pyo3 warnings Implicit defaults for trailing optional arguments have been deprecated in pyo3 v0.22.0 PyO3/pyo3#4078 * update object_store dependency * change PyExpr -> PySortExpr * comment out key.extract::<&PyTuple>() condition statement * change more instances of PyExpr > PySortExpr * update function signatures to use _bound versions * remove clone * Working through some of the sort requirement changes * remove unused import * expr.display_name is deprecated, used format!() + schema_name() instead * expr.canonical_name() is deprecated, use format!() expr instead * remove comment * fix tuple extraction in dataframe.__getitem__() * remove unneeded import * Add docstring comments to SortExpr python class * change extract() to downcast() Co-authored-by: Michael J Ward <[email protected]> * deprecate Expr::display_name Ref: apache/datafusion#11797 * fix lint errors * update datafusion commit hash * fix type in cargo file for arrow features * upgrade to datafusion 42 * cleanup --------- Co-authored-by: Tim Saucer <[email protected]> Co-authored-by: Michael J Ward <[email protected]> Co-authored-by: Michael-J-Ward <[email protected]>

Icxolu force-pushed the fix-2935 branch from 827312d to 75b8bc2 Compare April 14, 2024 15:54

Icxolu marked this pull request as ready for review April 14, 2024 15:58

davidhewitt reviewed May 9, 2024

View reviewed changes

davidhewitt mentioned this pull request May 9, 2024

allow constructor customization of complex enum variants #4158

Merged

Icxolu force-pushed the fix-2935 branch 2 times, most recently from a927c7e to bc2f105 Compare May 9, 2024 10:10

davidhewitt reviewed May 9, 2024

View reviewed changes

Icxolu force-pushed the fix-2935 branch from bc2f105 to 0cfb93c Compare May 9, 2024 11:46

davidhewitt approved these changes May 9, 2024

View reviewed changes

pyo3-macros-backend/src/deprecations.rs Show resolved Hide resolved

davidhewitt requested changes May 9, 2024

View reviewed changes

davidhewitt approved these changes May 10, 2024

View reviewed changes

davidhewitt reviewed May 10, 2024

View reviewed changes

guide/src/migration.md Show resolved Hide resolved

Icxolu force-pushed the fix-2935 branch from eac6a44 to 8a21afa Compare May 10, 2024 10:13

Icxolu added 4 commits May 10, 2024 12:17

deprecate "trailing optional arguments" implicit default behaviour

34b361a

add newsfragment

2fa9a2a

generate individual deprecation messages per function

adc2b5a

add migration guide entry

6cf6532

Icxolu force-pushed the fix-2935 branch from 8a21afa to 6cf6532 Compare May 10, 2024 10:17

Icxolu enabled auto-merge May 10, 2024 10:18

Icxolu added this pull request to the merge queue May 10, 2024

Merged via the queue into PyO3:main with commit aef0a05 May 10, 2024
43 checks passed

Icxolu deleted the fix-2935 branch May 10, 2024 11:06

Michael-J-Ward mentioned this pull request Sep 10, 2024

Upgrade datafusion apache/datafusion-python#867

Merged

emgeee added a commit to probably-nothing-labs/datafusion-python that referenced this pull request Sep 10, 2024

fix pyo3 warnings

d60168e

Implicit defaults for trailing optional arguments have been deprecated in pyo3 v0.22.0 PyO3/pyo3#4078

emgeee added a commit to probably-nothing-labs/datafusion-python that referenced this pull request Sep 10, 2024

fix pyo3 warnings

4b45a4b

Implicit defaults for trailing optional arguments have been deprecated in pyo3 v0.22.0 PyO3/pyo3#4078

davidhewitt mentioned this pull request Oct 11, 2024

Relax "REQUIRED_ARGUMENT_AFTER_OPTION" deprecation #3735

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deprecate implicit default for trailing optional arguments #4078

deprecate implicit default for trailing optional arguments #4078

Icxolu commented Apr 13, 2024 •

edited

Loading

codspeed-hq bot commented Apr 14, 2024 •

edited

Loading

davidhewitt commented Apr 14, 2024

davidhewitt left a comment

davidhewitt May 9, 2024

Icxolu May 9, 2024

Icxolu commented May 9, 2024

davidhewitt May 9, 2024

davidhewitt commented May 9, 2024

davidhewitt left a comment

Icxolu commented May 9, 2024

davidhewitt left a comment

deprecate implicit default for trailing optional arguments #4078

deprecate implicit default for trailing optional arguments #4078

Conversation

Icxolu commented Apr 13, 2024 • edited Loading

Motivation

Migration

codspeed-hq bot commented Apr 14, 2024 • edited Loading

CodSpeed Performance Report

Merging #4078 will not alter performance

🎉 Hooray! codspeed-rust just leveled up to 2.6.0!

Summary

davidhewitt commented Apr 14, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt May 9, 2024

Choose a reason for hiding this comment

Icxolu May 9, 2024

Choose a reason for hiding this comment

Icxolu commented May 9, 2024

davidhewitt May 9, 2024

Choose a reason for hiding this comment

davidhewitt commented May 9, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

Icxolu commented May 9, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

Icxolu commented Apr 13, 2024 •

edited

Loading

codspeed-hq bot commented Apr 14, 2024 •

edited

Loading

🎉 Hooray! `codspeed-rust` just leveled up to 2.6.0!