Implementation of __cause__ for ValidationError using ExceptionGroups

[zakstucke]

Change Summary

Out of date, contents has changed significantly from initial draft. See the comments for the path to the current version.

Draft wanting feedback

• Fully working Python 3.7 upwards
• Requires exceptiongroup dependency for ExceptionGroup backport pre 3.11
• Attaches causes of user made validation errors to the __cause__ of a ValidationError, including the respective loc for each cause chain
• See below for new example traceback output

Following on from my issue in pydantic main, if ValidationError(s) utilised __cause__ they would become immensely more powerful in non-user facing, debugging scenarios.

I've got a working but dirty implementation here, was trying for minimal lines.

Before I spend more time on it making it pretty, safe and optimised, I'm looking for thoughts on whether this is something Pydantic is willing to support.

This example implementation works with Python 3.7 upwards for compatibility, using the exceptiongroup backport to that effect. Works great in all python versions. (obviously add_note() is only used 3.11 upwards, but the chaining and groups are available on all supported versions).

For 3.11 upwards this could be improved to:

• Use add_note() on the main exception instead of the extra UserWarningException to specify the loc
• Not require the backport exception group lib

If you're not liking the bloat for the average new user, maybe this could be enabled with a global flag.

This also seems to break 0 tests as nothing as of yet relies on __cause__.

Edit: i've noticed something off with the second error group, aka the singular exception for bar, not sure why its not but should show the error location same as the source error of foo chain, would fix.

Implements an ExceptionGroup as the cause of a ValidationError, to display user-defined callback errors that caused the ValidationError aesthetically. Really useful for debugging as a user.

Pre 3.11 it uses the exceptiongroup backport, 3.11+ uses the rust-native BaseExceptionGroup. In both cases, the cause is created lazily on read, so no performance cost. For everything other than PyPy, the lazy cause is only built once.

The below example output is up to date.

Related issue number

pydantic/pydantic#6498 (comment)

Checklist

• Unit tests for the changes exist
• Documentation reflects the changes where applicable
• Pydantic tests pass with this pydantic-core (except for expected changes)
• My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

from pydantic import BaseModel, field_validator

def check(v: int):
    if v < 0:
        raise ValueError("INNER ERR")
    return v

class Foo(BaseModel):
    foo: int
    bar: int

    @field_validator("foo")
    def check_foo(cls, v):
        if v < 0:
            try:
                check(v)
            except ValueError as e:
                new_err = ValueError("OUTER ERR")
                new_err.add_note("SOME EXTRA INFORMATION") # NOTE: 3.11+ only
                raise new_err from e
        return v
    
    @field_validator("bar")
    def check_bar(cls, v):
        if v < 0:
            check(v)
        return v    
    
Foo(foo=-1, bar=-1)

3.11+

  | ExceptionGroup: Pydantic User Code Exceptions (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 51, in check_foo
    |     check(v)
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 40, in check
    |     raise ValueError("INNER ERR")
    | ValueError: INNER ERR
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 55, in check_foo
    |     raise new_err from e
    | ValueError: OUTER ERR
    | SOME EXTRA INFORMATION
    | 
    | Pydantic: cause of loc: foo
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 61, in check_bar
    |     check(v)
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 40, in check
    |     raise ValueError("INNER ERR")
    | ValueError: INNER ERR
    | 
    | Pydantic: cause of loc: bar
    +------------------------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "xxxxxxxxxxx/pydantic-core/test.py", line 64, in <module>
    Foo(foo=-1, bar=-1)
  File "xxxxxxxxxxx/pydantic-core/venv/lib/python3.11/site-packages/pydantic/main.py", line 150, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for Foo
foo
  Value error, OUTER ERR [type=value_error, input_value=-1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.0.3/v/value_error
bar
  Value error, INNER ERR [type=value_error, input_value=-1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.0.3/v/value_error

Pre 3.11 & PyPy:

  | exceptiongroup.ExceptionGroup: Pydantic User Code Exceptions (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 16, in check_foo
    |     check(v)
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 5, in check
    |     raise ValueError("INNER ERR")
    | ValueError: INNER ERR
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 20, in check_foo
    |     raise new_err from e
    | ValueError: OUTER ERR
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | UserWarning: Pydantic: cause of loc: foo
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 26, in check_bar
    |     check(v)
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 5, in check
    |     raise ValueError("INNER ERR")
    | ValueError: INNER ERR
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | UserWarning: Pydantic: cause of loc: bar
    +------------------------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "xxxxxxxxxxx/pydantic-core/test.py", line 29, in <module>
    Foo(foo=-1, bar=-1)
  File "xxxxxxxxxxx/pydantic-core/venv10/lib/python3.10/site-packages/pydantic/main.py", line 150, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for Foo
foo
  Value error, OUTER ERR [type=value_error, input_value=-1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.0.3/v/value_error
bar
  Value error, INNER ERR [type=value_error, input_value=-1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.0.3/v/value_error

Selected Reviewer: @dmontagu

[codecov]

Codecov Report

Merging #780 (f09d56b) into main (f6b14cc) will decrease coverage by 0.03%.
The diff coverage is 85.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #780      +/-   ##
==========================================
- Coverage   93.80%   93.78%   -0.03%     
==========================================
  Files         105      105              
  Lines       15468    15539      +71     
  Branches       25       25              
==========================================
+ Hits        14510    14573      +63     
- Misses        952      960       +8     
  Partials        6        6              

Files Changed	Coverage Δ
src/errors/validation_exception.rs	91.64% <75.00%> (-2.04%)	⬇️
src/validators/generator.rs	90.94% <95.45%> (+0.27%)	⬆️
python/pydantic_core/core_schema.py	96.77% <100.00%> (+<0.01%)	⬆️
src/validators/function.rs	93.80% <100.00%> (+1.23%)	⬆️
src/validators/mod.rs	95.12% <100.00%> (+0.04%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6b14cc...f09d56b. Read the comment docs.

[codspeed-hq]

CodSpeed Performance Report

Merging #780 will degrade performances by 21.18%

_{Comparing zakstucke:main (f09d56b) with main (f6b14cc)}

Summary

🔥 4 improvements
❌ 1 regressions
✅ 133 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	zakstucke:main	Change
🔥	test_core_string_lax_wrong	63.4 µs	37 µs	+71.49%
🔥	test_core_string_strict_wrong	49.2 µs	36.7 µs	+33.98%
🔥	test_strict_union_error_core	59.6 µs	47.7 µs	+25%
🔥	test_raise_error_custom	69.5 µs	58.6 µs	+18.55%
❌	test_dont_raise_error	30.9 µs	39.2 µs	-21.18%

[adriangb]

I think this is an interesting idea!

I haven't looked in detail at how it works, but I will say that what really should happen in the end is that ValidationError should be itself an ExceptionGroup. It's just not trivial to make that happen due to a combination of the state of the current codebase, limitations of multiple inheritance with PyO3/lack of an ExceptionGroup C API and performance of calling into Python (like you are doing in this PR).

I will look in more detail but just because of the calling into Python that's being done here I suspect there is going to be a huge performance hit to benchmarks (and it looks like CodSpeed is already flagging test_raise_error_value_error as problematic) that means we can't move forward with this as-is :(

[zakstucke]

@adriangb if the performance of the backport is the problem, this could be supported 3.11+ only.

I've updated the PR to only include the logic when python is 3.11+ (compile time excluded code on older versions), and uses the rust PyBaseExceptionGroup, so no python callbacks.

My focus is 3.11+, I personally wouldn't mind if this was a limitation.

However the arguments for using the backport on older versions:

• This is a crazy fast function, this addition also doesn't build the error group (aka close to zero cost) when the user isn't including python callback validators. This is why you see it affect so few benchmarks, so the cost of going from 0.0000562 to 0.000145 seconds (with the pre 3.11 backport version), given the user is already guaranteed to be running heavy callbacks to python for it to build (i think?!), must be pretty negligible.
• It can be immensely useful info, even if older versions have to manually enable with a global flag, I think a lot of people would want and enable it today if they could.

Out of interest, why would you eventually want the ValidationError itself to be an ExceptionGroup? To my knowledge you're only ever getting one validation error per rust call (adding to the negligability of a small performance hit), which can have lots of children excs. So in my understanding having a singular validation exception, with an ExceptionGroup (that is built to house children excs, not a parent and children together) in the __cause__ like this, is already as good as it gets with groups?

[adriangb]

Out of interest, why would you eventually want the ValidationError itself to be an ExceptionGroup? To my knowledge you're only ever getting one validation error per rust call (adding to the negligability of a small performance hit), which can have lots of children excs. So in my understanding having a singular validation exception, with an ExceptionGroup (that is built to house children excs, not a parent and children together) in the cause like this, is already as good as it gets with groups?

A ValidationError is an error that represents a group of errors. That's what an ExceptionGroup is! There's also a tree-like relationship in errors that is currently present in the loc aspect of ValidationError but not in the error structure itself. That is, if you have a union and the union fails validation the cause of that failure is actually all of the failures that happened with the variants. So making ValidationError an ExceptionGroup would allow us to express that much better, in addition to enabling the 3.11+ syntax for filtering and traversing this tree when using except.

You can also do weird things (which I think this PR should consider btw) like having a python function validator which does:

try:
    handler(x)
catch ValidationError as e:
    raise ValueError(...) from e

So now you have a ValidationError -> ValueError -> ValidationError relationship.

[zakstucke]

Makes sense you're right. I guess it then comes down to the issues you'd mentioned with how infeasible it would be to convert ValidationError to an ExceptionGroup in the near future.

I've just tested out your nested ValidationError edge case, it's actually an example where this could be super helpful for debugging:

import typing as tp
from pydantic import BaseModel, field_validator, ValidationError, TypeAdapter, AfterValidator


def check(v: int):
    if v < 0:
        raise ValueError("INNER ERR")
    return v

sub_validator = TypeAdapter(tp.Annotated[int, AfterValidator(check)])

class Baz(BaseModel):
    bar: int

    @field_validator("bar")
    def check_bar(cls, v):
        try:
            return sub_validator.validate_python(v)
        except ValidationError as e:
            raise ValueError("SUB VALIDATION ERROR!") from e    

class Foo(BaseModel):
    foo: int
    baz: Baz    

    @field_validator("foo")
    def check_foo(cls, v):
        if v < 0:
            try:
                check(v)
            except ValueError as e:
                new_err = ValueError("OUTER ERR")
                new_err.add_note("SOME EXTRA INFORMATION")
                raise new_err from e
        return v
    
Foo(foo=-1, baz={"bar": -1})

Outputs:

  | ExceptionGroup: Pydantic User Code Exceptions (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 65, in check_foo
    |     check(v)
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 42, in check
    |     raise ValueError("INNER ERR")
    | ValueError: INNER ERR
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 69, in check_foo
    |     raise new_err from e
    | ValueError: OUTER ERR
    | SOME EXTRA INFORMATION
    | 
    | Pydantic: cause of loc: foo
    +---------------- 2 ----------------
    | ExceptionGroup: Pydantic User Code Exceptions (1 sub-exception)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "xxxxxxxxxxx/pydantic-core/test.py", line 42, in check
      |     raise ValueError("INNER ERR")
      | ValueError: INNER ERR
      | 
      | Pydantic: cause of loc: root
      +------------------------------------
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 53, in check_bar
    |     return sub_validator.validate_python(v)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "xxxxxxxxxxx/pydantic-core/venv/lib/python3.11/site-packages/pydantic/type_adapter.py", line 206, in validate_python
    |     return self.validator.validate_python(__object, strict=strict, from_attributes=from_attributes, context=context)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | pydantic_core._pydantic_core.ValidationError: 1 validation error for function-after[check(), int]
  Value error, INNER ERR [type=value_error, input_value=-1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.0.3/v/value_error
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File "xxxxxxxxxxx/pydantic-core/test.py", line 55, in check_bar
    |     raise ValueError("SUB VALIDATION ERROR!") from e    
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | ValueError: SUB VALIDATION ERROR!
    | 
    | Pydantic: cause of loc: baz.bar
    +------------------------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "xxxxxxxxxxx/pydantic-core/test.py", line 72, in <module>
    Foo(foo=-1, baz={"bar": -1})
  File "xxxxxxxxxxx/pydantic-core/venv/lib/python3.11/site-packages/pydantic/main.py", line 150, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for Foo
foo
  Value error, OUTER ERR [type=value_error, input_value=-1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.0.3/v/value_error
baz.bar
  Value error, SUB VALIDATION ERROR! [type=value_error, input_value=-1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.0.3/v/value_error

Which works reasonably well out the box for the weird double ValidationError case, just creates a tree of groups.

[zakstucke]

I'm close to happy with this now. Fixed the random exception missing its traceback in the result and switched to a note rather than an extra warning exception.

The changes made to convert_err() & the py_err_string! macro are a bugfix to stop throwing away the top level traceback from user errors. Before this, an error raised in a validator has no traceback at all.
It seems like it might be a Pyo3 quirk, but PyErr.into_py(py) seems to throw away the traceback.

@davidhewitt do you have any idea why this is and if there's a better way to solve it than I've done? PyErr.state is private and there's no interface to re-attach a traceback / prevent losing it on conversion to PyObject that I can see.

[davidhewitt]

It seems like it might be a Pyo3 quirk, but PyErr.into_py(py) seems to throw away the traceback.

Are you willing to reduce this to a minimal repro which can be reported as an issue to PyO3, and we can discuss there if it's a bug? Before 3.12 the Python exception state is stored as split type / value / traceback and so I'd be willing to believe the PyErr::into_py() implementation needs adjusting.

[zakstucke]

@davidhewitt done :), if there's a cleaner way of getting around it on the current pyo3 version for the use here, please let me know.

[davidhewitt]

Thanks - I think your implementation here is correct (and likely what we would land upstream in PyO3).

[davidhewitt]

@adriangb and I just had a catch up about this PR. Overall we like this a lot and we're happy to offer users a way to get at inner exceptions.

Some thoughts which we had:

• Forwards compatibility with changing ValidationError to itself be an ExceptionGroup. This seems ok, we could make ValidationError.__cause__ just be a reference to itself (a bit weird but would avoid breaking users who access __cause__ as implemented in this PR).
• Performance. We wondered whether we should add #[getter] fn __cause__ to lazily create the ExceptionGroup rather than always create __cause__ immediately. (We could cache it to create it just once.) Not a deal breaker but potentially nice to have.
• Feature set. If we create __cause__ lazily we could also support <3.11 using the exceptiongroup backport. __cause__ could throw on access if the backport isn't installed on those versions. What do you think about this?
• Structure of __cause__. This we'd like to make sure we're happy with before we merge this PR. Is it API breaking if we changed the contents of these exceptions later?
  • If we deem it is API breaking to change the inner exceptions later, we need to be sure we're happy with them in various combinations of schemas / validators / etc., because it will impact our ability to iterate if users become highly sensitive to exception structure.
  • If it's not API breaking to change the inner exceptions later, then we're obviously fine to merge the structure as-is.

[zakstucke]

@davidhewitt great to hear!

Forwards compatibility ... we could make ValidationError.__cause__ just be a reference to itself

No qualms against this for programmatic access, would definitely work.

Performance ... lazy creation

TLDR it doesn't work.

I'd actually already looked into this when trying to ease @adriangb's performance concerns with the backport.
It would work for a user catching the exception and programatically acessing the __cause__, but for the main benefit of this, traceback crashing the process and having full context, that flow happens directly in CPython, __cause__ isn't used in that case. See print_exception_cause_and_context() here in cpython source which is what I think happens; the order of logic in there means attaching the lazy creation to even something like repr doesn't seem to work.

If you've got any ideas on this let me know, I gave up.

Structure of __cause__ / API break

Valid concern, given how much looks like needs improving in ValidationError I don't think it would be a good idea to guarantee a specific structure in __cause__. It's current intended use should be for visual context in stack traces, not programatic access. If a user starts searching through this error group for errors they're asking for trouble!

An option for making the above enforceable/covered pd side, we could add another method to the ValidationError api, user_errs() -> list[Exception], which guarantees to produce a flat list of all user validator-caused errors inside the ValidationError. With the current implementation this would be just return __cause__.exceptions if __cause__ else [], but would alleviate the worry of modifying __cause__ down the line as everyone can be told to use that.

So if lazy doesn't work, unless you guys think of something new it doesn't, options for backport are:

• No support, sad
• Full support, accept the 3x slowdown in ValidationError.into_py (on my machine 1us to 3us), but remember above comments about how imo this is neglible as the user would already been running python callbacks.
• Put behind feature flag, e.g. pydantic_core.enable_user_err_context = True

[zakstucke]

@davidhewitt I have actually found a way to get around the lazy problem but it's a bit weird:

import sys

original_excepthook = sys.excepthook

# Todo: handle already seen / infinite loops
def traverse_causes_on_crash(start_exc: BaseException):
    excs: list[BaseException] = [start_exc]
    if isinstance(start_exc, BaseExceptionGroup):
        excs.extend(start_exc.exceptions)

    for exc in excs:
        cause = getattr(exc, '__cause__', None)
        if cause is not None:
            traverse_causes_on_crash(cause)

def replacement_excepthook(exc_type, exc_value, exc_traceback):
    # Traverse the causes:
    traverse_causes_on_crash(exc_value)

    # Call the original excepthook:
    original_excepthook(exc_type, exc_value, exc_traceback)

if original_excepthook is not replacement_excepthook:
    sys.excepthook = replacement_excepthook

This allows the __cause__ hydration to happen just before C checks the cause.
Is this too weird or something to keep looking into?

But this would make the whole thing 0 cost, snippet could either just be run on pydantic_core import or enabled with pydantic_core.enable_validation_cause_tracebacks().

traceback.print_exc() etc already seems to work out the box (with lazy __cause__) when other libs/apps are catching errs and logging them themselves, so adding this would cover everything I think.

[zakstucke]

@adriangb @davidhewitt

Other than PyPy, this is now working on all versions with no performance cost, the test-pydantic-integration failure I believe is unrelated to this PR.

I'm more or less finished with it, so its ready for you to let me know what you want changing.

If you see what might be causing the segfault in PyPy (in the __cause__ getter) let me know!

exceptiongroup dep:
If you want this to be optional, it would need to fail silently, I don't see a good way to raise an error on __cause__ access, because this would be accessed whenever an exception is formatted with traceback and every time a validation error crashed the program (aka if someone on 3.10 got a validation error that stopped the process, they'd be told they need to install the backport)

Imo it should just be a dep as I've got it now, not just silently omit the group unless the backports installed.
I'm not sure how to make a python version specific dep in the pyproject.toml, but that could be done too.

[zakstucke]

PyPy fixed, it uses __cause__ itself in ffi::PyException_GetCause & ffi::PyException_SetCause so was causing infinite recursion, disabled ffi setting for pypy.

Converting from draft!

Please review

[zakstucke]

@davidhewitt the note (3.11+) / UserWarning (<3.11) is used to attach the source of the error and is all about improving the traceback debugging experience.

If you check all the way to the top of this PR i give the examples for both versions. If we don't include the UserWarning <3.11, these users have a worse debugging experience as they can't find exactly where in the model being analysed the error occurred, unlike all other ValidationError's.
Given we've already agreed stable programmatic access isn't currently being promised, I think it's on the stronger side of the tradeoff personally.

3.11+

Pre:

[davidhewitt]

Thanks, yes that makes sense from a UX perspective and fits with our statement that this won't be stable.

[samuelcolvin]

otherwise LGTM.

Assuming @davidhewitt is happy with the rust code.

[samuelcolvin]

better to have two tests and add add a pytest.mark.skipif to both, so only one runs.

[zakstucke]

I've separated out the contents tests as you suggested here :)

[samuelcolvin]

this seems like a separate test.

[zakstucke]

Done, new traceback preservation check test.

[samuelcolvin]

better not to use functions like this as it becomes hard to understand upon failure, especially as it only removes one duplication.

[zakstucke]

No prob, duplicated the code!

[samuelcolvin]

please use parameterize rather than a loop - loops in tests are impossible to understand/debug.

[zakstucke]

Done!

[zakstucke]

@samuelcolvin @davidhewitt sorry for the delay, refactored the tests as requested :)

[davidhewitt]

Thanks again, this LGTM and it's nice that it delivers some neat performance improvements on the error pathway too! Sorry that this has had such long cycle times on my end.

[zakstucke]

No problem and thanks! 🎉