make SchemaError and SchemaErrors picklable

[matthiashuschle]

description

• Solves SchemaError breaks pickle #713
• The issue extends to ParserError.
• As BaseException implements __reduce__, which expects an args attribute in the object, it needs to be overridden.
• To abstract the logic for all three affected classes, ReducedPickleExceptionBase was created as base class for exceptions that need to convert attributes before pickling, so actual changes in the final exception classes are minimal.
  • Its __reduce__ composes the state from args, and __dict__, and converts all __dict__ elements listed in TO_STRING_KEYS to strings. The return signature of __reduce__ uses the three-tuple, that uses __new__ for object creation and then updates it via __setstate__.
  • Its __setstate__ creates a warning stating the non-conservation of state, listing converted keys.
• Most non-primitive members are converted to string.
• SchemaErrors.schema_errors is not primitive, but not transformed in the process. I did not encounter an unpicklable case and it is highly unlikely to become problematic in size.

rejected ideas - up for discussion

• using sample or head instead of string conversion to reduce memory footprint of data members. The idea is to allow some degree of inspection. However, all ways to reduce data size are arbitrary to some degree, and string representation breaks any attempt to accidentally use incomplete data in further processing.

[codecov]

Codecov Report

Merging #722 (7ee5d82) into master (eb130b4) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #722   +/-   ##
=======================================
  Coverage   98.40%   98.40%           
=======================================
  Files          39       39           
  Lines        3757     3773   +16     
=======================================
+ Hits         3697     3713   +16     
  Misses         60       60           

Impacted Files	Coverage Δ
pandera/errors.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb130b4...7ee5d82. Read the comment docs.

[cosmicBboy]

Thanks @matthiashuschle !

SchemaError.failure_cases seems to be limited to 10 entries. Is this documented somewhere? I'd like to add a hint to TestSchemaError._validate_error

It's not great, but the only place this is documented is the Check init signature: https://pandera.readthedocs.io/en/stable/reference/generated/pandera.checks.Check.html#pandera.checks.Check

Added a warning on unpickling (to be discussed).

This makes sense to do. Maybe state explicitly what state isn't preserved? (e.g. "Pickling ParserError does not preserve the following state: failure_cases, data, etc.")

[cosmicBboy]

also, be sure to set up pre-commit to catch mypy/linting/unit test errors

[matthiashuschle]

Thanks @cosmicBboy !
The latest commit does some streamlining and I consider this now in shape for reviewing. The Linter errors also appear in the current dev branch. I don't think they are related to my changes.

[cosmicBboy]

okay, gonna merge this into master, lint tests aren't related to PR

[cosmicBboy]

thanks @matthiashuschle, and congrats on your first contribution to pandera! 🎉🚀