Add DataTypes documentation

[jeffzi]

This PR is a follow-up to the dtypes refactor started in #490 and initially discussed in #369. It adds documentation for the DataType in the api reference section as well as a section about customizing data types.

I also added some niceties:

• re-organized api reference into sub-sections to keep the side menu tidy. Sphinx defaults to lexical order and it was a mess with
  all the data type additions.
• added xdoctest to CI to test docstrings
• added doctest builder to sphinx-build when running locally (replicate github CI).
• ignored python prompts when copying example from doc (strip >>> present in docstring examples).
• unpinned sphinx, working so far...

I also had to pin the furo theme. The latest release change the path to the assets reference in conf.py. I did not want to mess with that in this PR...

@cosmicBboy Let me know what you think about the documentation. I was trying to remain concise and not bother users with irrelevant details. Someone who wants to add a completely new third-party library will likely get in touch and we could then explain the finer details.

[jeffzi]

CI does not pick up the PR, will re-create

[jeffzi]

No idea why github checks are not added. They worked on previous PR #504, opening a new PR #537 did not solve the problem. The only modification to ci-tests.yml is an added test here.

[cosmicBboy]

@jeffzi thanks! looks like there are some conflicts, probably because I messed around with the docs/dtype modules a little: 6127454 (feel free to undo my changes)

edit: just rolled it back

[codecov]

Codecov Report

Merging #536 (f0a33c2) into dtypes (460663d) will decrease coverage by 0.46%.
The diff coverage is 89.47%.

@@            Coverage Diff             @@
##           dtypes     #536      +/-   ##
==========================================
- Coverage   97.91%   97.45%   -0.47%     
==========================================
  Files          24       24              
  Lines        3116     3101      -15     
==========================================
- Hits         3051     3022      -29     
- Misses         65       79      +14     

Impacted Files	Coverage Δ
pandera/checks.py	98.54% <ø> (ø)
pandera/engines/numpy_engine.py	87.50% <ø> (ø)
pandera/schemas.py	99.63% <ø> (ø)
pandera/engines/pandas_engine.py	95.00% <87.50%> (-1.11%)	⬇️
pandera/dtypes.py	90.35% <100.00%> (-6.11%)	⬇️
pandera/engines/engine.py	93.40% <100.00%> (-0.22%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 460663d...f0a33c2. Read the comment docs.

[jeffzi]

looks like there are some conflicts,

Interesting, I didn't know conflicts could prevent CI from running. Thanks for the help :)

Code coverage could be improved but it's probably better to save that for another PR. I think dtypes branch looks ready to be merged into dev 🤞

[cosmicBboy]

Code coverage could be improved but it's probably better to save that for another PR. I think dtypes branch looks ready to be merged into dev 🤞

@jeffzi very soon! actually dtypes should be merged onto the release/0.7.0 branch, since dev is for bugfix releases.

Gonna have to do a little dance to get this to all work smoothly, but basically there are a couple of things I wanted to do as part of 0.7.0 release. Now that pandera has a steady stream of regular users and new ones coming in every few weeks (I'd say pandera is no longer a personal project), I'd like to break as little as possible and put a deprecation policy in place.

It's gonna be a little more work, but I think it'll be worth it so that we set a good precedent for not pulling the rug from under unsuspecting users :)

So what does this mean? #539

1. Add back the Column(pandas_dtype=...) kwarg (to exist alongside dtype arg)
   • user cannot specify both pandas_dtype and dtype
   • if using pandas_dtype, DeprecationWarning is raised telling the user to use dtype
2. Add back PandasDtype enum that simply aliases the types that you introduced (based on issues raised at least a few people are using PandasDtype
   • DeprecationWarning is raised whenever this enum is used

On a related note, I'm wondering if maintaining bugfix patches to 0.6.* would make sense. It might be a pain, but I've already gotten a request to loosen the pandas requirements due to legacy codebases using pandas==0.23.4 but wanting to use pandera (someone has even asked if they can fork and publish a pandera-legacy package on pypi for legacy pandas.

[jeffzi]

I agree with putting back pandera_dtype and PandasDtype + deprecation warnings (targeted to 0.8.0 or 0.9.0?).

Personally, I think maintaining 0.6.* patches is overkill, even pandas doesn't do that for versions < 1.0. Especially since pandera is still maturing at a very fast pace. Legacy pandas indeed proved to be a pain to maintain. I don't really see the point of a pandera-legacy Vs locking the requirements to 0.6.0. The fork won't be able to keep up with new features imho.

re: 0.7.0. Let me know where you'd prefer me to help. I mean, maybe you'd want to handle deprecation warnings yourself?

[cosmicBboy]

re: 0.7.0. Let me know where you'd prefer me to help. I mean, maybe you'd want to handle deprecation warnings yourself?

Yeah I'll work on that, shouldn't be too much effort.

Personally, I think maintaining 0.6.* patches is overkill

Yeah, that's what I'm leaning towards right now... anyway it's open source, so whoever wants to make it work with legacy pandas can do it themselves :)