decouple pandera and pandas dtypes #559
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…taType hierarchy (#504) * fix pandas_engine.Interval * fix Timedelta64 registration with pandas_engine.Engine * add DataType helpers * add DataType.continuous attribute * add dtypes.is_numeric * refactor schema_statistics based on DataType hierarchy * refactor schema_inference based on DataType hierarchy * fix numpy_engine.Timedelta64.type * add is_subdtype helper * add Engine.get_registered_dtypes * fix Engine error when registering a base DataType * fix pandas_engine DateTime string alias * clean up test_dtypes * fix test_extensions * refactor strategies based on DataType hierarchy * refactor io based on DataType hierarchy * replace dtypes module by new DataType hierarchy * fix black * delete dtypes_.py * drop legacy pandas and python 3.6 from CI * fix mypy errors * fix ci-docs * fix conda dependencies * fix lint, update noxfile * simplify nox tests, fix test_io * update ci build * update nox * pin nox, handle windows data types * fix windows platform * fix pandas_engine on windows platform * fix test_dtypes on windows platform * force pip on docs CI * test out windows dtype stuff * more messing around with windows * more debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * revert ci * increase cache * testing Co-authored-by: cosmicBboy <[email protected]>
* delete print statements * pin furo * fix generated docs not removed by nox * re-organize API section * replace aliased pandas_engine data types with their aliases * drop warning when calling Engine.register_dtype without arguments * add data types to api reference doc * add document for DataType refactor * unpin sphinx and drop sphinx_rtd_theme * add xdoctest * ignore prompt when copying example from doc * add doctest builder when running sphinx-build locally * fix dtypes doc examples * fix pandas_engine.DataType.check * fix pylint * remove whitespaces in dtypes doc * Update docs/source/dtypes.rst * Update dtypes.rst * update docs structure * update nox file * force pip on doctests * update test_schemas * fix docs session not overriding html with doctest output Co-authored-by: Niels Bantilan <[email protected]>
* remove auto-generated docs * add deprecation warnings, support pandas>=1.3.0 * add deprecation warnings for PandasDtype enum * fix sphinx * fix windows * fix windows
* add support for pyarrow backed string data type * fix regression for pandas < 1.3.0 * add verbosity to test run * loosen strategies unit tests deadline, exclude windows ci * loosen test_strategies.py tests * use "dev" hypothesis profile for python 3.7 * add pandas==1.2.5 test * fix ci * ci typo * don't install environment.yml on unit tests * install nox in ci * remove environment.yml * update environment in ci Co-authored-by: cosmicBboy <[email protected]>
Codecov Report
@@ Coverage Diff @@
## release/0.7.0 #559 +/- ##
=================================================
- Coverage 99.47% 97.58% -1.89%
=================================================
Files 21 25 +4
Lines 2661 3194 +533
=================================================
+ Hits 2647 3117 +470
- Misses 14 77 +63
Continue to review full report at Codecov.
|
|
Exciting news ! For 0.7.1 I'll take a look at opened issues that can be fixed using the new internals.
My pleasure 🎉 |
cosmicBboy
added a commit
that referenced
this pull request
Jul 22, 2021
* refactor PandasDtype into class hierarchy supported by engines * refactor DataFrameSchema based on DataType hierarchy * refactor SchemaModel based on DataType hierarchy * revert fix coerce=True and dtype=None should be a noop * apply code style * fix running tests/core with nox * consolidate dtype names * consolidate engine internal naming * disable inherited __init__ with immutable(init=False) * delete duplicated immutable * disambiguate dtype variables * add warning on base pandas_engine, numpy_engine.DataType init * fix pylint, mypy errors * fix DataFrameSchema.dtypes return type * enable CI on dtypes branch * Refactor inference, schema_statistics, strategies and io using the DataType hierarchy (#504) * fix pandas_engine.Interval * fix Timedelta64 registration with pandas_engine.Engine * add DataType helpers * add DataType.continuous attribute * add dtypes.is_numeric * refactor schema_statistics based on DataType hierarchy * refactor schema_inference based on DataType hierarchy * fix numpy_engine.Timedelta64.type * add is_subdtype helper * add Engine.get_registered_dtypes * fix Engine error when registering a base DataType * fix pandas_engine DateTime string alias * clean up test_dtypes * fix test_extensions * refactor strategies based on DataType hierarchy * refactor io based on DataType hierarchy * replace dtypes module by new DataType hierarchy * fix black * delete dtypes_.py * drop legacy pandas and python 3.6 from CI * fix mypy errors * fix ci-docs * fix conda dependencies * fix lint, update noxfile * simplify nox tests, fix test_io * update ci build * update nox * pin nox, handle windows data types * fix windows platform * fix pandas_engine on windows platform * fix test_dtypes on windows platform * force pip on docs CI * test out windows dtype stuff * more messing around with windows * more debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * revert ci * increase cache * testing Co-authored-by: cosmicBboy <[email protected]> * Add DataTypes documentation (#536) * delete print statements * pin furo * fix generated docs not removed by nox * re-organize API section * replace aliased pandas_engine data types with their aliases * drop warning when calling Engine.register_dtype without arguments * add data types to api reference doc * add document for DataType refactor * unpin sphinx and drop sphinx_rtd_theme * add xdoctest * ignore prompt when copying example from doc * add doctest builder when running sphinx-build locally * fix dtypes doc examples * fix pandas_engine.DataType.check * fix pylint * remove whitespaces in dtypes doc * Update docs/source/dtypes.rst * Update dtypes.rst * update docs structure * update nox file * force pip on doctests * update test_schemas * fix docs session not overriding html with doctest output Co-authored-by: Niels Bantilan <[email protected]> * add deprecation warnings for pandas_dtype and PandasDtype enum (#547) * remove auto-generated docs * add deprecation warnings, support pandas>=1.3.0 * add deprecation warnings for PandasDtype enum * fix sphinx * fix windows * fix windows * add support for pyarrow backed string data type (#548) * add support for pyarrow backed string data type * fix regression for pandas < 1.3.0 * add verbosity to test run * loosen strategies unit tests deadline, exclude windows ci * loosen test_strategies.py tests * use "dev" hypothesis profile for python 3.7 * add pandas==1.2.5 test * fix ci * ci typo * don't install environment.yml on unit tests * install nox in ci * remove environment.yml * update environment in ci Co-authored-by: cosmicBboy <[email protected]> Co-authored-by: Jean-Francois Zinque <[email protected]>
cosmicBboy
added a commit
that referenced
this pull request
Jul 24, 2021
* Feature/420 (#454) * parse frictionless schema - using frictionless-py for some of the heavy lifting - accept yaml/json/frictionless schema files/objects directly - frictionless becomes a new requirement for io - apply pre-commit formatting updates to other code in pandera.io - add test to validate schema parsing, from yaml and json sources * improve documentation * update docstrings per code review Co-authored-by: Niels Bantilan <[email protected]> * add type hints * standardise class properties for easier re-use in future * simplify key check * add missing alternative type * update docstring * align name with Column arg * fix NaN check * fix type assertion * create empty dict if constraints not provided Co-authored-by: Niels Bantilan <[email protected]> * decouple pandera and pandas dtypes (#559) * refactor PandasDtype into class hierarchy supported by engines * refactor DataFrameSchema based on DataType hierarchy * refactor SchemaModel based on DataType hierarchy * revert fix coerce=True and dtype=None should be a noop * apply code style * fix running tests/core with nox * consolidate dtype names * consolidate engine internal naming * disable inherited __init__ with immutable(init=False) * delete duplicated immutable * disambiguate dtype variables * add warning on base pandas_engine, numpy_engine.DataType init * fix pylint, mypy errors * fix DataFrameSchema.dtypes return type * enable CI on dtypes branch * Refactor inference, schema_statistics, strategies and io using the DataType hierarchy (#504) * fix pandas_engine.Interval * fix Timedelta64 registration with pandas_engine.Engine * add DataType helpers * add DataType.continuous attribute * add dtypes.is_numeric * refactor schema_statistics based on DataType hierarchy * refactor schema_inference based on DataType hierarchy * fix numpy_engine.Timedelta64.type * add is_subdtype helper * add Engine.get_registered_dtypes * fix Engine error when registering a base DataType * fix pandas_engine DateTime string alias * clean up test_dtypes * fix test_extensions * refactor strategies based on DataType hierarchy * refactor io based on DataType hierarchy * replace dtypes module by new DataType hierarchy * fix black * delete dtypes_.py * drop legacy pandas and python 3.6 from CI * fix mypy errors * fix ci-docs * fix conda dependencies * fix lint, update noxfile * simplify nox tests, fix test_io * update ci build * update nox * pin nox, handle windows data types * fix windows platform * fix pandas_engine on windows platform * fix test_dtypes on windows platform * force pip on docs CI * test out windows dtype stuff * more messing around with windows * more debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * debugging * revert ci * increase cache * testing Co-authored-by: cosmicBboy <[email protected]> * Add DataTypes documentation (#536) * delete print statements * pin furo * fix generated docs not removed by nox * re-organize API section * replace aliased pandas_engine data types with their aliases * drop warning when calling Engine.register_dtype without arguments * add data types to api reference doc * add document for DataType refactor * unpin sphinx and drop sphinx_rtd_theme * add xdoctest * ignore prompt when copying example from doc * add doctest builder when running sphinx-build locally * fix dtypes doc examples * fix pandas_engine.DataType.check * fix pylint * remove whitespaces in dtypes doc * Update docs/source/dtypes.rst * Update dtypes.rst * update docs structure * update nox file * force pip on doctests * update test_schemas * fix docs session not overriding html with doctest output Co-authored-by: Niels Bantilan <[email protected]> * add deprecation warnings for pandas_dtype and PandasDtype enum (#547) * remove auto-generated docs * add deprecation warnings, support pandas>=1.3.0 * add deprecation warnings for PandasDtype enum * fix sphinx * fix windows * fix windows * add support for pyarrow backed string data type (#548) * add support for pyarrow backed string data type * fix regression for pandas < 1.3.0 * add verbosity to test run * loosen strategies unit tests deadline, exclude windows ci * loosen test_strategies.py tests * use "dev" hypothesis profile for python 3.7 * add pandas==1.2.5 test * fix ci * ci typo * don't install environment.yml on unit tests * install nox in ci * remove environment.yml * update environment in ci Co-authored-by: cosmicBboy <[email protected]> Co-authored-by: Jean-Francois Zinque <[email protected]> * improve coverage * fix docs * add pandas accessor tests * pin sphinx * fix lint Co-authored-by: Tom Collingwood <[email protected]> Co-authored-by: Jean-Francois Zinque <[email protected]>
This was referenced Oct 8, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #369
@jeffzi finally getting this into the 0.7.0 release branch! Thanks for all your hard work on this, it's a huge step that will make pandera more accessible in the DS/ML ecosystem!