feat(datasets): Extend preview mechanism #595
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 tasks
5 tasks
…rg/kedro-plugins into extend-preview-mechanism
SajidAlamQB
changed the title
SajidAlamQB
changed the title
|
@SajidAlamQB Is this still a draft or waiting for something? |
Its ready for review, just some coverage tests that are needed. |
…rg/kedro-plugins into extend-preview-mechanism
…rg/kedro-plugins into extend-preview-mechanism
noklam
reviewed
Mar 21, 2024
noklam
reviewed
Mar 21, 2024
noklam
reviewed
Mar 21, 2024
There was a problem hiding this comment.
Few comments:
- I would try to stick with
_loadas much as possible to avoid re-implementation and edge case that run into difference load data. There may be some special case ifloadandpreviewexpect different data, we discussed this in tech design before but that's another topic. - Performance - the
previewshould be a lightweight solution and should never read the full data.
rashidakanchwala
approved these changes
Apr 3, 2024
noklam
reviewed
Apr 3, 2024
noklam
reviewed
Apr 3, 2024
noklam
reviewed
Apr 3, 2024
noklam
approved these changes
Apr 3, 2024
tgoelles
pushed a commit
to tgoelles/kedro-plugins
that referenced
this pull request
Jun 6, 2024
* Extend preview to Parquet * Update sql_dataset.py * Update sql_dataset.py * update preview method for parquetdataset * Update sql_dataset.py * extend preview to JSONDataset * add json preview * add preview for pickledataset * Update json_dataset.py * lint * add tests for parquet and json * lint * rem pickle fix docstring * Fix parquet test * fix pandas.json tests * add coverage for sqldataset * lint * coverage for sanitisation of sql * changes based on review * use pyarrow for parquet preview * align jsondataset with spike * Update json_dataset.py * Update json_dataset.py * pass lines=true and nrows * update docstring * Update test_json_dataset.py * revert change * use sqlalchemy instead of query * fix sql tests Signed-off-by: tgoelles <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Related to: kedro-org/kedro-viz#1622
This PR introduces an update to extend the preview functionality across more kedro-datasets, extending support for different data formats.
List of Datasets we are aiming to support in this PR:
ParquetDatasetSQLTableDatasetpandas.JSONDatasetJSONDatasetDevelopment notes
Checklist
RELEASE.mdfile