Export to csv and json #42
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
isichei
suggested changes
Jan 8, 2021
isichei
reviewed
Jan 20, 2021
| pd_date_type=date_args, | ||
| pd_timestamp_type=date_args, | ||
| ) | ||
| # Write to StringIO then convert to BytesIO so Arrow can read it |
There was a problem hiding this comment.
I like this comment. Succinctly points users to a gotcha.
isichei
approved these changes
Jan 20, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds export functions and tests to meet issue #5. The options and formatting are set up so if you read a table using a schema, export it to csv or json, then read the exported file with the same schema, the tables will be identical.
This doesn't implement the option to conform a table to a schema as part of exporting. Currently the export functions don't give much control over the format of the exported data - particularly for dates and times. But I think this is okay, as the way the data is written to file is less important than how the package reads those files.
These export functions mostly use default pandas export options. However, period types have to be reformatted as date strings so Arrow can read them. Exporting dates and times to json also requires explicitly converting them to strings to avoid epoch values (which Arrow reads differently from Pandas) and problems with missing values.