-
Notifications
You must be signed in to change notification settings - Fork 224
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wrap GMT's standard data type GMT_DATASET for table inputs #2729
Conversation
This reverts commit 19eb3aa.
This benchmark report looks scary, but it doesn't match my test in #2730 (comment). I guess it's because codspeed doesn't count the I/O time (via pd.read_csv). |
Ping @GenericMappingTools/pygmt-maintainers for review on this PR. There are so many pending changes/PRs that rely on the changes in this PR. |
Co-authored-by: Wei Ji <[email protected]>
Co-authored-by: Wei Ji <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Super big thank you for pushing this through @seisman, especially in cherry-picking many of the changes into separate PRs to make this easier to review! I'm still a little unsure about the triple-nested classes as mentioned at #2729 (comment), but we can always un-nest them later if needed (since they're marked as private classes).
Won't hold this PR up any longer, excited to have PyGMT not store intermediate CSV files in the next release 🚀
Description of proposed changes
Similar to #2398, this PR contains some experimental codes to support GMT's standard data type
GMT_DATASET
for table inputs.Here are some technical background. In GMT, table inputs are stored in a
GMT_DATASET
object. AGMT_DATASET
object can contain several table files (stored inGMT_DATATABLE
) and each table can contain several segments (stored inGMT_DATASEGMENT
), and each data segment have M rows and N columns.References: https://docs.generic-mapping-tools.org/latest/devdocs/api.html#struct-dataset
The ultimate goals are:
GMTTempfile
for modules that output a table [tracked in Get rid of temporary files from pygmt functions and plotting methods #2730]GMT_DATASET
object and pass it as input to some modulesCurrently, I'm doing too much work in this PR, for example:
Wrapping the GMT C APIMerged to main in clib: Wrap the GMT API function GMT_Read_VirtualFile #2993GMT_Read_Virtualfile
Provide a new session methodMerged to main in clib: Add the virtualfile_out method for creating output virtualfile #3057virtualfile_to_data
to allow GMT modules write output into a virtual file.GMT_DATATABLE
andGMT_DATASEGMENT
grd2xyz
,grdtrack
) to show how this PR will change the way we handle table-like outputs.Thus, we need to split this PR into a few smaller PRs if this PR is proven to be working.