Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

netcdf save rehash : beginnings of append support #52

Open
wants to merge 4 commits into
base: working_nc_rehash_WORKINGREFACTOR
Choose a base branch
from

Conversation

pp-mo
Copy link
Owner

@pp-mo pp-mo commented Feb 26, 2019

WIP : DO NOT MERGE
Post-refactor (#51) code, beginning work on implementing a netcdf-append strategy

Status: highly preliminary (!) :

  • the _offset_keys routine may not now be necessary
    see http://docs.dask.org/en/latest/array-api.html#dask.array.store
    -- use of the regions key may be able to replace this.
  • the actual "append" operation is really very simple ...
  • BUT the 'validity checking' code in the _prepare_append method is huge + still untested
    it has become a monster !!
  • the 'netcdf data representation' classes introduced in netcdf save rehash : working refactor (to support future append) #51 now look doubtful to me
    • use of '_SlotsHolder' may be excessive : simpler to define classes directly ?
    • or replace with xarray objects (???)
  • the use of attached "tracking" properties in the 'label_links' routine (inner routine to _prepare_append) seems inappropriate
    • almost certainly clearer to store relational info in separate mapping objects (and maybe even simpler)

@pp-mo pp-mo changed the title Working nc rehash netcdf save rehash : beginnings of append support Feb 26, 2019
@pp-mo
Copy link
Owner Author

pp-mo commented Feb 26, 2019

Regarding "replace with xarray objects (?)",
Maybe this whole effort would sit better in the xarray project (where they already support appending writes, though at present just for adding new variables, not extending existing ones AFAICT)
- in that case, we should approach it in Iris by improving xarray integration instead : provide save-to-xarray as alternative to save-to-file.

Presumably that should mean loading, too.

Interoperability is currently provided in xarray (to_iris + from_iris),
but of course that doesn't implement all the CF logic that Iris can.
If Iris has a more sophisticated view of CF data, then it makes sense to define xarray interoperability in Iris, instead ??

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant