Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposed: make option "recordBased" for iterationEncoding #236

Open
cosmonaut-ok opened this issue Jan 4, 2021 · 1 comment
Open

Proposed: make option "recordBased" for iterationEncoding #236

cosmonaut-ok opened this issue Jan 4, 2021 · 1 comment
Labels
major change non-backwards compatible change question

Comments

@cosmonaut-ok
Copy link

It's not comfortable to make separate file/group for each iteration in case of active working with time series (e.g. if required to select single point from every iteration). Much better to make n+1-dimensional array and write data to it. But, it breaks openPMD compatibility.

@ax3l
Copy link
Member

ax3l commented Jan 7, 2021

Hi @cosmonaut-ok,

Thank you for the idea!

We plan something very similar, currently drafted as "variableBased" encoding related to #221.
In that, we will use the intrinsic capabilities of e.g. the ADIOS2 format to encode steps inside a data set (variable) if the file format supports this.

n+1 dimensionality in other formats, e.g. HDF5, is thinkable as a work-around for those formats. Yet we need to be aware that contrary to ADIOS1/2 "step" encoding, HDF5 would then be limited to stay with the same shape of the data set for all iterations; which is limiting for some applications (e.g. particle data shape changes over time).

@ax3l ax3l added major change non-backwards compatible change question labels Jan 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
major change non-backwards compatible change question
Projects
None yet
Development

No branches or pull requests

2 participants