Series Considerations for Streaming Improvements

[ax3l]

As a quick reminder, openPMD is not a file format or even a format per se, it's a meta-data markup. We just implement it on top of popular hierarchical file formats as a reference.

@franzpoeschel et al. already implemented working implementations of streaming openPMD data and meta-data with ADIOS2 SST openPMD/openPMD-api#570.

We should consider several updates to openPMD to make streaming workflows more natural.

IterationEncoding in Series

A third iteration encoding mode in data series, maybe called "variableBased" could express that new iterations (snapshots) of data are stored as as implementation-specific "variable update" of the data. This is what ADIOS supports for example (begin and end step on a variable). Other kinds of storage media, such as object storage and databases of all kinds implement similar things.

Attributes

On a high level, if an attribute did not change since the last iteration (snapshot), it might be allowed to be omitted. Potentially we want to add a reference mechanism to "link" the last iteration (snapshot) that had the meta-data attributes of interest.