IterationEncoding: variableBased #250

ax3l · 2021-03-08T23:37:44Z

Please add a brief description (one sentence) here and link the issue this pull-request implements

Implements issue: #221 #236

Description

Add standard guidance for stepBased iteration encoding.

stepBased iteration encoding uses features of a storage, backend API or file format to encode time-varying data sets and attributes.

Affected Components

base
FORMAT: ADIOS

Logic Changes

Instead of storing iterations (snapshots) in individual groups, we rely on internal capabilities of a data format to store updates/revisions.

add a new snapshot variable to map backend steps to openPMD iterations (Variable-Based iteration layout openPMD-api#855)

ADIOS:

use variables for attributes (Use ADIOS variables for openPMD attributes openPMD-api#813)
document /__data__ used for openPMD record components (Use ADIOS variables for openPMD attributes openPMD-api#813)

Writer Changes

openPMD-api proof of concept: Use ADIOS variables for openPMD attributes openPMD-api#813 Variable-Based iteration layout openPMD-api#855

Reader Changes

openPMD-validator: https://github.com/openPMD/openPMD-validator/...
openPMD-viewer: https://github.com/openPMD/openPMD-viewer/...
yt: N/A (hdf5 only atm.)
VisIt: https://github.com/openPMD/openPMD-visit-plugin
openPMD-api: Use ADIOS variables for openPMD attributes openPMD-api#813 Variable-Based iteration layout openPMD-api#855 Mapping between ADIOS steps and openPMD iterations openPMD-api#949

Data Converter

Since this is a new iteration encoding that exists in parallel to existing iteration encodings, no conversion is necessary.

FORMAT_ADIOS.md

ax3l · 2021-03-09T01:15:45Z

FORMAT_ADIOS.md

+
+In order to correlate openPMD iterations with ADIOS steps, the *root* group (path `/`) in ADIOS must contain a variable:
+
+  - `__step__`


Make sure we used __step__ and not __steps__

Discussed with @franzpoeschel let's call this snapshot without __ and with openPMD 2.0 "iteration" naming: #148

We read snapshot in frontend classes in openPMD-api and it might be useful for other use cases, e.g. data sets with only one snapshot (iteration).

Note that in the current implementations this attribute is not part of the root group, but in /data, so it's /data/snapshot.
In variable-based encoding this makes the attribute look part of the iteration:

> bpls ../samples/variableBasedSeries.bp4 -alt Step 0: string /basePath attr = "/data/%T/" uint64_t /data/changing_value attr = 0 double /data/dt attr = 1 double /data/meshes/E/0/position attr = 0 uint64_t /data/meshes/E/0/shape attr = 1 double /data/meshes/E/0/unitSI attr = 1 uint64_t /data/meshes/E/0/value attr = 0 uint64_t /data/meshes/E/attr_0 attr = 0 string /data/meshes/E/axisLabels attr = {"x"} string /data/meshes/E/dataOrder attr = "C" string /data/meshes/E/geometry attr = "cartesian" double /data/meshes/E/gridGlobalOffset attr = 0 double /data/meshes/E/gridSpacing attr = 1 double /data/meshes/E/gridUnitSI attr = 1 float /data/meshes/E/timeOffset attr = 0 double /data/meshes/E/unitDimension attr = {0, 0, 0, 0, 0, 0, 0} int32_t /data/meshes/E/x {1000} double /data/meshes/E/x/position attr = 0 double /data/meshes/E/x/unitSI attr = 1 int32_t /data/meshes/E/y {1} double /data/meshes/E/y/position attr = 0 double /data/meshes/E/y/unitSI attr = 1 string /data/meshes/changing_constant/axisLabels attr = {"x"} uint64_t /data/meshes/changing_constant/changing_constant/shape attr = 0 uint64_t /data/meshes/changing_constant/changing_constant/value attr = 0 string /data/meshes/changing_constant/dataOrder attr = "C" string /data/meshes/changing_constant/geometry attr = "cartesian" double /data/meshes/changing_constant/gridGlobalOffset attr = 0 double /data/meshes/changing_constant/gridSpacing attr = 1 double /data/meshes/changing_constant/gridUnitSI attr = 1 double /data/meshes/changing_constant/position attr = 0 uint64_t /data/meshes/changing_constant/shape attr = 0 float /data/meshes/changing_constant/timeOffset attr = 0 double /data/meshes/changing_constant/unitDimension attr = {0, 0, 0, 0, 0, 0, 0} double /data/meshes/changing_constant/unitSI attr = 1 uint64_t /data/meshes/changing_constant/value attr = 0 uint64_t /data/particles/changing_constant/position/position/shape attr = 0 uint64_t /data/particles/changing_constant/position/position/value attr = 0 uint64_t /data/particles/changing_constant/position/shape attr = 0 float /data/particles/changing_constant/position/timeOffset attr = 0 double /data/particles/changing_constant/position/unitDimension attr = {1, 0, 0, 0, 0, 0, 0} double /data/particles/changing_constant/position/unitSI attr = 1 uint64_t /data/particles/changing_constant/position/value attr = 0 uint64_t /data/snapshot attr = 0 double /data/time attr = 0 double /data/timeUnitSI attr = 1 string /date attr = "2022-08-19 08:35:01 +0000" string /iterationEncoding attr = "variableBased" string /iterationFormat attr = "/data" string /meshesPath attr = "meshes/" string /openPMD attr = "1.1.0" uint32_t /openPMDextension attr = 0 string /particlesPath attr = "particles/" string /software attr = "openPMD-api" string /softwareVersion attr = "0.15.0-dev"

In group-based encoding however, the snapshot attribute is then at the level above the single iterations:

> bpls ../samples/bp4steps_yes_yes.bp/ -alt Step 0: string /basePath attr = "/data/%T/" double /data/0/dt attr = 1 string /data/0/meshes/E/axisLabels attr = {"x"} string /data/0/meshes/E/dataOrder attr = "C" string /data/0/meshes/E/geometry attr = "cartesian" double /data/0/meshes/E/gridGlobalOffset attr = 0 double /data/0/meshes/E/gridSpacing attr = 1 double /data/0/meshes/E/gridUnitSI attr = 1 float /data/0/meshes/E/timeOffset attr = 0 double /data/0/meshes/E/unitDimension attr = {0, 0, 0, 0, 0, 0, 0} string /data/0/meshes/E/vector_of_string attr = {"vector", "of", "string"} int32_t /data/0/meshes/E/x {10} double /data/0/meshes/E/x/position attr = 0 double /data/0/meshes/E/x/unitSI attr = 1 double /data/0/time attr = 0 double /data/0/timeUnitSI attr = 1 uint64_t /data/snapshot attr = 0 string /date attr = "2022-08-19 08:38:04 +0000" string /iterationEncoding attr = "groupBased" string /iterationFormat attr = "/data/%T/" string /meshesPath attr = "meshes/" string /openPMD attr = "1.1.0" uint32_t /openPMDextension attr = 0 string /software attr = "openPMD-api" string /softwareVersion attr = "0.15.0-dev"

Add standard guidance for stepBased iteration encoding.

ax3l · 2021-04-07T22:01:19Z

STANDARD.md

+
+In order to correlate openPMD iterations with an index of data-format internal updates/steps or an index in the slowest varying dimension of an array, the *root* group (path `/`) must contain an additional variable once `stepBased` is chosen for `iterationEncoding`:
+
+  - `snapshot`


Question: we could allow to skip this if only one iteration (snapshot) is written.
In that case, the implied value should be 0 and there must be exactly one update/step in the data format.

This is what the other backends actually do when that iteration encoding is chosen, see the variableBasedSingleIteration test. The snapshot attribute is not written.

With the current state of openPMD/openPMD-api#949, the snapshot attribute is always written, but not required at read-time (then assumed to be 0). I should add a test somehow to ensure that reading without snapshot works as intended.

ax3l · 2021-04-24T01:27:36Z

FORMAT_ADIOS.md

@@ -32,3 +32,28 @@ Output from `bpls -A` for a boolean attribute `pybool` stored in the location of

 There is no convention yet for a unique representation of ADIOS2 variables with boolean type.
 Thus, implementations should cast the data to and from `unsigned char` instead.
+
+## `stepBased` Encoding of Iterations


Rename: I am not sure why, but for some reason we now call this variableBased in openPMD/openPMD-api#855

@franzpoeschel let's clarify what we pick, shall I update the standard PR to be named variableBased, too?

Yep, we discussed this a while ago and came to the conclusion to call it variable-based since steps are an ADIOS2-specific feature, but this encoding generally relies on a backend's ability to have variable datasets.

ax3l · 2021-07-29T17:30:33Z

FORMAT_ADIOS.md

+
+## Attributes
+
+openPMD **attributes** stored as ADIOS `Variables` at the location where they would usually be stored.


@franzpoeschel raised this is currently implemented that way and I think since we don't need to change the type of attributes over time, we can keep it so:

Suggested change

openPMD **attributes** stored as ADIOS `Variables` at the location where they would usually be stored.

openPMD **attributes** stored as ADIOS `Variables` at the location where they would usually be stored.

The `__is_boolean__/...` qualifiers are still stored as ADIOS `Attribute`.

openPMD attributes stored as ADIOS Variables

Is this outdated? We will remove the new ADIOS2 schema where this happens

franzpoeschel · 2022-01-25T10:22:48Z

FORMAT_ADIOS.md

+
+## `stepBased` Encoding of Iterations
+
+The `iterationEncoding` mode `stepBased` must be implemented via ADIOS steps.


Suggested change

The `iterationEncoding` mode `stepBased` must be implemented via ADIOS steps.

The `iterationEncoding` mode `variableBased` must be implemented via a backend's feature to describe *variable* datasets and attributes.

This means that such datasets and attributes are present in different versions with different contents.

I think I missed here that this file was ADIOS-specific.

franzpoeschel · 2022-01-25T10:23:28Z

FORMAT_ADIOS.md

+
+## Datasets
+
+An openPMD **data set** is represented by an group prefix that contains an ADIOS variable `__data__`.


This describes the ADIOS2 schema that we will abolish

Generally, we will stick with a schema that is similar to the current schema (datasets and attributes are distinguished by using different backend features (variables and attributes)), but with two additions:

Attributes can be variable now too

We will use some protocol for identifying if a group is active in the current step

Since that updated schema is not yet implemented, I'd suggest we don't describe this just yet. I would not like to standardize something that in the end turns out to not work well.

I've started exploring that new schema here.

franzpoeschel · 2022-01-25T10:24:22Z

FORMAT_ADIOS.md

+
+An openPMD **data set** is represented by an group prefix that contains an ADIOS variable `__data__`.
+
+**attributes** are defined further below and can also appear at the dataset's **group** prefix level.


What does "further below" mean in this context?

franzpoeschel · 2022-01-25T10:40:41Z

FORMAT_ADIOS.md

+  double    /data/meshes/E/x/__data__        10*{1000}
+  double    /data/meshes/E/x/position        10*{1}
+  double    /data/meshes/E/x/unitSI          10*scalar
+```


Outdated. Since the old ADIOS2 schema does not yet support variable-based iteration encoding, we currently only have this kind of experimental implementation.

franzpoeschel · 2022-01-25T10:41:41Z

STANDARD.md

@@ -212,6 +216,7 @@ Each file's *root* group (path `/`) must further define the attributes:
    - allowed values:
      - `fileBased` (multiple files)
      - `groupBased` (one file)
+      - `stepBased` (one file with internal encoding for iterations, if supported by the data format)


Suggested change

- `stepBased` (one file with internal encoding for iterations, if supported by the data format)

- `variableBased` (one file with internal encoding for iterations, if supported by the data format)

franzpoeschel · 2022-01-25T10:41:59Z

STANDARD.md

+    - allowed values:
+      - see *Iterations and Time Series* below
+      - for `fileBased` and `groupBased`, this is fixed to `/data/%T/`
+      - for `stepBased` this is fixed to `/data/`


Suggested change

- for `stepBased` this is fixed to `/data/`

- for `variableBased` this is fixed to `/data/`

franzpoeschel · 2022-01-25T10:44:32Z

STANDARD.md

+        - data-format internal convention
+        - *slowest varying index* of data
+
+### `stepBased` Encoding of Iterations


Suggested change

### `stepBased` Encoding of Iterations

### `variableBased` Encoding of Iterations

franzpoeschel · 2022-01-25T10:45:22Z

STANDARD.md

+
+### `stepBased` Encoding of Iterations
+
+In order to correlate openPMD iterations with an index of data-format internal updates/steps or an index in the slowest varying dimension of an array, the *root* group (path `/`) must contain an additional variable once `stepBased` is chosen for `iterationEncoding`:


Suggested change

In order to correlate openPMD iterations with an index of data-format internal updates/steps or an index in the slowest varying dimension of an array, the *root* group (path `/`) must contain an additional variable once `stepBased` is chosen for `iterationEncoding`:

In order to correlate openPMD iterations with an index of data-format internal updates/steps or an index in the slowest varying dimension of an array, the *root* group (path `/`) must contain an additional variable once `variableBased` is chosen for `iterationEncoding`:

franzpoeschel · 2022-01-25T10:47:54Z

STANDARD.md

+In order to correlate openPMD iterations with an index of data-format internal updates/steps or an index in the slowest varying dimension of an array, the *root* group (path `/`) must contain an additional variable once `stepBased` is chosen for `iterationEncoding`:
+
+  - `snapshot`
+    - type: 1-dimensional array containing N *(int)* elements, where N is the number of updates/steps in the data format


This is currently not implemented as an array, but as a scalar variable that changes across steps.

The implementation actually accepts arrays at read-time, but I should test that it works. At write time, the API currently only produces scalars.

franzpoeschel · 2022-01-25T10:49:27Z

STANDARD.md

+    - description: for each update/step in a data format, this variable needs to be updated with the corresponding openPMD iteration.
+    - note: in some data formats, updates/steps are absolute and not every update/step contains an update for each declared openPMD record
+    - advice to implementers: an openPMD iteration might be spread over multiple updates/steps, but not vice versa.
+                              In such a scenario, an individual openPMD record's update/step must appear exactly once per iteration.


Note: A similar situation can occur when using Append mode: An iteration is then present multiple times with redundant definitions. This will either be solved by truncation or by reading only the first/last instance of that iteration.

franzpoeschel · 2022-01-25T10:49:48Z

STANDARD.md

-files  (`fileBased`) or series of groups (`groupBased`) should have
-attributes that describe the current time and the last
-time step.
+In addition to holding information about the iteration, each series of files  (`fileBased`), series of groups (`groupBased`) or internally encoded iterations (`stepBased`) should have attributes that describe the current time and the last time step.


Suggested change

In addition to holding information about the iteration, each series of files (`fileBased`), series of groups (`groupBased`) or internally encoded iterations (`stepBased`) should have attributes that describe the current time and the last time step.

In addition to holding information about the iteration, each series of files (`fileBased`), series of groups (`groupBased`) or internally encoded iterations (`variableBased`) should have attributes that describe the current time and the last time step.

Co-authored-by: Axel Huebl <[email protected]>

Make the example more realistic, also minor fixes

Updates to variable-based encoding

ax3l added the major change non-backwards compatible change label Mar 8, 2021

ax3l requested a review from franzpoeschel March 8, 2021 23:37

ax3l assigned franzpoeschel Mar 8, 2021

ax3l changed the title ~~[WIP] IterationEncoding: stepBased~~ IterationEncoding: stepBased Mar 9, 2021

ax3l force-pushed the topic-stepBasedEncoding branch from 4abd2bf to b22738b Compare March 9, 2021 01:14

ax3l commented Mar 9, 2021

View reviewed changes

FORMAT_ADIOS.md Outdated Show resolved Hide resolved

ax3l commented Mar 9, 2021

View reviewed changes

ax3l mentioned this pull request Apr 6, 2021

Variable-Based iteration layout openPMD/openPMD-api#855

Merged

13 tasks

ax3l added 2 commits April 7, 2021 14:36

IterationEncoding: stepBased

367efba

Add standard guidance for stepBased iteration encoding.

ADIOS: New Attribute, Variable & stepbased Encoding

c81864a

ax3l force-pushed the topic-stepBasedEncoding branch 3 times, most recently from 299e892 to 7823519 Compare April 7, 2021 21:59

ax3l commented Apr 7, 2021

View reviewed changes

Snapshot naming, Remove Specs Mapping

cee7330

ax3l force-pushed the topic-stepBasedEncoding branch from 7823519 to cee7330 Compare April 7, 2021 22:02

ax3l changed the title ~~IterationEncoding: stepBased~~ IterationEncoding: variableBased Apr 24, 2021

ax3l changed the title ~~IterationEncoding: variableBased~~ IterationEncoding: stepBased Apr 24, 2021

ax3l commented Apr 24, 2021

View reviewed changes

ax3l mentioned this pull request Apr 24, 2021

[WIP] openPMD: variableBased IterationEncoding ECP-WarpX/WarpX#1909

Closed

ax3l changed the title ~~IterationEncoding: stepBased~~ IterationEncoding: variableBased May 20, 2021

ax3l commented Jul 29, 2021

View reviewed changes

franzpoeschel reviewed Jan 25, 2022

View reviewed changes

franzpoeschel suggested changes Jan 25, 2022

View reviewed changes

ax3l mentioned this pull request Jun 7, 2022

Mapping between ADIOS steps and openPMD iterations openPMD/openPMD-api#949

Merged

11 tasks

franzpoeschel and others added 5 commits August 19, 2022 11:43

Update stepBased -> variableBased, remove schema2021-specifics

2a6938c

Document updated usage of snapshot attribute

34f9cc7

definition updates from today

a285b69

Apply suggestions from code review

1f5aa32

Co-authored-by: Axel Huebl <[email protected]>

Update after review

d7f3d59

Make the example more realistic, also minor fixes

Merge pull request #1 from franzpoeschel/topic-variableBasedEncoding

3f202a7

Updates to variable-based encoding

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IterationEncoding: variableBased #250

IterationEncoding: variableBased #250

ax3l commented Mar 8, 2021 •

edited

Loading

ax3l Mar 9, 2021 •

edited

Loading

ax3l Apr 6, 2021 •

edited

Loading

franzpoeschel Aug 19, 2022

ax3l Apr 7, 2021

franzpoeschel Jan 25, 2022

franzpoeschel Aug 19, 2022

franzpoeschel Aug 19, 2022

ax3l Apr 24, 2021

franzpoeschel Jan 25, 2022

ax3l Jul 29, 2021 •

edited

Loading

franzpoeschel Jan 25, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Aug 19, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Aug 19, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Aug 19, 2022

franzpoeschel Aug 19, 2022

franzpoeschel Jan 25, 2022

franzpoeschel Jan 25, 2022


		In order to correlate openPMD iterations with ADIOS steps, the root group (path `/`) in ADIOS must contain a variable:

		- `__step__`


		In order to correlate openPMD iterations with an index of data-format internal updates/steps or an index in the slowest varying dimension of an array, the root group (path `/`) must contain an additional variable once `stepBased` is chosen for `iterationEncoding`:

		- `snapshot`


		## Attributes

		openPMD attributes stored as ADIOS `Variables` at the location where they would usually be stored.

	openPMD attributes stored as ADIOS `Variables` at the location where they would usually be stored.
	openPMD attributes stored as ADIOS `Variables` at the location where they would usually be stored.
	The `__is_boolean__/...` qualifiers are still stored as ADIOS `Attribute`.


		## `stepBased` Encoding of Iterations

		The `iterationEncoding` mode `stepBased` must be implemented via ADIOS steps.

	The `iterationEncoding` mode `stepBased` must be implemented via ADIOS steps.
	The `iterationEncoding` mode `variableBased` must be implemented via a backend's feature to describe variable datasets and attributes.
	This means that such datasets and attributes are present in different versions with different contents.


		## Datasets

		An openPMD data set is represented by an group prefix that contains an ADIOS variable `__data__`.


		An openPMD data set is represented by an group prefix that contains an ADIOS variable `__data__`.

		attributes are defined further below and can also appear at the dataset's group prefix level.

	- `stepBased` (one file with internal encoding for iterations, if supported by the data format)
	- `variableBased` (one file with internal encoding for iterations, if supported by the data format)

	- for `stepBased` this is fixed to `/data/`
	- for `variableBased` this is fixed to `/data/`

	### `stepBased` Encoding of Iterations
	### `variableBased` Encoding of Iterations

	In addition to holding information about the iteration, each series of files (`fileBased`), series of groups (`groupBased`) or internally encoded iterations (`stepBased`) should have attributes that describe the current time and the last time step.
	In addition to holding information about the iteration, each series of files (`fileBased`), series of groups (`groupBased`) or internally encoded iterations (`variableBased`) should have attributes that describe the current time and the last time step.

IterationEncoding: variableBased #250

Are you sure you want to change the base?

IterationEncoding: variableBased #250

Conversation

ax3l commented Mar 8, 2021 • edited Loading

Description

Affected Components

Logic Changes

Writer Changes

Reader Changes

Data Converter

ax3l Mar 9, 2021 • edited Loading

Choose a reason for hiding this comment

ax3l Apr 6, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ax3l Jul 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ax3l commented Mar 8, 2021 •

edited

Loading

ax3l Mar 9, 2021 •

edited

Loading

ax3l Apr 6, 2021 •

edited

Loading

ax3l Jul 29, 2021 •

edited

Loading