GridSpacing: Per-Component UnitSI & UnitDimension

[RemiLehe]

Implements issue: Close #122

Description

This allows meshes to have axes with different dimensions (e.g. for phase space data where one axis is a position, and the second axis is a momentum.)

What is introduced, removed or renamed and why?

The attributes gridUnitDimension and gridUnitSI are now N times longer, where N is the dimension of the mesh.
In particular gridUnitDimension is a 1D array of length 7*N. This was preferred over a 2D array of dimension (N,7) since it seems that h5py does not support multidimensional arrays as attributes.

Affected Components

• base: All mesh data

Writer Changes

How does this change affect data writers?
Data writers should now write the longer array for gridUnitSI and add gridUnitDimension.

Note: We could make this backward compatible by saying that, if the writer wrote the previous short-array format, then this applies to each axis... @ax3l Is this worth it or too confusing. -> too confusing, we have the updater for that :)

• openPMD-api (upcoming): https://github.com/openPMD/openPMD-api/...

Reader Changes

How does this change affect data readers?
Data readers should now extract the dimension of each axis.

• openPMD-validator: Create arrays that represent the dimension of each grid axis openPMD-validator#40

Data Converter

How does this affect already existing files with previous versions of the standard?
For existing attributes should be duplicated N times, in order to convert to the new format.

• GridSpacing: Per-Component UnitSI & UnitDimension openPMD-updater#7

[ax3l]

Note: We could make this backward compatible by saying that, if the writer wrote the previous short-array format, then this applies to each axis...

Very good point. I think it's too special to allow this short-cut which just adds extra-checks in readers and writers. So let's not allow to just write it once as it was allowed before.

[ax3l]

while we are on it: it's not only the openPMD parsing tools but general openPMD data readers

[ax3l]

Maybe we better stay with L and M * L / T as defined in unitDimension?

[ax3l]

This is related to #125 & #129 (PR in #194): if we flatten the array we have to specify how.
I would propose to use the wording from #129 as such:
, dimensions shall be ordered from fastest to slowest varying index [of the described mesh].

[ax3l]

update: we probably will go from slowest to fastest in #194

[ax3l]

we need to add the same comment with order as above for flattening

[DavidSagan]

My thought would be to not even mandate any precision (ie just say "real"). For example, why should the standard disallow the use of real4?

[ax3l]

The unit dimension attribute is not a record but just describing the dimensionality ("unit"). It introduces no problem keeping it fixed and makes parsing very simple.

[ax3l]

@RemiLehe pls replace array with 1-dimensional array for consistency.

Suggested change

	- type: array of 7 N *(float64 / REAL8)*
	- type: 1-dimensional array of 7 N *(float64 / REAL8)*

[ax3l]

Annotated the changed index order which is introduced in #194

[ax3l]

will be inverted

[ax3l]

slowest to fastest

[ax3l]

slowest to fastest

[ax3l]

Changes look great!

Let us just add the PR to the converter before we merge.

[PrometheusPi]

@ax3l Does this require changing how we write hdf5 data in the phase space and radiation plugin in PIConGPU?

[ax3l]

@PrometheusPi there is a whole bunch of changes that are changing openPMD in a incompatible way in 2.0.0. This is one of them and will enable that we can describe phase space and radiation plugin in valid openPMD.

When we have all changes incorporated, a changelog will summarize all of them again. Also, we can then use openPMD-api instead of libSplash to write from all PIConGPU plugins.

We will then update all places of PIConGPU and then add openPMD to phase space and radiation plugin.

[ax3l]

Overlong line. Maybe put "Dimensions shall be..." on the next line (indented to same block).

[ax3l]

I would try to make this more clear how the arrays are stored by writing:

"The order of the N 7-value arrays must be identical to the axes in axisLabels."

This makes it consistent with the other attributes above that have the same order and uses the same wording, also consistently with #194.

Suggested change

	elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from slowest to fastest varying index of the described mesh.
	elements, where N is the number of dimensions in the simulation.
	The order of the N 7-value arrays must be identical to the axes in `axisLabels`.

[ax3l]

Suggested change

	elements, where N is the number of dimensions in the simulation. Dimensions shall be ordered from slowest to fastest varying index of the described mesh.
	elements, where N is the number of dimensions in the simulation.
	The order of the N values must be identical to the axes in `axisLabels`.

[franzpoeschel]

Note: This does not extend, but instead changes the definition of gridUnitSI. According to this wording, writing a single scalar to be used for all axes will no longer be supported for 2.0, meaning that all writers need to change this.
Do we additionally want to allow scalars here in 2.0, or should writers migrate?

Currently implementing this by printing a warning for legacy-defined scalar values for gridUnitSI in openPMD/openPMD-api#1534:

Mesh &Mesh::setGridUnitSI(double gusi)
{
    setAttribute("gridUnitSI", gusi);
    if (auto standard = IOHandler()->m_standard;
        standard >= OpenpmdStandard::v_2_0_0)
    {
        std::cerr << "[Mesh::setGridUnitSI] Warning: Setting a scalar "
                     "`gridUnitSI` in a file with openPMD version '" +
                std::string(auxiliary::formatStandard(standard)) +
                "'. Consider specifying a vector instead in order to "
                "specify the gridUnitSI per axis (ref.: "
                "https://github.com/openPMD/openPMD-standard/pull/193).\n";
    }
    return *this;
}