Does scipp support dependent/conditional indices across dimensions?

[fkiraly]

This may be answered by working through docs, but I thought I'd just ask about the feature here.

Does scipp support dependent indices across dimensions?

With this I mean, an index set that may be conditional on an index in a lower level/dimension.

An example would be a collection of measurements at temperature/pressure points. Where the pressure range you are able to measure depends on the temperature, and where the granularity (step size) of pressure you measure at depends on temperature. Because otherwise your device explodes or whatever.

This would be a great feature to have!

On the code side, I explicitly mean conditional index sets, not using "NA" or similar in a Cartesian product index. Because NA can mean many things, for instance one can't easily distinguish whether it corresponds to a failed measurement, or whether the measurement wasn't even attempted (or makes no sense).

[SimonHeybrock]

There is some support for this in form of "multi-dimensional coordinates", albeit potentially too limited for some applications.

The main limitation is that the array sizes have to be regular, i.e. rectangular. Here is an example:

import scipp as sc

temperature = sc.linspace('T', 1, 4, num=3, unit='K')
pressure = sc.array(
    dims=('T', 'p'), unit='MPa', values=[[1, 2, 3], [1.5, 2.5, 3.5], [2, 4, 6]]
)
da = sc.DataArray(temperature * pressure, coords={'T': temperature, 'p': pressure})
da.plot()

Note some more limitations:

• Label-based indexing with the dependent coord will not work (since the result-shape would not be rectangular).
• You cannot have X depend on Y and Y depend on X (well, Scipp supports storing such coordinates, but all but basic operations won't work).

Aside from this, Scipp supports Binned Data. This may be the right choice if you have measurements of more random nature, which you later want to histogram. This would also allow for storing data with sub-lists of unequal length.

On the code side, I explicitly mean conditional index sets, not using "NA" or similar in a Cartesian product index. Because NA can mean many things, for instance one can't easily distinguish whether it corresponds to a failed measurement, or whether the measurement wasn't even attempted (or makes no sense).

In case the requirement for rectangular arrays with multi-dim coordinates is too limiting, also note Scipp's support for Masking. This is more powerful than Numpy's masked arrays, since Scipp uses a dictionary of masks. With this one could use NAN for values that were not measured while maintaining a named mask that indicates the meaning of those NAN values. This would allow for distinguishing different reasons for NAN values.

[fkiraly]

Thanks for the swift reply, @SimonHeybrock!

The array index feature is nice!

Slightly disappointed that it's not possible without nans, but that's how it usually is. Multi-column index for long format is of course always an option.

[SimonHeybrock]

Slightly disappointed that it's not possible without nans, but that's how it usually is.

I'd be interested if you are aware of good solutions to this? By "good" I probably mean general purpose and not overly complex.

[fkiraly]

No, I am not aware of good solutions for this, that's why I'm always on the lookout...

There are ok-ish solutions:

• ragged arrays a la awkward-array, but that does not have dimension/column labels
• using native pandas multi-index in long format, similar to data representation pd-multiindex in sktime
• nested pandas, but that is low performance, not officially supported, and stopped working well around pandas 1.5.0
• pandas ExtensionArray, but there is no accepted design and it's fiddly and brittle
• nested dict, but this is highly custom and therefore maintenance heavy

[fkiraly]

from an impact perspective, the time series community would love to see sth that's good. scipp seems close.

If you decide to build this, you might have an instant adopter in sktime :-)

[SimonHeybrock]

My question would be: how big a gap is there between what Scipp's "binned data" can do and what you need?

Here is an example:

import scipp as sc
import numpy as np

rng = np.random.default_rng(0)
pressure = sc.array(dims=['measurement'], values=rng.uniform(1000.0, 2000.0, size=1000), unit='pascal')
temperature = sc.array(dims=['measurement'], values=rng.integers(4, 10, size=1000), unit='K', dtype='float64')
values = sc.array(dims=['measurement'], values=rng.uniform(0.0, 1.0, size=1000), unit='m')
measurement = sc.DataArray(data=values, coords={'temperature': temperature, 'pressure': pressure})
measurement = sc.sort(measurement, 'pressure')  # optional, Scipp grouping/binning preserves order
measurement = measurement.group('temperature')

# Select all data points with pressure between 1500 and 1700 Pa
pstart = 1500.0 * sc.Unit('pascal')
pstop = 1700.0 * sc.Unit('pascal')
measurement.bins['pressure', pstart:pstop]  # returns a copy

# Access a single temperature point
measurement['temperature', 5.0 * sc.Unit('K')].value