Problematic interpolation on mismatched time series

[smithara]

I found this problem specifically with the usage of the B_NEC or F measurement - model residuals when also accessing the EFI dataset, but it probably has broader effects...

Using EFI means that the data cadence is set to 0.5-second. B_NEC/F measurements then use a nearest-neighbour interpolation. The model evaluations instead appear to use the higher cadence positions from the EFI dataset. This leads to an oscillating point-to-point variation in the residual as shown in the image here.

viresclient notebook:
https://nbviewer.jupyter.org/github/smithara/viresclient_examples/blob/master/joint_MAG_EFI_IBI.ipynb
In particular look at the B_NEC and model values at the bottom of the notebook - B_NEC stays the same over two sample points whereas the model changes at every point. [edit: see also the VirES workspace file below]

I am not sure what the "correct" behaviour should be. Is it a problem to allow different datasets to retain their original sampling points rather than forcing them onto a shared time series?

[smithara]

20190915T184235_vires_settings.zip

VirES workspace state file attached

[pacesm]

Is it a problem to allow different datasets to retain their original sampling points rather than forcing them onto a shared time series?

Unfortunately, it is. The web client, as it is designed, can not handle datasets of one spacecraft with different sampling. But we could try to mitigate these artifacts.

This actually how the oscillation is created ...

The MAGx_LR data sampled 1Hz and and aligned to the edge of a UTC second (blues rectangles)
are interpolated to the EFIx_LP samples (~2Hz sampling, not aligned with UTC, with occasional drop-outs) using the Nearest-Neighbor method (see the green circles). Then the magnetic models are evaluated at the EFIx_LP times and positions and subtracted from the measurements (magenta circles). The models are therefore evaluated with a small spatio-temporal offset compared to the original MAGx_LR measurements (orange rectangles).

There are two possible ways how to get rid of the oscillations:

1. Evaluate the magnetic models for the actual MAGx_LR times and locations and then interpolate them the same way as the measurements.
2. Use a better interpolation for the MAGx_LR (F, B_NEC) measurements so that they are closer.

The first would require significant effort to rewrite the server-side data pipeline.

The latter can be done easily by switching to the recently added linear interpolation ESA-VirES/VirES-Server@0eb1c1a

[smithara]

It looks better now with the new linear interpolation.

There's likely some issues elsewhere with the interpolation behaviour. At the very least it is moving data points from how they are in the original data, which could cause problems when downloading the data for further processing. Maybe we should consider some health warning about selecting multiple datasets - that there is this merging behaviour going on.

[pacesm]

There's likely some issues elsewhere with the interpolation behaviour. At the very least it is moving data points from how they are in the original data, which could cause problems when downloading the data for further processing. Maybe we should consider some health warning about selecting multiple datasets - that there is this merging behaviour going on.

I agree.