Currently failing orso validation

[andyfaff]

@bmaranville @pkienzle changes made to refl1d last week have caused the orso validation run to start failing for refl1d.

Test 0 and 6 fail when using refl1d.reflectivity.reflectivity_amplitude, but pass with refl1d.abeles.refl. Those tests aren't using any resolution smearing.

Test4 fails when using refl1d.abeles.refl, this one is a resolution smearing test. The test script tries a QProbe oversampling with a factor of 21, and if that fails tries to create its own oversampling.

Can you look into this issue and see whether it's truly an issue with refl1d, or with the orso validation code? Some of the divergences from the 'known good' arrays are quite large.

[bmaranville]

Thanks for keeping an eye on this! I was aware that some of the validation from ORSO were failing - I'll be looking into these before we make a release.

We just completely switched from using C-extensions to numba for all the reflectivity calculations and convolutions etc., so it is a big change and we will be running many tests before release.

In particular I may need to pay attention to the combination of QProbe and oversampling, which doesn't really come up for real experimental datasets but is important for testing.

[bmaranville]

The tests are now passing - though there is still a warning for QProbe oversampling, so I'll leave this open for now.

[andyfaff]

Great. The tolerances for the resolution smearing test are interesting to think about. I'm not sure quite how tight/loose they should be before they don't make sense.

[bmaranville]

There is something funny going on with the resolution smearing... just by looking at it, it doesn't seem like the results make sense for the very last point in test5.txt
I'm investigating right now. Are you treating the dQ as FWHM or as 1-sigma? I think we're using only 1-sigma in refl1d now.

[andyfaff]

1-sigma is used in the validation. I thought test5 was passing. I've seen that last point weirdness before, but it wasn't always there.

[bmaranville]

It is passing - it just looks like the value calculated using refl1d built-in oversampling might be more correct than the accepted answer. Here is a zoomed view of the last point - the red star is the "accepted" value in data6.txt, the blue "+" is the result with manual oversampling you apply after the built-in oversampling fails, while the green point is the "failed" value using refl1d built-in oversampling. The blue line is R for a densely-sampled Q with no convolution.
(dQ = 0.00637 for the point in question)

[andyfaff]

I think I generated the test dataset with a super fine mesh. So the question could be how far an NR smear should go in the tails.
However I've also seen the last point of a refl1d smear be way too high in the past, as in way higher than the previous points.

[bmaranville]

You are correct, of course - with a very fine mesh and manual convolution loop, I get back very nearly the "accepted" R. It never hurts to check! The refl1d oversampling method needs to get another look to find out why it is not generating a sufficient basis even when the number of points is the same as your manual oversampling.

[andyfaff]

Whilst great for avoiding aliasing, surely the random nature of the oversampling means effects like this will be more likely, at least for lower factors of oversampling.

These questions remain:

• what kind of validation tolerances are acceptable? how are they related to the accuracy of any modelling?
• how many sigma should a Gaussian kernel be integrated over?
• are there situations when you need to include tails?
• how does the community advance to a non Gaussian kernel?
• what's the best way of implementing NGK?

[andyfaff]

Typically what kind of probability distribution does a monochromatic instrument have for the wavelength? Uniform/Gaussian/Trapezoidal? What kind of width?

[bmaranville]

I agree that we should address your questions. It is probably a good topic for an ORSO Analysis working group meeting. We have not had enough of those.

To answer some of your questions in short:
Our wavelength is roughly Gaussian, but not with long tails. It is produced by "perfectly imperfect" pyrolytic graphite crystals that are engineered to have a range of crystallite angles around the normal. We probably have some characterization curves from x-rays of the PG material when we bought it. But they certainly only contain a finite range of angles. Our typical monochromator/analyzer crystals are 0.4 to 0.5 deg FWHM. For some instruments we stack (usually 3) crystals with slight offsets to get a total FWHM around 1.2 to 1.5 deg total wavelength spread. In that case the wavelength distribution is roughly trapezoidal. We could probably use triangular or trapezoidal for almost all our resolutions.

When evaluating real convolution implementations for fitting, the tolerances that matter are all relative to the uncertainty in the measured R. E.g. at high Q, your dR/R will be much higher than it is at low Q. We tolerance for real fits is not just ((calculated_convolution - ideal_convolution) / calculated_convolution) but should be (calculated_convolution - ideal_convolution) / (dR).
That way you can specify that you want your oversampling to be high enough that it doesn't ever cause more than e.g. 1% change in R relative dR for (all points?), when compared to an "infinitely oversampled" case.

For non-Gaussian kernels I would think linear segments (piecewise trapezoid) or box-models (pointwise definition) would be pretty straightforward. You have more experience with this that we do - if you can share your particular instrument NGK shape that would be a good way to seed the conversation.

[andyfaff]

We tolerance for real fits is not just ((calculated_convolution - ideal_convolution) / calculated_convolution) but should be (calculated_convolution - ideal_convolution) / (dR). That way you can specify that you want your oversampling to be high enough that it doesn't ever cause more than e.g. 1% change in R relative dR for (all points?), when compared to an "infinitely oversampled" case.

Does this mean that during a fitting process there should be a learning process where the oversampling needs to be readjusted as the difference between calculated and ideal changes?

For the NGK see something like fig6b in this paper (that paper was written whilst on sabbatical at NIST in 2013). The resolution function for our instruments is a convolution of a few Uniform distributions (wavelength) and a Trapezoid (angular). Sometimes the main wavelength contribution (dlambda/lambda = 8) is much broader than the angular part (dtheta/theta = 3.3), which can lead to some funny resolution kernels.
The issue is frequently observed around the critical edge when the wavelength resolution is relaxed (fig7). When the SLD profile is 'diffuse' the signal drops precipitously above Qc, which amplifies the effect presented in that figure.

[bmaranville]

I've been investigating this issue - I'm beginning to lean toward using a simple linspace as our Q-basis for convolution, like you do in the validation code. It seems to be more efficient in several ways, and it's already what we offer for extra oversampling near the critical edge.