docs: add description of data reduction procedure #65

jokasimr · 2024-07-23T11:17:09Z

Fixes #7

This is still a bit coarse, but I think it contains the important bits.

~~I'm marking this as a draft for now~~

jl-wynen · 2024-08-07T14:02:37Z

docs/user-guide/data-reduction.md

+
+The purpose of this text is not to describe how the event coordinates $Q$ and $\theta$ are derived from the raw event data and the geometry information, so for now just take those relations for given.
+
+To avoid overcomplicating the description it is assumed that the sample- and reference measurements were made over the same length of time, and it is assumed the neutron intensity from the source did not vary between the two measurements.


Please explain that this only applies to the equations given here. The actual workflow has no such restriction.

I think the actual workflow does currently have that restriction. Or am I mistaken?

If it does, then it needs to change! But I guess that relates to #78.

Shouldn't this be easy to include in the maths? Isn't it just an extra factor in the normalisation? Something like $I_{\text{sample}} / I_{\text{ref}}$.

If it does, then it needs to change!

Sure, that's what the issue is for. I've put it in the selected column on the dev board now.

Shouldn't this be easy to include in the maths? Isn't it just an extra factor in the normalisation? Something like

Yeah probably, but I don't plan to include that in this PR. This is something we will have to iterate on as new features are added to the workflow.

jl-wynen · 2024-08-07T14:03:57Z

docs/user-guide/data-reduction.md

+The sample reflectivity is related to the intensity of neutron counts in the detector by the model
+
+$$
+I_{sam}(\lambda, j) = F(\theta(\lambda, j, \mu_{sam}), w_{sam}) \cdot R(Q(\lambda, \theta(\lambda, j, \mu_{sam}))) \cdot I_{ideal}(\lambda, j)


Suggested change

I_{sam}(\lambda, j) = F(\theta(\lambda, j, \mu_{sam}), w_{sam}) \cdot R(Q(\lambda, \theta(\lambda, j, \mu_{sam}))) \cdot I_{ideal}(\lambda, j)

I_{\text{sam}}(\lambda, j) = F(\theta(\lambda, j, \mu_{\text{sam}}), w_{\text{sam}}) \cdot R(Q(\lambda, \theta(\lambda, j, \mu_{\text{sam}}))) \cdot I_{\text{ideal}}(\lambda, j)

And everywhere else.

Or use \mathsf instead of \text if you prefer that.

jl-wynen · 2024-08-07T14:07:27Z

docs/user-guide/data-reduction.md

+$$
+I_{sam}(\lambda, j) = F(\theta(\lambda, j, \mu_{sam}), w_{sam}) \cdot R(Q(\lambda, \theta(\lambda, j, \mu_{sam}))) \cdot I_{ideal}(\lambda, j)
+$$ (model)
+where $I_{sam}(\lambda, j)$ represents the number of neutrons detected in the $j$ pixel of the detector having a wavelength in the interval $[\lambda, \lambda + d\lambda]$. $I_{ideal}$ represents the number of neutrons that would have been detected if the sample was a perfect reflector and large enough so that the footprint of the focused beam on the sample was small compared to the sample. $F(\theta, w)$ is the fraction of the focused beam that hits the sample. It depends on the incidence angle $\theta$ and on the size of the sample represented by $w$. $\mu_{sam}$ is the sample rotation.


$\lambda$ would typically be reported as a bin-centre. Here, it is the lower edge. This is in principle fine. But it may lead to confusion between the maths here and actual data.

Good point!

jl-wynen · 2024-08-07T14:09:23Z

docs/user-guide/data-reduction.md

+$$
+I_{sam}(\lambda, j) = F(\theta(\lambda, j, \mu_{sam}), w_{sam}) \cdot R(Q(\lambda, \theta(\lambda, j, \mu_{sam}))) \cdot I_{ideal}(\lambda, j)
+$$ (model)
+where $I_{sam}(\lambda, j)$ represents the number of neutrons detected in the $j$ pixel of the detector having a wavelength in the interval $[\lambda, \lambda + d\lambda]$. $I_{ideal}$ represents the number of neutrons that would have been detected if the sample was a perfect reflector and large enough so that the footprint of the focused beam on the sample was small compared to the sample. $F(\theta, w)$ is the fraction of the focused beam that hits the sample. It depends on the incidence angle $\theta$ and on the size of the sample represented by $w$. $\mu_{sam}$ is the sample rotation.


Missing definition of $R$

Missing definition of $Q$

Is $\mu_\text{sam}$ really the sample rotation? What about $\omega$?

$R$ is reflectivity, $Q$ momentum transfer and yes $\mu$ is the sample rotation. I'll add the definitions 👍

Why did you ask about $/mu$ specifically?

The definition for R, and Q are in the first paragraph of the page.

jl-wynen · 2024-08-07T14:11:24Z

docs/user-guide/data-reduction.md

+$$
+I_{sam}(\lambda, j) = F(\theta(\lambda, j, \mu_{sam}), w_{sam}) \cdot R(Q(\lambda, \theta(\lambda, j, \mu_{sam}))) \cdot I_{ideal}(\lambda, j)
+$$ (model)
+where $I_{sam}(\lambda, j)$ represents the number of neutrons detected in the $j$ pixel of the detector having a wavelength in the interval $[\lambda, \lambda + d\lambda]$. $I_{ideal}$ represents the number of neutrons that would have been detected if the sample was a perfect reflector and large enough so that the footprint of the focused beam on the sample was small compared to the sample. $F(\theta, w)$ is the fraction of the focused beam that hits the sample. It depends on the incidence angle $\theta$ and on the size of the sample represented by $w$. $\mu_{sam}$ is the sample rotation.


A drawing of the sample with all angles would help a lot. Older versions of the docs used figure 5 of https://doi.org/10.1016/j.nima.2016.03.007

I'll save this for the PR with documentation about the coordinate transformations.

jl-wynen · 2024-08-07T14:21:07Z

docs/user-guide/data-reduction.md

+$$
+
+## Estimating intensities from detector counts
+The neutron counts are Poisson distributed.


This is true for counts in a 'spatial' binning. E.g., in pixel-tof-space. But is this also true after transforming to Q? It seems odd to me that we would get a Poisson distribution in Q bins. If this was true, we would not need to do any error propagation at all before we compute a Q histogram.

The reason is that binning in Q combines events from multiple pixel-tof bins in a non-trivial way.

Concerning your equations, $\sum_k$ is the expectation value. So you don't need an extra $E[...]$. Unless I misunderstand something?

It seems odd to me that we would get a Poisson distribution in Q bins. If this was true, we would not need to do any error propagation at all before we compute a Q histogram.

Yes my understanding is that the event count is Poisson distributed also in Q bins. The only assumptions that makes the count Poisson distributed are:

Constant (in time) expected event rate.

Events arrive independently from each other.

Those assumptions are independent of if we bin in a spatial binning or wavelength or $Q$.

If this was true, we would not need to do any error propagation at all before we compute a Q histogram.

Yes and we typically don't right?

But I see what you mean. There's something a bit wrong here. The events are weighted. So one event is actually not a single events, but a "package" of events. Individual packages of events arrive independently, but inside a package the events don't arrive independently.
I think we should talk about this in person to get the formulation right here. 👍

Concerning your equations, is the expectation value. So you don't need an extra . Unless I misunderstand something?

The event count (the sum) is a random variable. We are interested in the expected value of that random variable because that's what's associated with the properties of the sample.

The event count (the sum) is a random variable. We are interested in the expected value of that random variable because that's what's associated with the properties of the sample.

Yes, the count is the random variable. And the events are samples. So the sum is the sample mean, up to a normalisation factor. And thus, it is an estimator of the expectation value of the count. Basically,

$$ E[\sum_k x_k] = \sum_k E[x_k] = \sum_k w_k $$

where $w_k$ s the weight of the events.

jl-wynen · 2024-08-07T14:27:38Z

docs/user-guide/data-reduction.md

+
+Therefore there's one more approximation that is used to further reduce the cost of computing the reference intensity.
+
+The Amor detector has three logical dimensions, `blade`, `wire` and `stripe`. It happens to be the case that $\theta(\lambda, j)$ is almost the same for all $j$ belonging to the same `stripe` of the detector.


Everything has been generic up to here. Please clearly delineate generic reflectometry and instrument-specific explanations.

jl-wynen · 2024-08-07T14:28:35Z

docs/user-guide/data-reduction.md

+The detector pixel positions are known and so is the position of the sample and the orientation of the sample.
+From that information we can compute the reflection angle (assumed equal to the incidence angle) $\theta$, and the momentum transfer $Q$ caused by the interaction with the sample.
+
+The purpose of this text is not to describe how the event coordinates $Q$ and $\theta$ are derived from the raw event data and the geometry information, so for now just take those relations for given.


If you say 'for now', I would expect a follow up that explains $Q$ and $\theta$.

I was planning to add that in a separate PR, together with the other coordinate transformations.

jl-wynen · 2024-08-07T14:30:24Z

docs/user-guide/data-reduction.md

This document only describes the basic procedure. It does not take instrument / facility specific details into account. (E.g., choppers or monitors) Please make it clear that the procedure outline here is not a complete reduction workflow.

Choppers and monitors are only relevant to compute $\lambda$. That's also something planned for a separate PR.

jokasimr · 2024-09-03T10:27:38Z

To avoid making this PR too large. I'll add separate issues for documenting:

the frame unwrapping / how tof is computed
coordinate transformations
footprint correction procedure
how the resolution function is estimated

jl-wynen · 2024-09-03T12:41:03Z

In general, splitting PRs into small units is good. But that only works when those units are self-contained. Here, I cannot judge whether the documentation is good without seeing all of it. In the last PR of this series, I would have to read everything again anyway. So can you please combine them into one PR?

jokasimr · 2024-09-03T13:01:05Z

The risk I see not splitting this effort into manageable pieces is that

It will never get finished, and never reviewed, because it will become to cumbersome. It's already quite a mouthful.
It will get out of sync with the code.
No one will have any use of it before everything is done.

I don't agree that the pieces are not self contained.
You don't need to know the geometry of the beamline, the chopper setup etc, to follow what is done here.
This part of the documentation assumes there exists some procedure to compute $\lambda, \theta$ etc but doesn't specify how that is done, and I think that is fine.

The same goes for the parts about footprint correction and the resolution function.

jl-wynen · 2024-09-20T07:14:11Z

If this was true, we would not need to do any error propagation at all before we compute a Q histogram.

Yes and we typically don't right?

But I see what you mean. There's something a bit wrong here. The events are weighted. So one > event is actually not a single events, but a "package" of events. Individual packages of events arrive independently, but inside a package the events don't arrive independently.
I think we should talk about this in person to get the formulation right here. 👍

We should

jokasimr marked this pull request as draft July 23, 2024 11:27

jl-wynen reviewed Aug 7, 2024

View reviewed changes

jokasimr marked this pull request as ready for review September 3, 2024 10:27

jokasimr requested a review from jl-wynen September 3, 2024 10:39

jokasimr added 2 commits September 4, 2024 12:54

docs: add description of data reduction procedure

426313b

fixup

c2cfd2b

jokasimr force-pushed the improve-docs branch from 7ded7c9 to c2cfd2b Compare September 4, 2024 10:54

reference

979c1fe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add description of data reduction procedure #65

docs: add description of data reduction procedure #65

jokasimr commented Jul 23, 2024 •

edited

Loading

jl-wynen Aug 7, 2024

jokasimr Aug 8, 2024 •

edited

Loading

jl-wynen Sep 20, 2024

jokasimr Sep 27, 2024

jokasimr Sep 27, 2024

jl-wynen Aug 7, 2024

jl-wynen Aug 7, 2024

jokasimr Aug 8, 2024

jl-wynen Aug 7, 2024

jokasimr Aug 8, 2024

jokasimr Sep 3, 2024

jl-wynen Aug 7, 2024

jokasimr Sep 3, 2024

jl-wynen Aug 7, 2024

jokasimr Aug 8, 2024 •

edited

Loading

jl-wynen Sep 20, 2024

jl-wynen Aug 7, 2024

jl-wynen Aug 7, 2024

jokasimr Sep 3, 2024 •

edited

Loading

jl-wynen Aug 7, 2024

jokasimr Sep 3, 2024

jokasimr commented Sep 3, 2024

jl-wynen commented Sep 3, 2024

jokasimr commented Sep 3, 2024 •

edited

Loading

jl-wynen commented Sep 20, 2024


		The purpose of this text is not to describe how the event coordinates $Q$ and $\theta$ are derived from the raw event data and the geometry information, so for now just take those relations for given.

		To avoid overcomplicating the description it is assumed that the sample- and reference measurements were made over the same length of time, and it is assumed the neutron intensity from the source did not vary between the two measurements.

	I_{sam}(\lambda, j) = F(\theta(\lambda, j, \mu_{sam}), w_{sam}) \cdot R(Q(\lambda, \theta(\lambda, j, \mu_{sam}))) \cdot I_{ideal}(\lambda, j)
	I_{\text{sam}}(\lambda, j) = F(\theta(\lambda, j, \mu_{\text{sam}}), w_{\text{sam}}) \cdot R(Q(\lambda, \theta(\lambda, j, \mu_{\text{sam}}))) \cdot I_{\text{ideal}}(\lambda, j)


		Therefore there's one more approximation that is used to further reduce the cost of computing the reference intensity.

		The Amor detector has three logical dimensions, `blade`, `wire` and `stripe`. It happens to be the case that $\theta(\lambda, j)$ is almost the same for all $j$ belonging to the same `stripe` of the detector.

docs: add description of data reduction procedure #65

Are you sure you want to change the base?

docs: add description of data reduction procedure #65

Conversation

jokasimr commented Jul 23, 2024 • edited Loading

Choose a reason for hiding this comment

jokasimr Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jokasimr Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jokasimr Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jokasimr commented Sep 3, 2024

jl-wynen commented Sep 3, 2024

jokasimr commented Sep 3, 2024 • edited Loading

jl-wynen commented Sep 20, 2024

jokasimr commented Jul 23, 2024 •

edited

Loading

jokasimr Aug 8, 2024 •

edited

Loading

jokasimr Aug 8, 2024 •

edited

Loading

jokasimr Sep 3, 2024 •

edited

Loading

jokasimr commented Sep 3, 2024 •

edited

Loading