Threshold TSO data by variance-computed weights in outlier detection

[stscijgbot-jp]

Issue JP-3779 was created on JIRA by Ned Molter:

This ticket corresponds to a comment I found in the outlier detection TSO code.  It reads:

    Prior to PR #⁠8473, the build_driz_weight function was used to
    create the weights for the input models for TSO data. However, that
    function was simply returning a copy of the DQ array because the
    var_noise was not being passed in by calwebb_tso3. As of PR #⁠8473,
    a cube model that includes the var_noise is passed into TSO
    outlier detection, so build_driz_weight would weight the cube model
    by the variance. Therefore build_driz_weight was removed in order to
    preserve the original behavior. If it is determined later that exposure
    time or inverse variance weighting should be used here, build_driz_weight
    should be re-implemented.

I have no idea whether or not the instrument teams want to implement this, i.e., if weighting based on the var_noise is desired for TSO data.  However, the way the code is currently written, the maskpt parameter is useless because it sees a weight array that is all either 0 or 1.  It would be helpful for clarity of both documentation and code to fix that (see this [GitHub issue|[https://github.com//issues/8879]).]  And it would be nice to remove the long comment.

 

[stscijgbot-jp]

Comment by David Law on JIRA:

Tagging Ian Wong ; any thoughts?

[stscijgbot-jp]

Comment by Ian Wong on JIRA:

For TSOs, my understanding is that outlier_detection does not resample the integrations. So in this sense, having the variance included would be to allow for filtering of the points that are included in the moving median across the integration axis and for the snr filtering. Is that correct?

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

Of course it could theoretically work however you / the instrument teams think is best.  But yes I think your comment nicely summarizes what the comment is trying to say.  If we were to put build_driz_weight back in, this would compute the weights on un-flagged data based on either the inverse variance or the measurement time.  Then the weights would be compared to a cutoff value, and the low-weight points would be excluded from the rolling median calculation.

 

Here's the source code of build_driz_weight if you are interested in that: https://github.com/spacetelescope/jwst/blob/main/jwst/resample/resample_utils.py#L182 

[stscijgbot-jp]

Comment by David Law on JIRA:

It's unclear to me whether we necessarily would want to use var_noise weights to determine which pixels to reject from the moving median, but it seems like it would be useful to have the plumbing hooked up regardless.

One note of caution though- I believe maskpt isn't specified in param ref files (at least for MIRI), meaning that it'll revert to the code default of 0.7, meaning that we might get quite different results from this step once things are hooked up to pass weights through.  We should ensure that this doesn't result in any surprises by keeping the current 0/1 weighting unless something else is specified.

[stscijgbot-jp]

Comment by Ian Wong on JIRA:

Re: maskpt

I agree that keeping the 0/1 weighting for now is a safe option while we test the functionality of including var_noise. I do find that in some cases, there are pixel values (often clusters of them) in the time series that have large uncertainties, presumably unflagged hot pixels or some other detector effect. I would like to play around with the outlier detection once the var_noise weights are included to see if improvements to light curve quality can be achieved with some default maskpt setting.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

 the functions to do this already exist and are used in outlier detection for other modes, so with regards to the plumbing being hooked up, I don't think there are really any useful changes to make that do not change the current behavior.

It sounds to me like we should not make any changes on the pipeline side at this time, and re-assign this ticket to someone on an instrument team to test whether these changes would improve outlier detection.  What do you think Ian Wong David Law Tyler Pauly ?

[stscijgbot-jp]

Comment by Ian Wong on JIRA:

I could definitely lead testing on the MIRI LRS/imaging TSO side

[stscijgbot-jp]

Comment by David Law on JIRA:

Sounds good to me, thanks Ian Wong !

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

We may want to alert the other teams as well - presumably each instrument will need to make this assessment (NRC_TSGRISM/IMAGE, NIS_SOSS, NRS_BOTS in addition to Ian/MIRI).