Adding AZSS correction per subscan #1065

susannaaz · 2024-12-10T21:37:38Z

Addresses remaining part of issue #1009, i.e. adding AZSS correction per subscan, as a continuation of the PR in #1064.

…mapmaking

…pdated noise logic to incorporate wrap_name from config\n\- added handling for 'fknee' and 'min_noise' in noise process based on user-defined thresholds

…stimation from get_azss with flag subscan, TODO: azss subtraction per subscan still problematic

msilvafe

Some inline comments.

sotodlib/preprocess/processes.py

msilvafe · 2024-12-18T22:02:47Z

sotodlib/preprocess/processes.py

+        proc_aman.flags.wrap("left_scan", aman.flags.left_scan)
+        proc_aman.flags.wrap("right_scan", aman.flags.right_scan)


These have to exist in the axis manager first which means that FlagTurnarounds must be run first with the argument merge_lr = True which is the default but can be overwritten. Not sure what the right path is here, perhaps if those don't exist then look in proc_aman for them and if they're not in either place then run the the get_turnaround_flags fn w/ merge_lr = True.

msilvafe · 2024-12-19T12:49:35Z

sotodlib/preprocess/processes.py

+    def calc_and_save(self, aman, proc_aman):
+        if 'flags' not in proc_aman._fields:
+            from sotodlib.core import FlagManager
+            proc_aman.wrap('flags', FlagManager.for_tod(proc_aman))


Typically we don't wrap into proc_aman directly in calc_and_save instead we make one new AxisManager here (say bss_aman--bad sub scan aman), wrap everything into that, and then pass that to save (in the field typically called calc_aman) and the save method wraps that into proc_aman so that everything for this one preprocess step is in a single field in proc_aman. Here you wrap into proc_aman directly and also wrap ss_aman, and det_aman all into separate fields of proc_aman. Let's put everything in one aman and send just that one to save.

sotodlib/preprocess/processes.py

msilvafe · 2024-12-19T14:06:25Z

sotodlib/preprocess/processes.py

+        subscan_stats_T = proc_aman[self.stats_name+"_T"]
+        subscan_stats_Q = proc_aman[self.stats_name+"_Q"]
+        subscan_stats_U = proc_aman[self.stats_name+"_U"]


So this requires that you run GetStats three times wrapping in the same name but appending _T, _Q, and _U to the end? That's a pretty specific prerequisite. Plus I think 90% of the time we are running GetStats it'll be to achieve what BadSubscanFlags is achieving (probably the other 10% of the time is getting some stats on the full TOD or PSD I guess). So why don't we instead check if self.stats_name+"_T" exists and if not call tod_ops.flags.get_stats to get the info you need.

sotodlib/tod_ops/flags.py

msilvafe · 2024-12-19T14:29:26Z

sotodlib/tod_ops/flags.py

@@ -998,3 +1021,73 @@ def noise_fit_flags(aman, low_wn, high_wn, high_fk):
        return flag_valid_fk
    else:
        return None
+
+def get_noisy_subscan_flags(aman, subscan_stats_T, subscan_stats_Q, subscan_stats_U,


Making this require T, Q, and U simultaneously makes this a very restrictive function (i.e. can only be run for the SATs not LAT and only after demodulation). Maybe we can make this take just one signal (i.e. signal = subscan_stats_T) and produce one dets x subscan boolean mask/RangesMatrix per signal then the preprocess method could choose how to combine these i.e. if there's one use 1, if there's >1 use logical and or logical or, etc. Thoughts?

chervias · 2024-12-19T14:32:03Z

sotodlib/tod_ops/flags.py

+            raise ValueError(f"Flag name {name} already exists in aman.flags")
+        aman.flags.wrap(name, noisy_subscan_flags)
+
+    return noisy_subscan_flags, noisy_detector_flags


Fix newline

sotodlib/preprocess/processes.py

earosenberg · 2024-12-19T14:14:32Z

sotodlib/preprocess/processes.py

+    def process(self, aman, proc_aman):
+        subtract = self.process_cfgs.get('subtract', False)
+        if self.proc_aman_turnaround_info:
+            _f = attergetter(self.proc_aman_turnaround_info)


typo: attrgetter

earosenberg · 2024-12-19T14:19:45Z

sotodlib/preprocess/processes.py

+        if 'flags' not in proc_aman._fields:
+            from sotodlib.core import FlagManager
+            proc_aman.wrap('flags', FlagManager.for_tod(proc_aman))
+        proc_aman.flags.wrap("left_scan", aman.flags.left_scan)


left_scan and right_scan are already in the proc_aman after FlagTurnarounds, just top-level not under "flags". We probably want to agree on one place for them and just look there

Nominally we shouldn't be wrapping anything into aman unless it's explicitly intended to be done in a process step and then merged into proc_aman in the calc_and_save step. However the wrapping into aman is happening in calc_and_save for FlagTurnarounds which should not be allowed. Long story short we should have all of the merge options hard coded false in the call to get_turnaround_flags in calc_and_save and any functions that look for info from TurnaroundFlags should look for it in proc_aman.

sotodlib/tod_ops/azss.py

earosenberg · 2024-12-19T14:27:19Z

sotodlib/tod_ops/azss.py

+            right_mask = turnaround_info.right_scan
+        else:
+            print("")
+            print("HEREEEEEEE")


Leftover debug statement

mmccrackan

Some minor comments/questions on formatting and organization.

mmccrackan · 2024-12-19T15:21:22Z

sotodlib/preprocess/processes.py

+            tod_ops.azss.get_azss(aman, azss_stats_name=self.azss_stats_name,
+                                  turnaround_info=turnaround_info,
+                                  merge_stats = True, merge_model=False,
+                                  subtract_in_place=subtract, **self.calc_cfgs)


This probably has already been discussed so feel free to ignore this, but this is using calc_cfgs within process which seems a bit counter-intuitive. I think it might be unavoidable given the way this needs to be run, but maybe these would make more sense as process_cfgs since process is always going to be run and will run first.

mmccrackan · 2024-12-19T15:35:06Z

sotodlib/tod_ops/azss.py

@@ -313,15 +435,31 @@ def subtract_azss(aman, signal='signal', azss_template_name='azss_model',
    else:
        raise TypeError("Signal must be None, str, or ndarray")

+    azss_stats, model_left, model_right = get_model_sig_tod(aman, azss_stats, az=None)
+
    if in_place:


Optional, but I think this logic can be simplified by assigning to a temporary variable, i.e.:

if in_place: if signal_name is None: sig = signal else: sig = aman[signal_name] else: if signal_name is None: sig = signal.copy() else: sig = aman[signal_name].copy()

As long as you use sig -= mask, the non-copy ones should stay as references.

mmccrackan · 2024-12-19T15:45:19Z

sotodlib/preprocess/processes.py

+        if self.azss_stats_name in proc_aman:
+            if subtract:
+                tod_ops.azss.subtract_azss(aman, proc_aman[self.azss_stats_name],
+                                           signal = self.calc_cfgs.get('signal'),


Very minor formatting on signal = self.calc_cfgs.get('signal') (space not needed).

sotodlib/tod_ops/azss.py

mmccrackan · 2024-12-19T15:56:09Z

sotodlib/preprocess/processes.py

+        else:
+            tod_ops.azss.get_azss(aman, azss_stats_name=self.azss_stats_name,
+                                  turnaround_info=turnaround_info,
+                                  merge_stats = True, merge_model=False,


Do we want merge_model hardcoded to False here (or elsewhere)? Is there any case where we'd want to be able to look at the model after fitting or interpolation?

Yes because it wraps in a n_dets x n_samps copy of the data, we can use get_model_sig_tod to reconstruct it from the info in azss_stats if we need to look at it later.

mmccrackan · 2024-12-19T15:57:36Z

sotodlib/preprocess/processes.py

+        else:
+            tod_ops.azss.get_azss(aman, azss_stats_name=self.azss_stats_name,
+                                  turnaround_info=turnaround_info,
+                                  merge_stats = True, merge_model=False,


Minor formatting with unneeded spaces on merge_stats.

susannaaz added 4 commits December 9, 2024 15:48

align with master

1a129de

cleanup

f211e88

Merge remote-tracking branch 'upstream/master' into 20241209_preproc_…

166aa3c

…mapmaking

temporarily taken away azss per subscan, do in another PR

5b5af2d

susannaaz marked this pull request as draft December 10, 2024 21:37

susannaaz and others added 14 commits December 10, 2024 17:40

cleanup functions

0c59a8d

eliminate duplicate functions in filters

25be19f

rewrite docstring

5f2e13e

cleaned darkdets function

7a73c18

minor cleanup functions and docstrings

15053cf

correct spacings

d1f3737

rm bad subscans flagging

138cb05

some minor format fixes

1ee65ec

some more minor format fixes

8dac03d

Merge branch 'master' into 20241209_preproc_mapmaking

bf0abb7

fix focalplane nan flagging

e8d4874

- refactor common mode deprojection to simultaneously treat Q/U\n\- u…

907289c

…pdated noise logic to incorporate wrap_name from config\n\- added handling for 'fknee' and 'min_noise' in noise process based on user-defined thresholds

Merge remote-tracking branch 'origin' into 20241209_preproc_mapmaking

1e07280

align init with deprojection functions

6d888d9

susannaaz force-pushed the 20241210_preproc_mapmaking_azsssubscan branch from 71891d7 to 6d888d9 Compare December 16, 2024 18:04

interfaced noisy subscans process with get stats, call subscan azss e…

539f72d

…stimation from get_azss with flag subscan, TODO: azss subtraction per subscan still problematic

susannaaz requested a review from earosenberg December 16, 2024 18:18

earosenberg and others added 3 commits December 16, 2024 13:11

azss: try to fix IndexError

b0cae07

function to flag noisy subscans

9a318b8

Pull out azss_model generation fn.

613323c

msilvafe mentioned this pull request Dec 16, 2024

Pull out azss_model generation fn. #1068

Closed

msilvafe added 4 commits December 16, 2024 14:52

Pull out azss_stats prep and left/right handling.

a85e273

Finish up tod_ops.azss

29582fe

Update process method.

c46c7e3

Fixed bugs.

b7ef718

msilvafe added 3 commits December 16, 2024 21:30

Merge branch 'azss_update' into 20241210_preproc_mapmaking_azsssubscan

7a7d6be

Fix broken test.

a6443e4

Address failing test.

0e546fc

msilvafe assigned msilvafe and unassigned msilvafe Dec 17, 2024

msilvafe and others added 2 commits December 16, 2024 22:32

Fix test_azss.py failures.

a42869a

Use subscan_flags and RangesMatrix ops in BadSubscans

75bf0d5

msilvafe self-assigned this Dec 17, 2024

susannaaz assigned susannaaz and msilvafe and unassigned msilvafe Dec 17, 2024

susannaaz requested review from msilvafe and mmccrackan December 17, 2024 18:45

susannaaz added 2 commits December 17, 2024 16:06

optional noise flags for noisy subscans

9a3b564

azss tested with bad subscans flags included

aa22a21

susannaaz marked this pull request as ready for review December 18, 2024 21:00

susannaaz requested a review from chervias December 18, 2024 21:46

msilvafe requested changes Dec 19, 2024

View reviewed changes

chervias reviewed Dec 19, 2024

View reviewed changes

earosenberg reviewed Dec 19, 2024

View reviewed changes

mmccrackan requested changes Dec 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding AZSS correction per subscan #1065

Adding AZSS correction per subscan #1065

susannaaz commented Dec 10, 2024

msilvafe left a comment

msilvafe Dec 18, 2024

msilvafe Dec 19, 2024

msilvafe Dec 19, 2024

msilvafe Dec 19, 2024

chervias Dec 19, 2024

earosenberg Dec 19, 2024

susannaaz Dec 19, 2024

earosenberg Dec 19, 2024

msilvafe Dec 20, 2024

earosenberg Dec 19, 2024

mmccrackan left a comment

mmccrackan Dec 19, 2024

mmccrackan Dec 19, 2024

mmccrackan Dec 19, 2024

mmccrackan Dec 19, 2024

msilvafe Dec 20, 2024

mmccrackan Dec 19, 2024

		proc_aman.flags.wrap("left_scan", aman.flags.left_scan)
		proc_aman.flags.wrap("right_scan", aman.flags.right_scan)

Adding AZSS correction per subscan #1065

Are you sure you want to change the base?

Adding AZSS correction per subscan #1065

Conversation

susannaaz commented Dec 10, 2024

msilvafe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmccrackan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment