tod_ops: gap-filling #81
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces simple polynomial (linear, really) gap filling, as well as "model-based" gap-filling that can use the results of a PCA (or common mode). The Extract and ExtractMatrix classes (the later of which is not quite compatible with storage in AxisManager, but could be moved in that direction) help with storing sparse subsets of a TOD signal efficiently.
|
If there are no volunteers to review this, I will merge it in 24 hours. |
tskisner
approved these changes
Jul 20, 2020
| """Computes samples to fill the gaps in data identified by flags. | ||
| Each flagged segment is modeled with a polynomial of the order | ||
| specified, based on up to nbuf points on each side of the segment. | ||
|
|
There was a problem hiding this comment.
General questions about the use case of this gap-filling: If you apply this kind of gap-filling technique to data which contains signal + noise, it will put some amount of fake signal in the gap which could then leak back into unflagged samples when applying a filter. The only kind of gap-filling I have experience with is the case of first estimating a noise timestream (by subtracting a signal estimate) and then using that to build a constrained noise realization across all gaps simultaneously (to ensure that long-timescale correlations are included).
Of course if your timestream is completely noise dominated, then you can skip the subtraction of the signal estimate. So are you just treating atmosphere as "noise" in this case?
Even in this noise dominated regime, are you not concerned about gap-filling individual gaps versus all of them simultaneously for the entire CES?
There was a problem hiding this comment.
I agree that, because of filtering, filling a single sample potentially has long-range consequences. The ACT map-maker (IIRC) actually solves for the value of flagged samples, so the gap-filled values are just starting points in the iterative solution.
However, the gap-filled data are used to estimate a noise model for the max-L solution. As you imply, the thing to worry about is the atmosphere; it is 1/f "noise" with strong correlations between detectors. We've actually seen that naive gap-filling can lead to noise models that have higher map noise, i.e. are highly sub-optimal, especially when the data contain gaps >= 1/f_knee. Our modern approach does two-passes -- at first you just fill single time-stream gaps, then you do a PCA to constrain the low-frequency correlated modes and use that for a superior gap fill. Then you estimate a (correlated) noise model.
But I agree with your statements (especially below) that some care should be taken with mapping of gap-filled time-streams. The present routines will get us through basic troubles such as "I can't plot or fft this timestream because there are huge glitches", and similar.
|
One overview question is whether we can gap-fill with some functions other than polynomials. I am asking since if we have the SAT time streams, which are modulated, it may be great if we can gap-fill with a sinusoidal function, matching the overall tend (frequency, phase, amplitude) of the unflagged data. |
The SAT timestreams will have the HWPSS in them, which for the most part won't be sinusoidal? They will have shapes depending on the position across the focal plane right? |
|
I guess my point is, "what are you doing with the gap filled data"? If you are just making a plot or a binned map, or something then that is fine. If you are applying a filter to the gap-filled timestream, then any signal in the gap will impact nearby samples, and one would need to expand the flagged region afterwards to avoid those contaminated regions. |
You are probably right, I just naively guessed that it will be a sinusoidal function... I think it will probably depend on the position of the focal plane. To answer @tskisner 's question, I was more thinking along the line of the ringing effects from FFT if the boundaries between the gap-filling part and the original data are not smooth. |
I think in general the only safe thing to do with non-trivial structured signal (whether it's atmosphere or HWPSS) is to model it as best you can using unflagged samples (and trivially gap-filled flagged samples if need be). Then subtract that first model estimate from the original data; gap fill those residuals; and add back in the model (if you want it to look like the original signal). Then iterate -- in cases where where your model fit itself requires some gap-filling. |
keskitalo
pushed a commit
that referenced
this pull request
Jan 28, 2021
Introduces simple polynomial (linear, really) gap filling, as well as "model-based" gap-filling that can use the results of a PCA (or common mode). The Extract and ExtractMatrix classes (the later of which is not quite compatible with storage in AxisManager, but could be moved in that direction) help with storing sparse subsets of a TOD signal efficiently.
keskitalo
added a commit
that referenced
this pull request
Feb 2, 2021
* add the new SSO simulation operator to sotodlib.observe * Introduce core submodule, and core.AxisManager * data/load_observation: return an AxisManager * Changes to AxisManager... won't be the last... * metadata module: for now this just imports everything from sotoddb * Context: support for loading site/user/dataset-specific settings * axisman: Bridge missing intersect1d feature in numpy < 1.15 Also fix a data scrambling bug, in passing. Also catch bad (non-disjoint) requests in from_resultset. * Context.load_obs(obs_id) To get that working, added string substitution (from context "tags"), function to trigger database (re)load. obs_id strings can include detector identifiers such as 'f150'; such tags may be drawn from detdb fields listed in context['obs_colon_tags']. * data.load.load_observation: load directly into an AxisManager * OBSLOADER_REGISTRY tracks named observation loading functions And you tell it what loader you need through the context key "obs_loader_type". Both 'default' and 'pipe-s0001' point to data.load.load_observation. For now. * Context.get_meta, can operate independently of get_obs Also fix a couple of other bugs in passing * data.load.load_file, as alternative load_observation It can be used in cases where an ObsFileDb is not available, for example. * Context: load obsdb * Context: obsdb / detdb / obsfiledb load is optional * Whitespace fixes; no functional change * add simple map binning pipeline to visualize so3g data dumps * pass sinc-modulation argument to todground * Modify matplotlib backend to PDF inside plot function Closes #32 * docs: add docstring to set matplotlib backend function * Invert phi to get azimuth in export * Add pipeline for measuring the filter&bin transfer function * tweak the transfer function pipeline * The transfer function pipeline can now run with either Madam or the TOAST mapmaker * support cosecant modulation in SO sims * load_observation: permit user to request dets=[] Sometimes you just want to look at the pointing. This is not terribly time efficient, since the whole file has to be read regardless. But we could imagine superior data formats where the ancillary data is stored separately from the bolometers. * Rearrange existing files to match (as much as possible) the proposed layout in #53. Only delete files that were obviously empty or never used. No changes or cleanup to the code here, just moving things around and changing imports so that unit tests pass. * Fix imports in pipelines * Move context and g3_core into core * Adjust imports of sotoddb classes into new structure This imports ResultSet and the metadata *Db directly to core.metadata, renaming the *DB classes as *Db. The more stuff-on-disk code in sotodlib.simple is imported as io.metadata.simple. This has been tested on simons1 with the SO sim and ACT Uranus demo code. * Fix imports so all tests pass This includes a bug-fix in axisman that would only activate with old numpy. * Fix stray bad import * fix pipeline_tools import * fix small issues in TOAST load/export unit tests * fix broken imports (#67) * fix broken imports * use relative imports * more relative imports * Trivial fix to unit tests * Modifications to allow (#68) * Modifications to allow * Simulating atmosphere on top of loaded signal * writing TOD from a pipeline that begins with loading TOD * fix export when loading from file and using an export key * Time constant (#52) * Initial implementation of the operator and pipeline tools. Thanks to Katie Harrington for the code. * unit test now passes * fix name errors * add missing import * add unit test file * Add units [s] to time constant doc strings * Add parameters to leave a time constant residual * fix time constant unit test * Improve unit tests * move time_constant.py * fix import; add __init__.py * more import fixes * Fully functional time constant operator * fix issue in pipeline_tools; correct the order of time constant and noise simulation * remove obsolete directory * add FlagManager to core imports * add functionality to FlagManager * grammar * initial commits of TOD operations * automatically add FlagManager to tod * add 2 working flags * add __setitem__ for fields only * add simulation mechanim for glitches * fix logic error on intersects * flag updates & fixes * expand the comments! * updates for review * Fix trivial typo * use the new TOAST el-nod facility (#73) * docs: update module paths in API reference * context: fix a critical default behavior get_obs() should refer to obsfiledb if user does not specify detsets. Otherwise, all dets in the detdb will be try-loaded, and that's often too many. * context: accept a dict as obs_id argument to get_obs This is so that a single row from obsdb.get() can be used with convenience. * Context: get_obs accepts a list of ints for detsets Also get_meta is documented and accepts a simple string obs_id. * add copy constructor and update __repr__ to work for inheritance * sotodlib.coords: pointing and coordinate support Integrates so3g accelerated pointing code with tod-in-AxisManager. * fix error in thinning the focal plane for the second band * Make sure the observations get independent instances of the noise PSD (#75) * Run unit tests with github actions. This work: - Disables travis tests - Adds a simple github workflow that: - pulls the latest so3g image - pip-installs dependencies into the image and commits it to a local temporary image - runs the unit tests in this temporary image * Change name of test workflow. * Update test badge * tod_ops.filters: add docstr and signature for fft_filter decorator * Import of sotoddb code -- from sotodlib commit f4ee4167 docs and tests are still to come. * metadata: fix SuperLoader Hdf5Loader issue * metadata: rename DB to Db in main class definitions This doesn't change the external interface. * metadata: rename proddb.py to manifest.py * metadata: import tests from sotoddb * docs: in passing, re-fix name errors for sim/vis_hardware * metadata: import sotoddb docs * tod_ops: PCA module Provides a few functions for decomposing tod covariance into eigenbasis; reconstructing limited mode sets for signal cleaning; and cleaning (subtraction of dominant modes). Also an independent function to find and remove "trends", and return them like they were an set of modes with couplings to the signal. * tod_ops: basic docs for filters and pca. * docs: mock 'yaml', 'spt3g', 'pyfftw' if not found. * docs: mock 'toml', 'sqlite3', 'so3g', 'toast', 'pixell' if not found. * docs: rebrand SOTODLib -> sotodlib; set some autodoc options * tod_ops: gap-filling (#81) Introduces simple polynomial (linear, really) gap filling, as well as "model-based" gap-filling that can use the results of a PCA (or common mode). The Extract and ExtractMatrix classes (the later of which is not quite compatible with storage in AxisManager, but could be moved in that direction) help with storing sparse subsets of a TOD signal efficiently. * core.metadata: Clean up ObsDb code; document it. This includes a minimal API for creating a usable ObsDb. Common database i/o functions created in core/metadata/common.py; other Dbs will need to start using those. * core.metadata: add "tags" functionality to ObsDb; unittest for ObsDb Other minor cleanup as well. * context: remove use of contextlib.nullcontext, since that needs >=3.7 * context: modernize ManifestDb and ObsFileDb with common to_file, from_file * context: raise informative errors when failing to broadcast/match metadata * tests/test_metadata.py: tests for the metadata resolution system The test is especially needed now, since this system works but I want to refactor the HDF5 stuff and still have it work the same... * metadata: ManifestDb raises informative error when too many matches * metadata: ResultSet.asarray has hdf_compat option And that option causes strings to be coded as byte strings in the output array, so the result can be written as an HDF5 dataset. * io.metadata: ResultSetHdfLoader and write_dataset PerDetectorHdf5 is now deprecated, but left in place as a shell (probably not for long). The ResultSetHdfLoader is tuned for the context/metadata system, with a single API (the one hacked in through PerDetectorHdf5.loader_class). For writing datasets, a function makes more sense than having a write_dataset method, assuming we don't want to put HDF5 i/o right in the ResultSet class. Also added a test for the array string conversion function. * context/metadata: remove PerDetectorHdf5; ResultSetHdf is now default This still requires a Context user to import sotodlib.io.metadata before trying metdata loads. * context: remove abandoned code in ManifestDb * context: the Context can include a list of "imports" This is useful to cause registration of data and metadata loaders for Context to use. * WIP: updates to context documentation. * Avoid slow slice operation when merging axis managers * Simplifiy and slightly generalize simplify_slice. It can now simplify along each dimension independently. * Avoid confusing situations where attempted updates to members of the axismanager instead end up creating an entry in the axismanager object itself, that masks the actually managed one * Fixed oversimplification of slice in simplify_slice. * context: get_obs has a "logic_only" arg And it causes the function to return the load specification (including what dets) without loading any data or metadata. * context/metadata: better handling of obs_colon_tags Raises errors if parent field not found, duplicated. * Docstrings for Superloader; pass in CelestialSightline to get_footprint * coords.get_supergeom: get a supermap to hold submaps * obsfiledb: fix how prefix is set when loading from context * tod_ops: iir_filter function for readout filter conv/deconv This sort of IIR filter can express Smurf (and MCE) readout filters. * metadata: improve docstrings; create ABC for Metadata Loaders. * docs: more context/metadata. * context: tweaks to metadata loader - When loading ManifestDb, map sqlite files as readonly without making a copy. This is the fastest. - colon_obs_tags: prepend keys with 'dets:' before passing to loader. - SuperLoader caches ManifestDb connections. This is probably of modest value for .sqlite on disk, but it helps a lot for .gz->memory databases. * Add support for multiple common modes and randomized coupling strengths. Fix a recent bug preventing simulation of common modes. (#88) * When running setup.py test, propagate the exit status back to the shell. * Context: fix a test that broke due to SuperLoader db caching * AxisManager: fix in_place=False bug, add a bunch of documentation. * HWP demodulation (#87) * Initial implememtation * Add support for caching demodulated 2f component * to dump demodulated data Co-authored-by: Satoru Takakura <[email protected]> * Implement h_n map operator (#91) * initial implementation * use the correct angle for h_n; add missing files * Add cross-linking map support * Use stationary detector weights * h_n should be computed per detector * draft instrument model revisions * added new readout defaults * changed wafer/wafers fields to wafer_slot/wafer_slots * readout naming updates * update detector band naming convention * change detector naming scheme to detectors.w##_p###_bnd_A/B * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change tubes to tube_slots, add fields for tube_name and receiver_name (default blank) * Add field for platform_name * Update toast_so_example.py * updated lat tube names to cryo convention * SAT3 to SAT4 * update test naming conventions * Rotate the LAT boresight according to observing elevation * Move all toast-related source into a toast subpackage. * Fix imports * Fix default value * Reduce size of simulated hardware test. * Support recentering coordinates on given object or location * Remove import added by mistake * Support cuts in coords/pmat.py. Small rewrite in get_footprint for clarity. Handling cuts in the pointing matrix this way isn't enough to suppor them in the maximum-likelihood framework, but it makes the rest of the implementation a bit cleaner. However, right now enabling pmat cuts leads to some unexpected small changes in the result, which should be investigated before they are used properly.\n\nThe get_footprint change was made while I was debugging something else. What was there before was fine too, but I left these changes in because I think it makes it a bit easier to see the logic. While list comprehensions are very explicit, their verbosity can make it hard to see the forest for all the trees. * Hopefully clearer derivation of wcs in get_footprint * Fixed argument order bugs in get_gap_fill * coords: improve compatibility check in get_supergeom; add tests Our tests initially included some supergeom arguments that should have caused failure, but didn't... so now the supergeom compatibilty check is a bit more complete than the pixell one. Here we do not assume that all cylindrical proj must have crval[1]==0. * Add pixell to install_deps so test_coords can run * Move sim_sso * upgrade to new directory structure * complete sso sim integration * Add tool for measuring wafer and optics tube offsets * Remove stale folder Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Andrea Zonca <[email protected]> Co-authored-by: Theodore Kisner <[email protected]> Co-authored-by: Katie Harrington <[email protected]> Co-authored-by: Yilun Guan <[email protected]> Co-authored-by: Sigurd Kirkevold Næss <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Satoru Takakura <[email protected]> Co-authored-by: smsimon <[email protected]>
keskitalo
added a commit
that referenced
this pull request
Feb 11, 2021
* add the new SSO simulation operator to sotodlib.observe * Introduce core submodule, and core.AxisManager * data/load_observation: return an AxisManager * Changes to AxisManager... won't be the last... * metadata module: for now this just imports everything from sotoddb * Context: support for loading site/user/dataset-specific settings * axisman: Bridge missing intersect1d feature in numpy < 1.15 Also fix a data scrambling bug, in passing. Also catch bad (non-disjoint) requests in from_resultset. * Context.load_obs(obs_id) To get that working, added string substitution (from context "tags"), function to trigger database (re)load. obs_id strings can include detector identifiers such as 'f150'; such tags may be drawn from detdb fields listed in context['obs_colon_tags']. * data.load.load_observation: load directly into an AxisManager * OBSLOADER_REGISTRY tracks named observation loading functions And you tell it what loader you need through the context key "obs_loader_type". Both 'default' and 'pipe-s0001' point to data.load.load_observation. For now. * Context.get_meta, can operate independently of get_obs Also fix a couple of other bugs in passing * data.load.load_file, as alternative load_observation It can be used in cases where an ObsFileDb is not available, for example. * Context: load obsdb * Context: obsdb / detdb / obsfiledb load is optional * Whitespace fixes; no functional change * add simple map binning pipeline to visualize so3g data dumps * pass sinc-modulation argument to todground * Modify matplotlib backend to PDF inside plot function Closes #32 * docs: add docstring to set matplotlib backend function * Invert phi to get azimuth in export * Add pipeline for measuring the filter&bin transfer function * tweak the transfer function pipeline * The transfer function pipeline can now run with either Madam or the TOAST mapmaker * support cosecant modulation in SO sims * load_observation: permit user to request dets=[] Sometimes you just want to look at the pointing. This is not terribly time efficient, since the whole file has to be read regardless. But we could imagine superior data formats where the ancillary data is stored separately from the bolometers. * Rearrange existing files to match (as much as possible) the proposed layout in #53. Only delete files that were obviously empty or never used. No changes or cleanup to the code here, just moving things around and changing imports so that unit tests pass. * Fix imports in pipelines * Move context and g3_core into core * Adjust imports of sotoddb classes into new structure This imports ResultSet and the metadata *Db directly to core.metadata, renaming the *DB classes as *Db. The more stuff-on-disk code in sotodlib.simple is imported as io.metadata.simple. This has been tested on simons1 with the SO sim and ACT Uranus demo code. * Fix imports so all tests pass This includes a bug-fix in axisman that would only activate with old numpy. * Fix stray bad import * fix pipeline_tools import * fix small issues in TOAST load/export unit tests * fix broken imports (#67) * fix broken imports * use relative imports * more relative imports * Trivial fix to unit tests * Modifications to allow (#68) * Modifications to allow * Simulating atmosphere on top of loaded signal * writing TOD from a pipeline that begins with loading TOD * fix export when loading from file and using an export key * Time constant (#52) * Initial implementation of the operator and pipeline tools. Thanks to Katie Harrington for the code. * unit test now passes * fix name errors * add missing import * add unit test file * Add units [s] to time constant doc strings * Add parameters to leave a time constant residual * fix time constant unit test * Improve unit tests * move time_constant.py * fix import; add __init__.py * more import fixes * Fully functional time constant operator * fix issue in pipeline_tools; correct the order of time constant and noise simulation * remove obsolete directory * add FlagManager to core imports * add functionality to FlagManager * grammar * initial commits of TOD operations * automatically add FlagManager to tod * add 2 working flags * add __setitem__ for fields only * add simulation mechanim for glitches * fix logic error on intersects * flag updates & fixes * expand the comments! * updates for review * Fix trivial typo * use the new TOAST el-nod facility (#73) * docs: update module paths in API reference * context: fix a critical default behavior get_obs() should refer to obsfiledb if user does not specify detsets. Otherwise, all dets in the detdb will be try-loaded, and that's often too many. * context: accept a dict as obs_id argument to get_obs This is so that a single row from obsdb.get() can be used with convenience. * Context: get_obs accepts a list of ints for detsets Also get_meta is documented and accepts a simple string obs_id. * add copy constructor and update __repr__ to work for inheritance * sotodlib.coords: pointing and coordinate support Integrates so3g accelerated pointing code with tod-in-AxisManager. * fix error in thinning the focal plane for the second band * Make sure the observations get independent instances of the noise PSD (#75) * Run unit tests with github actions. This work: - Disables travis tests - Adds a simple github workflow that: - pulls the latest so3g image - pip-installs dependencies into the image and commits it to a local temporary image - runs the unit tests in this temporary image * Change name of test workflow. * Update test badge * tod_ops.filters: add docstr and signature for fft_filter decorator * Import of sotoddb code -- from sotodlib commit f4ee4167 docs and tests are still to come. * metadata: fix SuperLoader Hdf5Loader issue * metadata: rename DB to Db in main class definitions This doesn't change the external interface. * metadata: rename proddb.py to manifest.py * metadata: import tests from sotoddb * docs: in passing, re-fix name errors for sim/vis_hardware * metadata: import sotoddb docs * tod_ops: PCA module Provides a few functions for decomposing tod covariance into eigenbasis; reconstructing limited mode sets for signal cleaning; and cleaning (subtraction of dominant modes). Also an independent function to find and remove "trends", and return them like they were an set of modes with couplings to the signal. * tod_ops: basic docs for filters and pca. * docs: mock 'yaml', 'spt3g', 'pyfftw' if not found. * docs: mock 'toml', 'sqlite3', 'so3g', 'toast', 'pixell' if not found. * docs: rebrand SOTODLib -> sotodlib; set some autodoc options * tod_ops: gap-filling (#81) Introduces simple polynomial (linear, really) gap filling, as well as "model-based" gap-filling that can use the results of a PCA (or common mode). The Extract and ExtractMatrix classes (the later of which is not quite compatible with storage in AxisManager, but could be moved in that direction) help with storing sparse subsets of a TOD signal efficiently. * core.metadata: Clean up ObsDb code; document it. This includes a minimal API for creating a usable ObsDb. Common database i/o functions created in core/metadata/common.py; other Dbs will need to start using those. * core.metadata: add "tags" functionality to ObsDb; unittest for ObsDb Other minor cleanup as well. * context: remove use of contextlib.nullcontext, since that needs >=3.7 * context: modernize ManifestDb and ObsFileDb with common to_file, from_file * context: raise informative errors when failing to broadcast/match metadata * tests/test_metadata.py: tests for the metadata resolution system The test is especially needed now, since this system works but I want to refactor the HDF5 stuff and still have it work the same... * metadata: ManifestDb raises informative error when too many matches * metadata: ResultSet.asarray has hdf_compat option And that option causes strings to be coded as byte strings in the output array, so the result can be written as an HDF5 dataset. * io.metadata: ResultSetHdfLoader and write_dataset PerDetectorHdf5 is now deprecated, but left in place as a shell (probably not for long). The ResultSetHdfLoader is tuned for the context/metadata system, with a single API (the one hacked in through PerDetectorHdf5.loader_class). For writing datasets, a function makes more sense than having a write_dataset method, assuming we don't want to put HDF5 i/o right in the ResultSet class. Also added a test for the array string conversion function. * context/metadata: remove PerDetectorHdf5; ResultSetHdf is now default This still requires a Context user to import sotodlib.io.metadata before trying metdata loads. * context: remove abandoned code in ManifestDb * context: the Context can include a list of "imports" This is useful to cause registration of data and metadata loaders for Context to use. * WIP: updates to context documentation. * Avoid slow slice operation when merging axis managers * Simplifiy and slightly generalize simplify_slice. It can now simplify along each dimension independently. * Avoid confusing situations where attempted updates to members of the axismanager instead end up creating an entry in the axismanager object itself, that masks the actually managed one * Fixed oversimplification of slice in simplify_slice. * context: get_obs has a "logic_only" arg And it causes the function to return the load specification (including what dets) without loading any data or metadata. * context/metadata: better handling of obs_colon_tags Raises errors if parent field not found, duplicated. * Docstrings for Superloader; pass in CelestialSightline to get_footprint * coords.get_supergeom: get a supermap to hold submaps * obsfiledb: fix how prefix is set when loading from context * tod_ops: iir_filter function for readout filter conv/deconv This sort of IIR filter can express Smurf (and MCE) readout filters. * metadata: improve docstrings; create ABC for Metadata Loaders. * docs: more context/metadata. * context: tweaks to metadata loader - When loading ManifestDb, map sqlite files as readonly without making a copy. This is the fastest. - colon_obs_tags: prepend keys with 'dets:' before passing to loader. - SuperLoader caches ManifestDb connections. This is probably of modest value for .sqlite on disk, but it helps a lot for .gz->memory databases. * Add support for multiple common modes and randomized coupling strengths. Fix a recent bug preventing simulation of common modes. (#88) * When running setup.py test, propagate the exit status back to the shell. * Context: fix a test that broke due to SuperLoader db caching * AxisManager: fix in_place=False bug, add a bunch of documentation. * HWP demodulation (#87) * Initial implememtation * Add support for caching demodulated 2f component * to dump demodulated data Co-authored-by: Satoru Takakura <[email protected]> * Implement h_n map operator (#91) * initial implementation * use the correct angle for h_n; add missing files * Add cross-linking map support * Use stationary detector weights * h_n should be computed per detector * draft instrument model revisions * added new readout defaults * changed wafer/wafers fields to wafer_slot/wafer_slots * readout naming updates * update detector band naming convention * change detector naming scheme to detectors.w##_p###_bnd_A/B * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change tubes to tube_slots, add fields for tube_name and receiver_name (default blank) * Add field for platform_name * Update toast_so_example.py * updated lat tube names to cryo convention * SAT3 to SAT4 * update test naming conventions * Rotate the LAT boresight according to observing elevation * Move all toast-related source into a toast subpackage. * Fix imports * Fix default value * Reduce size of simulated hardware test. * Support recentering coordinates on given object or location * Remove import added by mistake * Support cuts in coords/pmat.py. Small rewrite in get_footprint for clarity. Handling cuts in the pointing matrix this way isn't enough to suppor them in the maximum-likelihood framework, but it makes the rest of the implementation a bit cleaner. However, right now enabling pmat cuts leads to some unexpected small changes in the result, which should be investigated before they are used properly.\n\nThe get_footprint change was made while I was debugging something else. What was there before was fine too, but I left these changes in because I think it makes it a bit easier to see the logic. While list comprehensions are very explicit, their verbosity can make it hard to see the forest for all the trees. * Hopefully clearer derivation of wcs in get_footprint * Fixed argument order bugs in get_gap_fill * coords: improve compatibility check in get_supergeom; add tests Our tests initially included some supergeom arguments that should have caused failure, but didn't... so now the supergeom compatibilty check is a bit more complete than the pixell one. Here we do not assume that all cylindrical proj must have crval[1]==0. * Add pixell to install_deps so test_coords can run * Move sim_sso * upgrade to new directory structure * complete sso sim integration * Add tool for measuring wafer and optics tube offsets * Remove stale folder Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Andrea Zonca <[email protected]> Co-authored-by: Theodore Kisner <[email protected]> Co-authored-by: Katie Harrington <[email protected]> Co-authored-by: Yilun Guan <[email protected]> Co-authored-by: Sigurd Kirkevold Næss <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Satoru Takakura <[email protected]> Co-authored-by: smsimon <[email protected]>
keskitalo
added a commit
that referenced
this pull request
Feb 12, 2021
* Rotate the LAT focalplane based on elevation and encode the corotator angle in the so3g files * Check export key for validity * add TOAST MapMaker and FilterBin support to toast_so_sim.py (#102) * Fix a recent bug introduced in writing demodulated TOD (#104) * Initial implementation (#101) * Sso sim (#103) * add the new SSO simulation operator to sotodlib.observe * Introduce core submodule, and core.AxisManager * data/load_observation: return an AxisManager * Changes to AxisManager... won't be the last... * metadata module: for now this just imports everything from sotoddb * Context: support for loading site/user/dataset-specific settings * axisman: Bridge missing intersect1d feature in numpy < 1.15 Also fix a data scrambling bug, in passing. Also catch bad (non-disjoint) requests in from_resultset. * Context.load_obs(obs_id) To get that working, added string substitution (from context "tags"), function to trigger database (re)load. obs_id strings can include detector identifiers such as 'f150'; such tags may be drawn from detdb fields listed in context['obs_colon_tags']. * data.load.load_observation: load directly into an AxisManager * OBSLOADER_REGISTRY tracks named observation loading functions And you tell it what loader you need through the context key "obs_loader_type". Both 'default' and 'pipe-s0001' point to data.load.load_observation. For now. * Context.get_meta, can operate independently of get_obs Also fix a couple of other bugs in passing * data.load.load_file, as alternative load_observation It can be used in cases where an ObsFileDb is not available, for example. * Context: load obsdb * Context: obsdb / detdb / obsfiledb load is optional * Whitespace fixes; no functional change * add simple map binning pipeline to visualize so3g data dumps * pass sinc-modulation argument to todground * Modify matplotlib backend to PDF inside plot function Closes #32 * docs: add docstring to set matplotlib backend function * Invert phi to get azimuth in export * Add pipeline for measuring the filter&bin transfer function * tweak the transfer function pipeline * The transfer function pipeline can now run with either Madam or the TOAST mapmaker * support cosecant modulation in SO sims * load_observation: permit user to request dets=[] Sometimes you just want to look at the pointing. This is not terribly time efficient, since the whole file has to be read regardless. But we could imagine superior data formats where the ancillary data is stored separately from the bolometers. * Rearrange existing files to match (as much as possible) the proposed layout in #53. Only delete files that were obviously empty or never used. No changes or cleanup to the code here, just moving things around and changing imports so that unit tests pass. * Fix imports in pipelines * Move context and g3_core into core * Adjust imports of sotoddb classes into new structure This imports ResultSet and the metadata *Db directly to core.metadata, renaming the *DB classes as *Db. The more stuff-on-disk code in sotodlib.simple is imported as io.metadata.simple. This has been tested on simons1 with the SO sim and ACT Uranus demo code. * Fix imports so all tests pass This includes a bug-fix in axisman that would only activate with old numpy. * Fix stray bad import * fix pipeline_tools import * fix small issues in TOAST load/export unit tests * fix broken imports (#67) * fix broken imports * use relative imports * more relative imports * Trivial fix to unit tests * Modifications to allow (#68) * Modifications to allow * Simulating atmosphere on top of loaded signal * writing TOD from a pipeline that begins with loading TOD * fix export when loading from file and using an export key * Time constant (#52) * Initial implementation of the operator and pipeline tools. Thanks to Katie Harrington for the code. * unit test now passes * fix name errors * add missing import * add unit test file * Add units [s] to time constant doc strings * Add parameters to leave a time constant residual * fix time constant unit test * Improve unit tests * move time_constant.py * fix import; add __init__.py * more import fixes * Fully functional time constant operator * fix issue in pipeline_tools; correct the order of time constant and noise simulation * remove obsolete directory * add FlagManager to core imports * add functionality to FlagManager * grammar * initial commits of TOD operations * automatically add FlagManager to tod * add 2 working flags * add __setitem__ for fields only * add simulation mechanim for glitches * fix logic error on intersects * flag updates & fixes * expand the comments! * updates for review * Fix trivial typo * use the new TOAST el-nod facility (#73) * docs: update module paths in API reference * context: fix a critical default behavior get_obs() should refer to obsfiledb if user does not specify detsets. Otherwise, all dets in the detdb will be try-loaded, and that's often too many. * context: accept a dict as obs_id argument to get_obs This is so that a single row from obsdb.get() can be used with convenience. * Context: get_obs accepts a list of ints for detsets Also get_meta is documented and accepts a simple string obs_id. * add copy constructor and update __repr__ to work for inheritance * sotodlib.coords: pointing and coordinate support Integrates so3g accelerated pointing code with tod-in-AxisManager. * fix error in thinning the focal plane for the second band * Make sure the observations get independent instances of the noise PSD (#75) * Run unit tests with github actions. This work: - Disables travis tests - Adds a simple github workflow that: - pulls the latest so3g image - pip-installs dependencies into the image and commits it to a local temporary image - runs the unit tests in this temporary image * Change name of test workflow. * Update test badge * tod_ops.filters: add docstr and signature for fft_filter decorator * Import of sotoddb code -- from sotodlib commit f4ee4167 docs and tests are still to come. * metadata: fix SuperLoader Hdf5Loader issue * metadata: rename DB to Db in main class definitions This doesn't change the external interface. * metadata: rename proddb.py to manifest.py * metadata: import tests from sotoddb * docs: in passing, re-fix name errors for sim/vis_hardware * metadata: import sotoddb docs * tod_ops: PCA module Provides a few functions for decomposing tod covariance into eigenbasis; reconstructing limited mode sets for signal cleaning; and cleaning (subtraction of dominant modes). Also an independent function to find and remove "trends", and return them like they were an set of modes with couplings to the signal. * tod_ops: basic docs for filters and pca. * docs: mock 'yaml', 'spt3g', 'pyfftw' if not found. * docs: mock 'toml', 'sqlite3', 'so3g', 'toast', 'pixell' if not found. * docs: rebrand SOTODLib -> sotodlib; set some autodoc options * tod_ops: gap-filling (#81) Introduces simple polynomial (linear, really) gap filling, as well as "model-based" gap-filling that can use the results of a PCA (or common mode). The Extract and ExtractMatrix classes (the later of which is not quite compatible with storage in AxisManager, but could be moved in that direction) help with storing sparse subsets of a TOD signal efficiently. * core.metadata: Clean up ObsDb code; document it. This includes a minimal API for creating a usable ObsDb. Common database i/o functions created in core/metadata/common.py; other Dbs will need to start using those. * core.metadata: add "tags" functionality to ObsDb; unittest for ObsDb Other minor cleanup as well. * context: remove use of contextlib.nullcontext, since that needs >=3.7 * context: modernize ManifestDb and ObsFileDb with common to_file, from_file * context: raise informative errors when failing to broadcast/match metadata * tests/test_metadata.py: tests for the metadata resolution system The test is especially needed now, since this system works but I want to refactor the HDF5 stuff and still have it work the same... * metadata: ManifestDb raises informative error when too many matches * metadata: ResultSet.asarray has hdf_compat option And that option causes strings to be coded as byte strings in the output array, so the result can be written as an HDF5 dataset. * io.metadata: ResultSetHdfLoader and write_dataset PerDetectorHdf5 is now deprecated, but left in place as a shell (probably not for long). The ResultSetHdfLoader is tuned for the context/metadata system, with a single API (the one hacked in through PerDetectorHdf5.loader_class). For writing datasets, a function makes more sense than having a write_dataset method, assuming we don't want to put HDF5 i/o right in the ResultSet class. Also added a test for the array string conversion function. * context/metadata: remove PerDetectorHdf5; ResultSetHdf is now default This still requires a Context user to import sotodlib.io.metadata before trying metdata loads. * context: remove abandoned code in ManifestDb * context: the Context can include a list of "imports" This is useful to cause registration of data and metadata loaders for Context to use. * WIP: updates to context documentation. * Avoid slow slice operation when merging axis managers * Simplifiy and slightly generalize simplify_slice. It can now simplify along each dimension independently. * Avoid confusing situations where attempted updates to members of the axismanager instead end up creating an entry in the axismanager object itself, that masks the actually managed one * Fixed oversimplification of slice in simplify_slice. * context: get_obs has a "logic_only" arg And it causes the function to return the load specification (including what dets) without loading any data or metadata. * context/metadata: better handling of obs_colon_tags Raises errors if parent field not found, duplicated. * Docstrings for Superloader; pass in CelestialSightline to get_footprint * coords.get_supergeom: get a supermap to hold submaps * obsfiledb: fix how prefix is set when loading from context * tod_ops: iir_filter function for readout filter conv/deconv This sort of IIR filter can express Smurf (and MCE) readout filters. * metadata: improve docstrings; create ABC for Metadata Loaders. * docs: more context/metadata. * context: tweaks to metadata loader - When loading ManifestDb, map sqlite files as readonly without making a copy. This is the fastest. - colon_obs_tags: prepend keys with 'dets:' before passing to loader. - SuperLoader caches ManifestDb connections. This is probably of modest value for .sqlite on disk, but it helps a lot for .gz->memory databases. * Add support for multiple common modes and randomized coupling strengths. Fix a recent bug preventing simulation of common modes. (#88) * When running setup.py test, propagate the exit status back to the shell. * Context: fix a test that broke due to SuperLoader db caching * AxisManager: fix in_place=False bug, add a bunch of documentation. * HWP demodulation (#87) * Initial implememtation * Add support for caching demodulated 2f component * to dump demodulated data Co-authored-by: Satoru Takakura <[email protected]> * Implement h_n map operator (#91) * initial implementation * use the correct angle for h_n; add missing files * Add cross-linking map support * Use stationary detector weights * h_n should be computed per detector * draft instrument model revisions * added new readout defaults * changed wafer/wafers fields to wafer_slot/wafer_slots * readout naming updates * update detector band naming convention * change detector naming scheme to detectors.w##_p###_bnd_A/B * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change tubes to tube_slots, add fields for tube_name and receiver_name (default blank) * Add field for platform_name * Update toast_so_example.py * updated lat tube names to cryo convention * SAT3 to SAT4 * update test naming conventions * Rotate the LAT boresight according to observing elevation * Move all toast-related source into a toast subpackage. * Fix imports * Fix default value * Reduce size of simulated hardware test. * Support recentering coordinates on given object or location * Remove import added by mistake * Support cuts in coords/pmat.py. Small rewrite in get_footprint for clarity. Handling cuts in the pointing matrix this way isn't enough to suppor them in the maximum-likelihood framework, but it makes the rest of the implementation a bit cleaner. However, right now enabling pmat cuts leads to some unexpected small changes in the result, which should be investigated before they are used properly.\n\nThe get_footprint change was made while I was debugging something else. What was there before was fine too, but I left these changes in because I think it makes it a bit easier to see the logic. While list comprehensions are very explicit, their verbosity can make it hard to see the forest for all the trees. * Hopefully clearer derivation of wcs in get_footprint * Fixed argument order bugs in get_gap_fill * coords: improve compatibility check in get_supergeom; add tests Our tests initially included some supergeom arguments that should have caused failure, but didn't... so now the supergeom compatibilty check is a bit more complete than the pixell one. Here we do not assume that all cylindrical proj must have crval[1]==0. * Add pixell to install_deps so test_coords can run * Move sim_sso * upgrade to new directory structure * complete sso sim integration * Add tool for measuring wafer and optics tube offsets * Remove stale folder Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Andrea Zonca <[email protected]> Co-authored-by: Theodore Kisner <[email protected]> Co-authored-by: Katie Harrington <[email protected]> Co-authored-by: Yilun Guan <[email protected]> Co-authored-by: Sigurd Kirkevold Næss <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Satoru Takakura <[email protected]> Co-authored-by: smsimon <[email protected]> * Add missing metadata required by atmospheric sims. * Swap Az/El and return the radius of the FOV * Rotate the LAT focalplane based on elevation and encode the corotator angle in the so3g files * write the corotator angle into so3g files * keep the cache object intact * Now receiver compensates for elevation rotation by default * Fix default corotator angle * Load the corotator angle when available * TOAST should also load the corotator angle Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Andrea Zonca <[email protected]> Co-authored-by: Theodore Kisner <[email protected]> Co-authored-by: Katie Harrington <[email protected]> Co-authored-by: Yilun Guan <[email protected]> Co-authored-by: Sigurd Kirkevold Næss <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Satoru Takakura <[email protected]> Co-authored-by: smsimon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces simple polynomial (linear, really) gap filling, as well as "model-based" gap-filling that can use the results of a PCA (or common mode).
The Extract and ExtractMatrix classes (the later of which is not quite compatible with storage in AxisManager, but could be moved in that direction) help with storing sparse subsets of a TOD signal efficiently.