Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make common mode noise more versatile #88

Merged
merged 1 commit into from
Sep 24, 2020
Merged

Conversation

keskitalo
Copy link
Member

  • Add support for multiple common modes and randomized coupling strengths.
  • Fix a recent bug preventing simulation of common modes.

…hs. Fix a recent bug preventing simulation of common modes.
@keskitalo keskitalo requested a review from tskisner August 24, 2020 16:30
@keskitalo
Copy link
Member Author

Self-merging this trivial PR.

@keskitalo keskitalo merged commit 1f8c730 into master Sep 24, 2020
@keskitalo keskitalo deleted the more_common_mode_noise branch September 24, 2020 14:01
keskitalo added a commit that referenced this pull request Jan 28, 2021
…hs. Fix a recent bug preventing simulation of common modes. (#88)
keskitalo added a commit that referenced this pull request Feb 2, 2021
* add the new SSO simulation operator to sotodlib.observe

* Introduce core submodule, and core.AxisManager

* data/load_observation: return an AxisManager

* Changes to AxisManager... won't be the last...

* metadata module: for now this just imports everything from sotoddb

* Context: support for loading site/user/dataset-specific settings

* axisman: Bridge missing intersect1d feature in numpy < 1.15

Also fix a data scrambling bug, in passing.  Also catch bad
(non-disjoint) requests in from_resultset.

* Context.load_obs(obs_id)

To get that working, added string substitution (from context "tags"),
function to trigger database (re)load.  obs_id strings can include
detector identifiers such as 'f150'; such tags may be drawn from detdb
fields listed in context['obs_colon_tags'].

* data.load.load_observation: load directly into an AxisManager

* OBSLOADER_REGISTRY tracks named observation loading functions

And you tell it what loader you need through the context key
"obs_loader_type".  Both 'default' and 'pipe-s0001' point to
data.load.load_observation.  For now.

* Context.get_meta, can operate independently of get_obs

Also fix a couple of other bugs in passing

* data.load.load_file, as alternative load_observation

It can be used in cases where an ObsFileDb is not available, for
example.

* Context: load obsdb

* Context: obsdb / detdb / obsfiledb load is optional

* Whitespace fixes; no functional change

* add simple map binning pipeline to visualize so3g data dumps

* pass sinc-modulation argument to todground

* Modify matplotlib backend to PDF inside plot function

Closes #32

* docs: add docstring to set matplotlib backend function

* Invert phi to get azimuth in export

* Add pipeline for measuring the filter&bin transfer function

* tweak the transfer function pipeline

* The transfer function pipeline can now run with either Madam or the TOAST mapmaker

* support cosecant modulation in SO sims

* load_observation: permit user to request dets=[]

Sometimes you just want to look at the pointing.  This is not terribly
time efficient, since the whole file has to be read regardless.  But
we could imagine superior data formats where the ancillary data is
stored separately from the bolometers.

* Rearrange existing files to match (as much as possible)
the proposed layout in #53.  Only delete files that were
obviously empty or never used.  No changes or cleanup to
the code here, just moving things around and changing
imports so that unit tests pass.

* Fix imports in pipelines

* Move context and g3_core into core

* Adjust imports of sotoddb classes into new structure

This imports ResultSet and the metadata *Db directly to core.metadata,
renaming the *DB classes as *Db.  The more stuff-on-disk code in
sotodlib.simple is imported as io.metadata.simple.  This has been
tested on simons1 with the SO sim and ACT Uranus demo code.

* Fix imports so all tests pass

This includes a bug-fix in axisman that would only activate with old
numpy.

* Fix stray bad import

* fix pipeline_tools import

* fix small issues in TOAST load/export unit tests

* fix broken imports (#67)

* fix broken imports

* use relative imports

* more relative imports

* Trivial fix to unit tests

* Modifications to allow (#68)

* Modifications to allow
* Simulating atmosphere on top of loaded signal
* writing TOD from a pipeline that begins with loading TOD

* fix export when loading from file and using an export key

* Time constant (#52)

* Initial implementation of the operator and pipeline tools. Thanks to Katie Harrington for the code.

* unit test now passes

* fix name errors

* add missing import

* add unit test file

* Add units [s] to time constant doc strings

* Add parameters to leave a time constant residual

* fix time constant unit test

* Improve unit tests

* move time_constant.py

* fix import; add __init__.py

* more import fixes

* Fully functional time constant operator

* fix issue in pipeline_tools; correct the order of time constant and noise simulation

* remove obsolete directory

* add FlagManager to core imports

* add functionality to FlagManager

* grammar

* initial commits of TOD operations

* automatically add FlagManager to tod

* add 2 working flags

* add __setitem__ for fields only

* add simulation mechanim for glitches

* fix logic error on intersects

* flag updates & fixes

* expand the comments!

* updates for review

* Fix trivial typo

* use the new TOAST el-nod facility (#73)

* docs: update module paths in API reference

* context: fix a critical default behavior

get_obs() should refer to obsfiledb if user does not specify detsets.
Otherwise, all dets in the detdb will be try-loaded, and that's often
too many.

* context: accept a dict as obs_id argument to get_obs

This is so that a single row from obsdb.get() can be used with convenience.

* Context: get_obs accepts a list of ints for detsets

Also get_meta is documented and accepts a simple string obs_id.

* add copy constructor and update __repr__ to work for inheritance

* sotodlib.coords: pointing and coordinate support

Integrates so3g accelerated pointing code with tod-in-AxisManager.

* fix error in thinning the focal plane for the second band

* Make sure the observations get independent instances of the noise PSD (#75)

* Run unit tests with github actions.  This work:

  - Disables travis tests

  - Adds a simple github workflow that:
    - pulls the latest so3g image
    - pip-installs dependencies into the image and commits
      it to a local temporary image
    - runs the unit tests in this temporary image

* Change name of test workflow.

* Update test badge

* tod_ops.filters: add docstr and signature for fft_filter decorator

* Import of sotoddb code -- from sotodlib commit f4ee4167

docs and tests are still to come.

* metadata: fix SuperLoader Hdf5Loader issue

* metadata: rename DB to Db in main class definitions

This doesn't change the external interface.

* metadata: rename proddb.py to manifest.py

* metadata: import tests from sotoddb

* docs: in passing, re-fix name errors for sim/vis_hardware

* metadata: import sotoddb docs

* tod_ops: PCA module

Provides a few functions for decomposing tod covariance into
eigenbasis; reconstructing limited mode sets for signal cleaning; and
cleaning (subtraction of dominant modes).  Also an independent
function to find and remove "trends", and return them like they were
an set of modes with couplings to the signal.

* tod_ops: basic docs for filters and pca.

* docs: mock 'yaml', 'spt3g', 'pyfftw' if not found.

* docs: mock 'toml', 'sqlite3', 'so3g', 'toast', 'pixell' if not found.

* docs: rebrand SOTODLib -> sotodlib; set some autodoc options

* tod_ops: gap-filling (#81)

Introduces simple polynomial (linear, really) gap filling, as well as
"model-based" gap-filling that can use the results of a PCA (or common
mode).

The Extract and ExtractMatrix classes (the later of which is not quite
compatible with storage in AxisManager, but could be moved in that
direction) help with storing sparse subsets of a TOD signal
efficiently.

* core.metadata:  Clean up ObsDb code; document it.

This includes a minimal API for creating a usable ObsDb.  Common
database i/o functions created in core/metadata/common.py; other Dbs
will need to start using those.

* core.metadata: add "tags" functionality to ObsDb; unittest for ObsDb

Other minor cleanup as well.

* context: remove use of contextlib.nullcontext, since that needs >=3.7

* context: modernize ManifestDb and ObsFileDb with common to_file, from_file

* context: raise informative errors when failing to broadcast/match metadata

* tests/test_metadata.py: tests for the metadata resolution system

The test is especially needed now, since this system works but I want
to refactor the HDF5 stuff and still have it work the same...

* metadata: ManifestDb raises informative error when too many matches

* metadata: ResultSet.asarray has hdf_compat option

And that option causes strings to be coded as byte strings in the
output array, so the result can be written as an HDF5 dataset.

* io.metadata: ResultSetHdfLoader and write_dataset

PerDetectorHdf5 is now deprecated, but left in place as a shell
(probably not for long).  The ResultSetHdfLoader is tuned for the
context/metadata system, with a single API (the one hacked in through
PerDetectorHdf5.loader_class).  For writing datasets, a function makes
more sense than having a write_dataset method, assuming we don't want
to put HDF5 i/o right in the ResultSet class.

Also added a test for the array string conversion function.

* context/metadata: remove PerDetectorHdf5; ResultSetHdf is now default

This still requires a Context user to import sotodlib.io.metadata
before trying metdata loads.

* context: remove abandoned code in ManifestDb

* context: the Context can include a list of "imports"

This is useful to cause registration of data and metadata loaders for
Context to use.

* WIP: updates to context documentation.

* Avoid slow slice operation when merging axis managers

* Simplifiy and slightly generalize simplify_slice. It can now simplify along each dimension independently.

* Avoid confusing situations where attempted updates to members of the axismanager instead end up creating an entry in the axismanager object itself, that masks the actually managed one

* Fixed oversimplification of slice in simplify_slice.

* context: get_obs has a "logic_only" arg

And it causes the function to return the load specification (including
what dets) without loading any data or metadata.

* context/metadata: better handling of obs_colon_tags

Raises errors if parent field not found, duplicated.

* Docstrings for Superloader; pass in CelestialSightline to get_footprint

* coords.get_supergeom: get a supermap to hold submaps

* obsfiledb: fix how prefix is set when loading from context

* tod_ops: iir_filter function for readout filter conv/deconv

This sort of IIR filter can express Smurf (and MCE) readout filters.

* metadata: improve docstrings; create ABC for Metadata Loaders.

* docs: more context/metadata.

* context: tweaks to metadata loader

- When loading ManifestDb, map sqlite files as readonly without making
  a copy.  This is the fastest.
- colon_obs_tags: prepend keys with 'dets:' before passing to loader.
- SuperLoader caches ManifestDb connections.  This is probably of
  modest value for .sqlite on disk, but it helps a lot for .gz->memory
  databases.

* Add support for multiple common modes and randomized coupling strengths. Fix a recent bug preventing simulation of common modes. (#88)

* When running setup.py test, propagate the exit status back to the shell.

* Context: fix a test that broke due to SuperLoader db caching

* AxisManager: fix in_place=False bug, add a bunch of documentation.

* HWP demodulation (#87)

* Initial implememtation

* Add support for caching demodulated 2f component

* to dump demodulated data

Co-authored-by: Satoru Takakura <[email protected]>

* Implement h_n map operator (#91)

* initial implementation

* use the correct angle for h_n; add missing files

* Add cross-linking map support

* Use stationary detector weights

* h_n should be computed per detector

* draft instrument model revisions

* added new readout defaults

* changed wafer/wafers fields to wafer_slot/wafer_slots

* readout naming updates

* update detector band naming convention

* change detector naming scheme to detectors.w##_p###_bnd_A/B

* change wafer naming to w##, card slot to card_slot##, crate slot crate_slot##

* change wafer naming to w##, card slot to card_slot##, crate slot crate_slot##

* change tubes to tube_slots, add fields for tube_name and receiver_name (default blank)

* Add field for platform_name

* Update toast_so_example.py

* updated lat tube names to cryo convention

* SAT3 to SAT4

* update test naming conventions

* Rotate the LAT boresight according to observing elevation

* Move all toast-related source into a toast subpackage.

* Fix imports

* Fix default value

* Reduce size of simulated hardware test.

* Support recentering coordinates on given object or location

* Remove import added by mistake

* Support cuts in coords/pmat.py. Small rewrite in get_footprint for clarity.

Handling cuts in the pointing matrix this way isn't enough to suppor
them in the maximum-likelihood framework, but it makes the rest of the
implementation a bit cleaner. However, right now enabling pmat cuts
leads to some unexpected small changes in the result, which should be
investigated before they are used properly.\n\nThe get_footprint change
was made while I was debugging something else. What was there before was
fine too, but I left these changes in because I think it makes it a bit
easier to see the logic. While list comprehensions are very explicit,
their verbosity can make it hard to see the forest for all the trees.

* Hopefully clearer derivation of wcs in get_footprint

* Fixed argument order bugs in get_gap_fill

* coords: improve compatibility check in get_supergeom; add tests

Our tests initially included some supergeom arguments that should have
caused failure, but didn't... so now the supergeom compatibilty check
is a bit more complete than the pixell one.  Here we do not assume
that all cylindrical proj must have crval[1]==0.

* Add pixell to install_deps so test_coords can run

* Move sim_sso

* upgrade to new directory structure

* complete sso sim integration

* Add tool for measuring wafer and optics tube offsets

* Remove stale folder

Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Andrea Zonca <[email protected]>
Co-authored-by: Theodore Kisner <[email protected]>
Co-authored-by: Katie Harrington <[email protected]>
Co-authored-by: Yilun Guan <[email protected]>
Co-authored-by: Sigurd Kirkevold Næss <[email protected]>
Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Satoru Takakura <[email protected]>
Co-authored-by: smsimon <[email protected]>
keskitalo added a commit that referenced this pull request Feb 11, 2021
* add the new SSO simulation operator to sotodlib.observe

* Introduce core submodule, and core.AxisManager

* data/load_observation: return an AxisManager

* Changes to AxisManager... won't be the last...

* metadata module: for now this just imports everything from sotoddb

* Context: support for loading site/user/dataset-specific settings

* axisman: Bridge missing intersect1d feature in numpy < 1.15

Also fix a data scrambling bug, in passing.  Also catch bad
(non-disjoint) requests in from_resultset.

* Context.load_obs(obs_id)

To get that working, added string substitution (from context "tags"),
function to trigger database (re)load.  obs_id strings can include
detector identifiers such as 'f150'; such tags may be drawn from detdb
fields listed in context['obs_colon_tags'].

* data.load.load_observation: load directly into an AxisManager

* OBSLOADER_REGISTRY tracks named observation loading functions

And you tell it what loader you need through the context key
"obs_loader_type".  Both 'default' and 'pipe-s0001' point to
data.load.load_observation.  For now.

* Context.get_meta, can operate independently of get_obs

Also fix a couple of other bugs in passing

* data.load.load_file, as alternative load_observation

It can be used in cases where an ObsFileDb is not available, for
example.

* Context: load obsdb

* Context: obsdb / detdb / obsfiledb load is optional

* Whitespace fixes; no functional change

* add simple map binning pipeline to visualize so3g data dumps

* pass sinc-modulation argument to todground

* Modify matplotlib backend to PDF inside plot function

Closes #32

* docs: add docstring to set matplotlib backend function

* Invert phi to get azimuth in export

* Add pipeline for measuring the filter&bin transfer function

* tweak the transfer function pipeline

* The transfer function pipeline can now run with either Madam or the TOAST mapmaker

* support cosecant modulation in SO sims

* load_observation: permit user to request dets=[]

Sometimes you just want to look at the pointing.  This is not terribly
time efficient, since the whole file has to be read regardless.  But
we could imagine superior data formats where the ancillary data is
stored separately from the bolometers.

* Rearrange existing files to match (as much as possible)
the proposed layout in #53.  Only delete files that were
obviously empty or never used.  No changes or cleanup to
the code here, just moving things around and changing
imports so that unit tests pass.

* Fix imports in pipelines

* Move context and g3_core into core

* Adjust imports of sotoddb classes into new structure

This imports ResultSet and the metadata *Db directly to core.metadata,
renaming the *DB classes as *Db.  The more stuff-on-disk code in
sotodlib.simple is imported as io.metadata.simple.  This has been
tested on simons1 with the SO sim and ACT Uranus demo code.

* Fix imports so all tests pass

This includes a bug-fix in axisman that would only activate with old
numpy.

* Fix stray bad import

* fix pipeline_tools import

* fix small issues in TOAST load/export unit tests

* fix broken imports (#67)

* fix broken imports

* use relative imports

* more relative imports

* Trivial fix to unit tests

* Modifications to allow (#68)

* Modifications to allow
* Simulating atmosphere on top of loaded signal
* writing TOD from a pipeline that begins with loading TOD

* fix export when loading from file and using an export key

* Time constant (#52)

* Initial implementation of the operator and pipeline tools. Thanks to Katie Harrington for the code.

* unit test now passes

* fix name errors

* add missing import

* add unit test file

* Add units [s] to time constant doc strings

* Add parameters to leave a time constant residual

* fix time constant unit test

* Improve unit tests

* move time_constant.py

* fix import; add __init__.py

* more import fixes

* Fully functional time constant operator

* fix issue in pipeline_tools; correct the order of time constant and noise simulation

* remove obsolete directory

* add FlagManager to core imports

* add functionality to FlagManager

* grammar

* initial commits of TOD operations

* automatically add FlagManager to tod

* add 2 working flags

* add __setitem__ for fields only

* add simulation mechanim for glitches

* fix logic error on intersects

* flag updates & fixes

* expand the comments!

* updates for review

* Fix trivial typo

* use the new TOAST el-nod facility (#73)

* docs: update module paths in API reference

* context: fix a critical default behavior

get_obs() should refer to obsfiledb if user does not specify detsets.
Otherwise, all dets in the detdb will be try-loaded, and that's often
too many.

* context: accept a dict as obs_id argument to get_obs

This is so that a single row from obsdb.get() can be used with convenience.

* Context: get_obs accepts a list of ints for detsets

Also get_meta is documented and accepts a simple string obs_id.

* add copy constructor and update __repr__ to work for inheritance

* sotodlib.coords: pointing and coordinate support

Integrates so3g accelerated pointing code with tod-in-AxisManager.

* fix error in thinning the focal plane for the second band

* Make sure the observations get independent instances of the noise PSD (#75)

* Run unit tests with github actions.  This work:

  - Disables travis tests

  - Adds a simple github workflow that:
    - pulls the latest so3g image
    - pip-installs dependencies into the image and commits
      it to a local temporary image
    - runs the unit tests in this temporary image

* Change name of test workflow.

* Update test badge

* tod_ops.filters: add docstr and signature for fft_filter decorator

* Import of sotoddb code -- from sotodlib commit f4ee4167

docs and tests are still to come.

* metadata: fix SuperLoader Hdf5Loader issue

* metadata: rename DB to Db in main class definitions

This doesn't change the external interface.

* metadata: rename proddb.py to manifest.py

* metadata: import tests from sotoddb

* docs: in passing, re-fix name errors for sim/vis_hardware

* metadata: import sotoddb docs

* tod_ops: PCA module

Provides a few functions for decomposing tod covariance into
eigenbasis; reconstructing limited mode sets for signal cleaning; and
cleaning (subtraction of dominant modes).  Also an independent
function to find and remove "trends", and return them like they were
an set of modes with couplings to the signal.

* tod_ops: basic docs for filters and pca.

* docs: mock 'yaml', 'spt3g', 'pyfftw' if not found.

* docs: mock 'toml', 'sqlite3', 'so3g', 'toast', 'pixell' if not found.

* docs: rebrand SOTODLib -> sotodlib; set some autodoc options

* tod_ops: gap-filling (#81)

Introduces simple polynomial (linear, really) gap filling, as well as
"model-based" gap-filling that can use the results of a PCA (or common
mode).

The Extract and ExtractMatrix classes (the later of which is not quite
compatible with storage in AxisManager, but could be moved in that
direction) help with storing sparse subsets of a TOD signal
efficiently.

* core.metadata:  Clean up ObsDb code; document it.

This includes a minimal API for creating a usable ObsDb.  Common
database i/o functions created in core/metadata/common.py; other Dbs
will need to start using those.

* core.metadata: add "tags" functionality to ObsDb; unittest for ObsDb

Other minor cleanup as well.

* context: remove use of contextlib.nullcontext, since that needs >=3.7

* context: modernize ManifestDb and ObsFileDb with common to_file, from_file

* context: raise informative errors when failing to broadcast/match metadata

* tests/test_metadata.py: tests for the metadata resolution system

The test is especially needed now, since this system works but I want
to refactor the HDF5 stuff and still have it work the same...

* metadata: ManifestDb raises informative error when too many matches

* metadata: ResultSet.asarray has hdf_compat option

And that option causes strings to be coded as byte strings in the
output array, so the result can be written as an HDF5 dataset.

* io.metadata: ResultSetHdfLoader and write_dataset

PerDetectorHdf5 is now deprecated, but left in place as a shell
(probably not for long).  The ResultSetHdfLoader is tuned for the
context/metadata system, with a single API (the one hacked in through
PerDetectorHdf5.loader_class).  For writing datasets, a function makes
more sense than having a write_dataset method, assuming we don't want
to put HDF5 i/o right in the ResultSet class.

Also added a test for the array string conversion function.

* context/metadata: remove PerDetectorHdf5; ResultSetHdf is now default

This still requires a Context user to import sotodlib.io.metadata
before trying metdata loads.

* context: remove abandoned code in ManifestDb

* context: the Context can include a list of "imports"

This is useful to cause registration of data and metadata loaders for
Context to use.

* WIP: updates to context documentation.

* Avoid slow slice operation when merging axis managers

* Simplifiy and slightly generalize simplify_slice. It can now simplify along each dimension independently.

* Avoid confusing situations where attempted updates to members of the axismanager instead end up creating an entry in the axismanager object itself, that masks the actually managed one

* Fixed oversimplification of slice in simplify_slice.

* context: get_obs has a "logic_only" arg

And it causes the function to return the load specification (including
what dets) without loading any data or metadata.

* context/metadata: better handling of obs_colon_tags

Raises errors if parent field not found, duplicated.

* Docstrings for Superloader; pass in CelestialSightline to get_footprint

* coords.get_supergeom: get a supermap to hold submaps

* obsfiledb: fix how prefix is set when loading from context

* tod_ops: iir_filter function for readout filter conv/deconv

This sort of IIR filter can express Smurf (and MCE) readout filters.

* metadata: improve docstrings; create ABC for Metadata Loaders.

* docs: more context/metadata.

* context: tweaks to metadata loader

- When loading ManifestDb, map sqlite files as readonly without making
  a copy.  This is the fastest.
- colon_obs_tags: prepend keys with 'dets:' before passing to loader.
- SuperLoader caches ManifestDb connections.  This is probably of
  modest value for .sqlite on disk, but it helps a lot for .gz->memory
  databases.

* Add support for multiple common modes and randomized coupling strengths. Fix a recent bug preventing simulation of common modes. (#88)

* When running setup.py test, propagate the exit status back to the shell.

* Context: fix a test that broke due to SuperLoader db caching

* AxisManager: fix in_place=False bug, add a bunch of documentation.

* HWP demodulation (#87)

* Initial implememtation

* Add support for caching demodulated 2f component

* to dump demodulated data

Co-authored-by: Satoru Takakura <[email protected]>

* Implement h_n map operator (#91)

* initial implementation

* use the correct angle for h_n; add missing files

* Add cross-linking map support

* Use stationary detector weights

* h_n should be computed per detector

* draft instrument model revisions

* added new readout defaults

* changed wafer/wafers fields to wafer_slot/wafer_slots

* readout naming updates

* update detector band naming convention

* change detector naming scheme to detectors.w##_p###_bnd_A/B

* change wafer naming to w##, card slot to card_slot##, crate slot crate_slot##

* change wafer naming to w##, card slot to card_slot##, crate slot crate_slot##

* change tubes to tube_slots, add fields for tube_name and receiver_name (default blank)

* Add field for platform_name

* Update toast_so_example.py

* updated lat tube names to cryo convention

* SAT3 to SAT4

* update test naming conventions

* Rotate the LAT boresight according to observing elevation

* Move all toast-related source into a toast subpackage.

* Fix imports

* Fix default value

* Reduce size of simulated hardware test.

* Support recentering coordinates on given object or location

* Remove import added by mistake

* Support cuts in coords/pmat.py. Small rewrite in get_footprint for clarity.

Handling cuts in the pointing matrix this way isn't enough to suppor
them in the maximum-likelihood framework, but it makes the rest of the
implementation a bit cleaner. However, right now enabling pmat cuts
leads to some unexpected small changes in the result, which should be
investigated before they are used properly.\n\nThe get_footprint change
was made while I was debugging something else. What was there before was
fine too, but I left these changes in because I think it makes it a bit
easier to see the logic. While list comprehensions are very explicit,
their verbosity can make it hard to see the forest for all the trees.

* Hopefully clearer derivation of wcs in get_footprint

* Fixed argument order bugs in get_gap_fill

* coords: improve compatibility check in get_supergeom; add tests

Our tests initially included some supergeom arguments that should have
caused failure, but didn't... so now the supergeom compatibilty check
is a bit more complete than the pixell one.  Here we do not assume
that all cylindrical proj must have crval[1]==0.

* Add pixell to install_deps so test_coords can run

* Move sim_sso

* upgrade to new directory structure

* complete sso sim integration

* Add tool for measuring wafer and optics tube offsets

* Remove stale folder

Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Andrea Zonca <[email protected]>
Co-authored-by: Theodore Kisner <[email protected]>
Co-authored-by: Katie Harrington <[email protected]>
Co-authored-by: Yilun Guan <[email protected]>
Co-authored-by: Sigurd Kirkevold Næss <[email protected]>
Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Satoru Takakura <[email protected]>
Co-authored-by: smsimon <[email protected]>
keskitalo added a commit that referenced this pull request Feb 12, 2021
* Rotate the LAT focalplane based on elevation and encode the corotator angle in the so3g files

* Check export key for validity

* add TOAST MapMaker and FilterBin support to toast_so_sim.py (#102)

* Fix a recent bug introduced in writing demodulated TOD (#104)

* Initial implementation (#101)

* Sso sim (#103)

* add the new SSO simulation operator to sotodlib.observe

* Introduce core submodule, and core.AxisManager

* data/load_observation: return an AxisManager

* Changes to AxisManager... won't be the last...

* metadata module: for now this just imports everything from sotoddb

* Context: support for loading site/user/dataset-specific settings

* axisman: Bridge missing intersect1d feature in numpy < 1.15

Also fix a data scrambling bug, in passing.  Also catch bad
(non-disjoint) requests in from_resultset.

* Context.load_obs(obs_id)

To get that working, added string substitution (from context "tags"),
function to trigger database (re)load.  obs_id strings can include
detector identifiers such as 'f150'; such tags may be drawn from detdb
fields listed in context['obs_colon_tags'].

* data.load.load_observation: load directly into an AxisManager

* OBSLOADER_REGISTRY tracks named observation loading functions

And you tell it what loader you need through the context key
"obs_loader_type".  Both 'default' and 'pipe-s0001' point to
data.load.load_observation.  For now.

* Context.get_meta, can operate independently of get_obs

Also fix a couple of other bugs in passing

* data.load.load_file, as alternative load_observation

It can be used in cases where an ObsFileDb is not available, for
example.

* Context: load obsdb

* Context: obsdb / detdb / obsfiledb load is optional

* Whitespace fixes; no functional change

* add simple map binning pipeline to visualize so3g data dumps

* pass sinc-modulation argument to todground

* Modify matplotlib backend to PDF inside plot function

Closes #32

* docs: add docstring to set matplotlib backend function

* Invert phi to get azimuth in export

* Add pipeline for measuring the filter&bin transfer function

* tweak the transfer function pipeline

* The transfer function pipeline can now run with either Madam or the TOAST mapmaker

* support cosecant modulation in SO sims

* load_observation: permit user to request dets=[]

Sometimes you just want to look at the pointing.  This is not terribly
time efficient, since the whole file has to be read regardless.  But
we could imagine superior data formats where the ancillary data is
stored separately from the bolometers.

* Rearrange existing files to match (as much as possible)
the proposed layout in #53.  Only delete files that were
obviously empty or never used.  No changes or cleanup to
the code here, just moving things around and changing
imports so that unit tests pass.

* Fix imports in pipelines

* Move context and g3_core into core

* Adjust imports of sotoddb classes into new structure

This imports ResultSet and the metadata *Db directly to core.metadata,
renaming the *DB classes as *Db.  The more stuff-on-disk code in
sotodlib.simple is imported as io.metadata.simple.  This has been
tested on simons1 with the SO sim and ACT Uranus demo code.

* Fix imports so all tests pass

This includes a bug-fix in axisman that would only activate with old
numpy.

* Fix stray bad import

* fix pipeline_tools import

* fix small issues in TOAST load/export unit tests

* fix broken imports (#67)

* fix broken imports

* use relative imports

* more relative imports

* Trivial fix to unit tests

* Modifications to allow (#68)

* Modifications to allow
* Simulating atmosphere on top of loaded signal
* writing TOD from a pipeline that begins with loading TOD

* fix export when loading from file and using an export key

* Time constant (#52)

* Initial implementation of the operator and pipeline tools. Thanks to Katie Harrington for the code.

* unit test now passes

* fix name errors

* add missing import

* add unit test file

* Add units [s] to time constant doc strings

* Add parameters to leave a time constant residual

* fix time constant unit test

* Improve unit tests

* move time_constant.py

* fix import; add __init__.py

* more import fixes

* Fully functional time constant operator

* fix issue in pipeline_tools; correct the order of time constant and noise simulation

* remove obsolete directory

* add FlagManager to core imports

* add functionality to FlagManager

* grammar

* initial commits of TOD operations

* automatically add FlagManager to tod

* add 2 working flags

* add __setitem__ for fields only

* add simulation mechanim for glitches

* fix logic error on intersects

* flag updates & fixes

* expand the comments!

* updates for review

* Fix trivial typo

* use the new TOAST el-nod facility (#73)

* docs: update module paths in API reference

* context: fix a critical default behavior

get_obs() should refer to obsfiledb if user does not specify detsets.
Otherwise, all dets in the detdb will be try-loaded, and that's often
too many.

* context: accept a dict as obs_id argument to get_obs

This is so that a single row from obsdb.get() can be used with convenience.

* Context: get_obs accepts a list of ints for detsets

Also get_meta is documented and accepts a simple string obs_id.

* add copy constructor and update __repr__ to work for inheritance

* sotodlib.coords: pointing and coordinate support

Integrates so3g accelerated pointing code with tod-in-AxisManager.

* fix error in thinning the focal plane for the second band

* Make sure the observations get independent instances of the noise PSD (#75)

* Run unit tests with github actions.  This work:

  - Disables travis tests

  - Adds a simple github workflow that:
    - pulls the latest so3g image
    - pip-installs dependencies into the image and commits
      it to a local temporary image
    - runs the unit tests in this temporary image

* Change name of test workflow.

* Update test badge

* tod_ops.filters: add docstr and signature for fft_filter decorator

* Import of sotoddb code -- from sotodlib commit f4ee4167

docs and tests are still to come.

* metadata: fix SuperLoader Hdf5Loader issue

* metadata: rename DB to Db in main class definitions

This doesn't change the external interface.

* metadata: rename proddb.py to manifest.py

* metadata: import tests from sotoddb

* docs: in passing, re-fix name errors for sim/vis_hardware

* metadata: import sotoddb docs

* tod_ops: PCA module

Provides a few functions for decomposing tod covariance into
eigenbasis; reconstructing limited mode sets for signal cleaning; and
cleaning (subtraction of dominant modes).  Also an independent
function to find and remove "trends", and return them like they were
an set of modes with couplings to the signal.

* tod_ops: basic docs for filters and pca.

* docs: mock 'yaml', 'spt3g', 'pyfftw' if not found.

* docs: mock 'toml', 'sqlite3', 'so3g', 'toast', 'pixell' if not found.

* docs: rebrand SOTODLib -> sotodlib; set some autodoc options

* tod_ops: gap-filling (#81)

Introduces simple polynomial (linear, really) gap filling, as well as
"model-based" gap-filling that can use the results of a PCA (or common
mode).

The Extract and ExtractMatrix classes (the later of which is not quite
compatible with storage in AxisManager, but could be moved in that
direction) help with storing sparse subsets of a TOD signal
efficiently.

* core.metadata:  Clean up ObsDb code; document it.

This includes a minimal API for creating a usable ObsDb.  Common
database i/o functions created in core/metadata/common.py; other Dbs
will need to start using those.

* core.metadata: add "tags" functionality to ObsDb; unittest for ObsDb

Other minor cleanup as well.

* context: remove use of contextlib.nullcontext, since that needs >=3.7

* context: modernize ManifestDb and ObsFileDb with common to_file, from_file

* context: raise informative errors when failing to broadcast/match metadata

* tests/test_metadata.py: tests for the metadata resolution system

The test is especially needed now, since this system works but I want
to refactor the HDF5 stuff and still have it work the same...

* metadata: ManifestDb raises informative error when too many matches

* metadata: ResultSet.asarray has hdf_compat option

And that option causes strings to be coded as byte strings in the
output array, so the result can be written as an HDF5 dataset.

* io.metadata: ResultSetHdfLoader and write_dataset

PerDetectorHdf5 is now deprecated, but left in place as a shell
(probably not for long).  The ResultSetHdfLoader is tuned for the
context/metadata system, with a single API (the one hacked in through
PerDetectorHdf5.loader_class).  For writing datasets, a function makes
more sense than having a write_dataset method, assuming we don't want
to put HDF5 i/o right in the ResultSet class.

Also added a test for the array string conversion function.

* context/metadata: remove PerDetectorHdf5; ResultSetHdf is now default

This still requires a Context user to import sotodlib.io.metadata
before trying metdata loads.

* context: remove abandoned code in ManifestDb

* context: the Context can include a list of "imports"

This is useful to cause registration of data and metadata loaders for
Context to use.

* WIP: updates to context documentation.

* Avoid slow slice operation when merging axis managers

* Simplifiy and slightly generalize simplify_slice. It can now simplify along each dimension independently.

* Avoid confusing situations where attempted updates to members of the axismanager instead end up creating an entry in the axismanager object itself, that masks the actually managed one

* Fixed oversimplification of slice in simplify_slice.

* context: get_obs has a "logic_only" arg

And it causes the function to return the load specification (including
what dets) without loading any data or metadata.

* context/metadata: better handling of obs_colon_tags

Raises errors if parent field not found, duplicated.

* Docstrings for Superloader; pass in CelestialSightline to get_footprint

* coords.get_supergeom: get a supermap to hold submaps

* obsfiledb: fix how prefix is set when loading from context

* tod_ops: iir_filter function for readout filter conv/deconv

This sort of IIR filter can express Smurf (and MCE) readout filters.

* metadata: improve docstrings; create ABC for Metadata Loaders.

* docs: more context/metadata.

* context: tweaks to metadata loader

- When loading ManifestDb, map sqlite files as readonly without making
  a copy.  This is the fastest.
- colon_obs_tags: prepend keys with 'dets:' before passing to loader.
- SuperLoader caches ManifestDb connections.  This is probably of
  modest value for .sqlite on disk, but it helps a lot for .gz->memory
  databases.

* Add support for multiple common modes and randomized coupling strengths. Fix a recent bug preventing simulation of common modes. (#88)

* When running setup.py test, propagate the exit status back to the shell.

* Context: fix a test that broke due to SuperLoader db caching

* AxisManager: fix in_place=False bug, add a bunch of documentation.

* HWP demodulation (#87)

* Initial implememtation

* Add support for caching demodulated 2f component

* to dump demodulated data

Co-authored-by: Satoru Takakura <[email protected]>

* Implement h_n map operator (#91)

* initial implementation

* use the correct angle for h_n; add missing files

* Add cross-linking map support

* Use stationary detector weights

* h_n should be computed per detector

* draft instrument model revisions

* added new readout defaults

* changed wafer/wafers fields to wafer_slot/wafer_slots

* readout naming updates

* update detector band naming convention

* change detector naming scheme to detectors.w##_p###_bnd_A/B

* change wafer naming to w##, card slot to card_slot##, crate slot crate_slot##

* change wafer naming to w##, card slot to card_slot##, crate slot crate_slot##

* change tubes to tube_slots, add fields for tube_name and receiver_name (default blank)

* Add field for platform_name

* Update toast_so_example.py

* updated lat tube names to cryo convention

* SAT3 to SAT4

* update test naming conventions

* Rotate the LAT boresight according to observing elevation

* Move all toast-related source into a toast subpackage.

* Fix imports

* Fix default value

* Reduce size of simulated hardware test.

* Support recentering coordinates on given object or location

* Remove import added by mistake

* Support cuts in coords/pmat.py. Small rewrite in get_footprint for clarity.

Handling cuts in the pointing matrix this way isn't enough to suppor
them in the maximum-likelihood framework, but it makes the rest of the
implementation a bit cleaner. However, right now enabling pmat cuts
leads to some unexpected small changes in the result, which should be
investigated before they are used properly.\n\nThe get_footprint change
was made while I was debugging something else. What was there before was
fine too, but I left these changes in because I think it makes it a bit
easier to see the logic. While list comprehensions are very explicit,
their verbosity can make it hard to see the forest for all the trees.

* Hopefully clearer derivation of wcs in get_footprint

* Fixed argument order bugs in get_gap_fill

* coords: improve compatibility check in get_supergeom; add tests

Our tests initially included some supergeom arguments that should have
caused failure, but didn't... so now the supergeom compatibilty check
is a bit more complete than the pixell one.  Here we do not assume
that all cylindrical proj must have crval[1]==0.

* Add pixell to install_deps so test_coords can run

* Move sim_sso

* upgrade to new directory structure

* complete sso sim integration

* Add tool for measuring wafer and optics tube offsets

* Remove stale folder

Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Andrea Zonca <[email protected]>
Co-authored-by: Theodore Kisner <[email protected]>
Co-authored-by: Katie Harrington <[email protected]>
Co-authored-by: Yilun Guan <[email protected]>
Co-authored-by: Sigurd Kirkevold Næss <[email protected]>
Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Satoru Takakura <[email protected]>
Co-authored-by: smsimon <[email protected]>

* Add missing metadata required by atmospheric sims.

* Swap Az/El and return the radius of the FOV

* Rotate the LAT focalplane based on elevation and encode the corotator angle in the so3g files

* write the corotator angle into so3g files

* keep the cache object intact

* Now receiver compensates for elevation rotation by default

* Fix default corotator angle

* Load the corotator angle when available

* TOAST should also load the corotator angle

Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Andrea Zonca <[email protected]>
Co-authored-by: Theodore Kisner <[email protected]>
Co-authored-by: Katie Harrington <[email protected]>
Co-authored-by: Yilun Guan <[email protected]>
Co-authored-by: Sigurd Kirkevold Næss <[email protected]>
Co-authored-by: Matthew Hasselfield <[email protected]>
Co-authored-by: Satoru Takakura <[email protected]>
Co-authored-by: smsimon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant