Revisit subpackage organization #53

tskisner · 2020-04-09T16:55:10Z

Recent changes have shown us that we should revisit the big-picture organization of sotodlib. @mhasself has floated the idea of bringing the sotoddb repo into this package under the meta subpackage. This issue is for tracking the summary of offline discussion and thoughts about this before we take any action.

The text was updated successfully, but these errors were encountered:

kmharrington · 2020-04-09T17:27:36Z

A couple thoughts: I've noticed it looks like there's going to be a lot of toast-related functions and a lot of Axis-manager related functions. Also, things "needed for simulations" and things "needed for analysis of simulations/real data" are pretty interspersed. I'd be inclined to separate those into top-level modules as well.

And then there's some random stuff I wrote awhile ago that's probably just deprecated already.

tskisner · 2020-04-09T18:17:00Z

Yeah... @mhasself and I started to think about this offline, but are going to first work on defining the AxisManager / TOAST interface a little better. I think that will help inform the best way of organizing the sub packages.

tskisner · 2020-04-16T02:19:13Z

Ok, after some more big-picture discussion offline, I think we are going to simplify the package structure and a proposed draft of the new organization is:

sotodlib.core : Things like data container classes, metadata containers, etc.
sotodlib.scripts : Entrypoints (*.py files which define a "main()" and which get
        installed by setup.py)
sotodlib.io : Code for doing I/O.  Specifically this is code for reading and 
        writing on-disk data (G3 and auxilliary data) and any metadata DB access.
sotodlib.utils : Code for helper operations and any data format conversions
        if needed.
sotodlib.* : "Analysis" tools that work with data can go in files at the top level of
        the package, at least until there are too many of them that it makes sense
        to create subdirectories.

Just a note that this plan involves more than just moving source files around. For example the functionality of the sotoddb package would get split into the "container" definition and the "I/O" (DB access). Also existing tools like the hardware layout simulation would get turned into a module at the top level that simulates the hardware and populates a "metadata container".

Because these changes involve several steps, I suggest we keep this issue as the "meta" issue discussing the big picture and then open smaller issues to track the intermediate steps. I'll start that process shortly.

tskisner · 2020-04-16T02:36:34Z

Just a note that although we are starting with the all the code that "does stuff" in the root of the package, we do still need to have some policies about interdependencies, which is what I think @kmharrington was getting at earlier in this thread. Any *.py file in the root of the package can safely import from the subpackages like:

from .core import AxisManager
from .utils import OtherUsefulClass

But we must avoid circular dependencies between these top-level *.py files. Hopefully documenting that policy combined with catching things during review of PRs will be sufficient.

the proposed layout in #53. Only delete files that were obviously empty or never used. No changes or cleanup to the code here, just moving things around and changing imports so that unit tests pass.

tskisner · 2020-10-28T17:25:31Z

Revisiting this issue now that there are tod_ops and modules directories... I have a proposal at the bottom of this post for those in a hurry :-) I'll try to summarize multiple previous discussions in this issue. I think the conclusion was that there will be the following categories:

(core) Classes and functions for in-memory representation of data and metadata. This includes the Context stuff, the AxisManager data container, and the Hardware class for synthetic instrument properties (i.e. before we have a database of real instrument properties). Note: since the underlying sqlite DBs are tightly coupled to these classes, they live here rather than in "io", even though they are technically touching disk. Note that in the case of TOAST, the Observation class is defined externally in the toast package, so we don't need to define it here (but this is where it would go if we did).
(io) Classes and functions for reading and writing data. This includes both AxisManager loading and also TOAST export / load operators.
(utils) Any auxiliary helper functions that do not logically go somewhere else. For example, when writing G3 frames from toast, there are some helper functions that live in io, close to the export code, which makes sense. Currently we have some toast pipeline helper functions in here, but most of that will be built in to the operators themselves in toast3.
(scripts) Entry points defined in setup.py. These are designed to be small stand-alone scripts that do specific things. Examples might be: script to query available local data products that can be loaded, scripts to manipulate synthetic instrument files, etc. These scripts should be focused on doing things directly relevant to the tools in sotodlib. Large workflows should go somewhere else (like the pipelines top level directory or in pwg-scripts, etc).
Other analysis tools. This includes both timestream processing tools and potentially tools that do things in the map domain. For timestream processing there will code working with both AxisManagers and toast observations.

Right now, we seem to be pushing AxisManager operations into tod_ops and TOAST operations into modules. And then we have several other things at the top level of the package. Rather than split code by people who are working on things, I think it would make our software more cohesive if we could split by type of operations. Here is a concrete proposal for consideration:

sim: Anything relating to simulations, including simulated hardware and any AxisManager or TOAST operations that simulate timestreams. If you just load (real or already-simulated) data off disk and do stuff with it, then you should not need to import this.
ops: Any AxisManager or TOAST operations for working with real or already-simulated data.
"top-level": Everyone seems to hate putting stuff in the top-level of the package, which is fine, but let's just clear it out and put code into sim or ops.
If we find that we want to put map-making related tools in sotodlib, we could make a new subpackage or just leave them in ops.

Thoughts / feedback are welcome, especially from @mhasself, @kmharrington and @keskitalo

kmharrington · 2020-10-29T14:24:19Z

I wonder about mixing the TOAST operators with the AxisManager ones. At some level, it's nice to load a package and know that all the functions in that package work on your data model. Even if we keep subpackages "all TOAST" or "all AxisManagers" you'll have to know which ones use.

Where do we want the tree to be?

ops
- ops.toast
- ops.aman
sim
- sim.toast
- sim.aman

Versus

toast
- toast.sim
- toast.ops
aman
- aman.sim
- aman.ops

tskisner · 2020-10-29T15:00:05Z

So if I have some common math functions used by operators in both frameworks, where would that go? If we have the sim and ops packages at the top level, then it would clearly go in the ops directory close to both of the operators that use it. Will we ever have a toast operator that calls axismanager ops? If so, where would that go, since it uses both frameworks. Currently we will have that situation in at least one place, but that will be in the io subpackage, so not a great example.

If we must "silo" the code in this way, then my preference is ops.toast and ops.aman. Then at least the common math functions independent of framework can go in the ops package. Same with any operators that use both frameworks.

So far, most of the functionality in the current tod_ops package is pretty trivial, so likely I will just port those to toast operators directly. However, at some point there may be more complicated processing functions shared between the two data models and I want to make sure there is a place for them.

mhasself · 2020-10-29T16:11:58Z

I tend to agree with @kmharrington on this. People will be working with TOAST representation, or axisman representation. The functions compatible with a particular TOD representation should be close together. Functions that aren't tied specifically to a representation should be elsewhere (utils?) so that they are available to either framework.

I think we might spend next week's PWG telecon talking about this.

tskisner · 2020-10-29T16:30:05Z

Ok, that's fine- but @kmharrington put forth two options for organization. I like the option with a top-level "category" (sim and ops) so that at least we can ignore simulation stuff when we are working with data from disk.

the proposed layout in #53. Only delete files that were obviously empty or never used. No changes or cleanup to the code here, just moving things around and changing imports so that unit tests pass.

* add the new SSO simulation operator to sotodlib.observe * Introduce core submodule, and core.AxisManager * data/load_observation: return an AxisManager * Changes to AxisManager... won't be the last... * metadata module: for now this just imports everything from sotoddb * Context: support for loading site/user/dataset-specific settings * axisman: Bridge missing intersect1d feature in numpy < 1.15 Also fix a data scrambling bug, in passing. Also catch bad (non-disjoint) requests in from_resultset. * Context.load_obs(obs_id) To get that working, added string substitution (from context "tags"), function to trigger database (re)load. obs_id strings can include detector identifiers such as 'f150'; such tags may be drawn from detdb fields listed in context['obs_colon_tags']. * data.load.load_observation: load directly into an AxisManager * OBSLOADER_REGISTRY tracks named observation loading functions And you tell it what loader you need through the context key "obs_loader_type". Both 'default' and 'pipe-s0001' point to data.load.load_observation. For now. * Context.get_meta, can operate independently of get_obs Also fix a couple of other bugs in passing * data.load.load_file, as alternative load_observation It can be used in cases where an ObsFileDb is not available, for example. * Context: load obsdb * Context: obsdb / detdb / obsfiledb load is optional * Whitespace fixes; no functional change * add simple map binning pipeline to visualize so3g data dumps * pass sinc-modulation argument to todground * Modify matplotlib backend to PDF inside plot function Closes #32 * docs: add docstring to set matplotlib backend function * Invert phi to get azimuth in export * Add pipeline for measuring the filter&bin transfer function * tweak the transfer function pipeline * The transfer function pipeline can now run with either Madam or the TOAST mapmaker * support cosecant modulation in SO sims * load_observation: permit user to request dets=[] Sometimes you just want to look at the pointing. This is not terribly time efficient, since the whole file has to be read regardless. But we could imagine superior data formats where the ancillary data is stored separately from the bolometers. * Rearrange existing files to match (as much as possible) the proposed layout in #53. Only delete files that were obviously empty or never used. No changes or cleanup to the code here, just moving things around and changing imports so that unit tests pass. * Fix imports in pipelines * Move context and g3_core into core * Adjust imports of sotoddb classes into new structure This imports ResultSet and the metadata *Db directly to core.metadata, renaming the *DB classes as *Db. The more stuff-on-disk code in sotodlib.simple is imported as io.metadata.simple. This has been tested on simons1 with the SO sim and ACT Uranus demo code. * Fix imports so all tests pass This includes a bug-fix in axisman that would only activate with old numpy. * Fix stray bad import * fix pipeline_tools import * fix small issues in TOAST load/export unit tests * fix broken imports (#67) * fix broken imports * use relative imports * more relative imports * Trivial fix to unit tests * Modifications to allow (#68) * Modifications to allow * Simulating atmosphere on top of loaded signal * writing TOD from a pipeline that begins with loading TOD * fix export when loading from file and using an export key * Time constant (#52) * Initial implementation of the operator and pipeline tools. Thanks to Katie Harrington for the code. * unit test now passes * fix name errors * add missing import * add unit test file * Add units [s] to time constant doc strings * Add parameters to leave a time constant residual * fix time constant unit test * Improve unit tests * move time_constant.py * fix import; add __init__.py * more import fixes * Fully functional time constant operator * fix issue in pipeline_tools; correct the order of time constant and noise simulation * remove obsolete directory * add FlagManager to core imports * add functionality to FlagManager * grammar * initial commits of TOD operations * automatically add FlagManager to tod * add 2 working flags * add __setitem__ for fields only * add simulation mechanim for glitches * fix logic error on intersects * flag updates & fixes * expand the comments! * updates for review * Fix trivial typo * use the new TOAST el-nod facility (#73) * docs: update module paths in API reference * context: fix a critical default behavior get_obs() should refer to obsfiledb if user does not specify detsets. Otherwise, all dets in the detdb will be try-loaded, and that's often too many. * context: accept a dict as obs_id argument to get_obs This is so that a single row from obsdb.get() can be used with convenience. * Context: get_obs accepts a list of ints for detsets Also get_meta is documented and accepts a simple string obs_id. * add copy constructor and update __repr__ to work for inheritance * sotodlib.coords: pointing and coordinate support Integrates so3g accelerated pointing code with tod-in-AxisManager. * fix error in thinning the focal plane for the second band * Make sure the observations get independent instances of the noise PSD (#75) * Run unit tests with github actions. This work: - Disables travis tests - Adds a simple github workflow that: - pulls the latest so3g image - pip-installs dependencies into the image and commits it to a local temporary image - runs the unit tests in this temporary image * Change name of test workflow. * Update test badge * tod_ops.filters: add docstr and signature for fft_filter decorator * Import of sotoddb code -- from sotodlib commit f4ee4167 docs and tests are still to come. * metadata: fix SuperLoader Hdf5Loader issue * metadata: rename DB to Db in main class definitions This doesn't change the external interface. * metadata: rename proddb.py to manifest.py * metadata: import tests from sotoddb * docs: in passing, re-fix name errors for sim/vis_hardware * metadata: import sotoddb docs * tod_ops: PCA module Provides a few functions for decomposing tod covariance into eigenbasis; reconstructing limited mode sets for signal cleaning; and cleaning (subtraction of dominant modes). Also an independent function to find and remove "trends", and return them like they were an set of modes with couplings to the signal. * tod_ops: basic docs for filters and pca. * docs: mock 'yaml', 'spt3g', 'pyfftw' if not found. * docs: mock 'toml', 'sqlite3', 'so3g', 'toast', 'pixell' if not found. * docs: rebrand SOTODLib -> sotodlib; set some autodoc options * tod_ops: gap-filling (#81) Introduces simple polynomial (linear, really) gap filling, as well as "model-based" gap-filling that can use the results of a PCA (or common mode). The Extract and ExtractMatrix classes (the later of which is not quite compatible with storage in AxisManager, but could be moved in that direction) help with storing sparse subsets of a TOD signal efficiently. * core.metadata: Clean up ObsDb code; document it. This includes a minimal API for creating a usable ObsDb. Common database i/o functions created in core/metadata/common.py; other Dbs will need to start using those. * core.metadata: add "tags" functionality to ObsDb; unittest for ObsDb Other minor cleanup as well. * context: remove use of contextlib.nullcontext, since that needs >=3.7 * context: modernize ManifestDb and ObsFileDb with common to_file, from_file * context: raise informative errors when failing to broadcast/match metadata * tests/test_metadata.py: tests for the metadata resolution system The test is especially needed now, since this system works but I want to refactor the HDF5 stuff and still have it work the same... * metadata: ManifestDb raises informative error when too many matches * metadata: ResultSet.asarray has hdf_compat option And that option causes strings to be coded as byte strings in the output array, so the result can be written as an HDF5 dataset. * io.metadata: ResultSetHdfLoader and write_dataset PerDetectorHdf5 is now deprecated, but left in place as a shell (probably not for long). The ResultSetHdfLoader is tuned for the context/metadata system, with a single API (the one hacked in through PerDetectorHdf5.loader_class). For writing datasets, a function makes more sense than having a write_dataset method, assuming we don't want to put HDF5 i/o right in the ResultSet class. Also added a test for the array string conversion function. * context/metadata: remove PerDetectorHdf5; ResultSetHdf is now default This still requires a Context user to import sotodlib.io.metadata before trying metdata loads. * context: remove abandoned code in ManifestDb * context: the Context can include a list of "imports" This is useful to cause registration of data and metadata loaders for Context to use. * WIP: updates to context documentation. * Avoid slow slice operation when merging axis managers * Simplifiy and slightly generalize simplify_slice. It can now simplify along each dimension independently. * Avoid confusing situations where attempted updates to members of the axismanager instead end up creating an entry in the axismanager object itself, that masks the actually managed one * Fixed oversimplification of slice in simplify_slice. * context: get_obs has a "logic_only" arg And it causes the function to return the load specification (including what dets) without loading any data or metadata. * context/metadata: better handling of obs_colon_tags Raises errors if parent field not found, duplicated. * Docstrings for Superloader; pass in CelestialSightline to get_footprint * coords.get_supergeom: get a supermap to hold submaps * obsfiledb: fix how prefix is set when loading from context * tod_ops: iir_filter function for readout filter conv/deconv This sort of IIR filter can express Smurf (and MCE) readout filters. * metadata: improve docstrings; create ABC for Metadata Loaders. * docs: more context/metadata. * context: tweaks to metadata loader - When loading ManifestDb, map sqlite files as readonly without making a copy. This is the fastest. - colon_obs_tags: prepend keys with 'dets:' before passing to loader. - SuperLoader caches ManifestDb connections. This is probably of modest value for .sqlite on disk, but it helps a lot for .gz->memory databases. * Add support for multiple common modes and randomized coupling strengths. Fix a recent bug preventing simulation of common modes. (#88) * When running setup.py test, propagate the exit status back to the shell. * Context: fix a test that broke due to SuperLoader db caching * AxisManager: fix in_place=False bug, add a bunch of documentation. * HWP demodulation (#87) * Initial implememtation * Add support for caching demodulated 2f component * to dump demodulated data Co-authored-by: Satoru Takakura <[email protected]> * Implement h_n map operator (#91) * initial implementation * use the correct angle for h_n; add missing files * Add cross-linking map support * Use stationary detector weights * h_n should be computed per detector * draft instrument model revisions * added new readout defaults * changed wafer/wafers fields to wafer_slot/wafer_slots * readout naming updates * update detector band naming convention * change detector naming scheme to detectors.w##_p###_bnd_A/B * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change tubes to tube_slots, add fields for tube_name and receiver_name (default blank) * Add field for platform_name * Update toast_so_example.py * updated lat tube names to cryo convention * SAT3 to SAT4 * update test naming conventions * Rotate the LAT boresight according to observing elevation * Move all toast-related source into a toast subpackage. * Fix imports * Fix default value * Reduce size of simulated hardware test. * Support recentering coordinates on given object or location * Remove import added by mistake * Support cuts in coords/pmat.py. Small rewrite in get_footprint for clarity. Handling cuts in the pointing matrix this way isn't enough to suppor them in the maximum-likelihood framework, but it makes the rest of the implementation a bit cleaner. However, right now enabling pmat cuts leads to some unexpected small changes in the result, which should be investigated before they are used properly.\n\nThe get_footprint change was made while I was debugging something else. What was there before was fine too, but I left these changes in because I think it makes it a bit easier to see the logic. While list comprehensions are very explicit, their verbosity can make it hard to see the forest for all the trees. * Hopefully clearer derivation of wcs in get_footprint * Fixed argument order bugs in get_gap_fill * coords: improve compatibility check in get_supergeom; add tests Our tests initially included some supergeom arguments that should have caused failure, but didn't... so now the supergeom compatibilty check is a bit more complete than the pixell one. Here we do not assume that all cylindrical proj must have crval[1]==0. * Add pixell to install_deps so test_coords can run * Move sim_sso * upgrade to new directory structure * complete sso sim integration * Add tool for measuring wafer and optics tube offsets * Remove stale folder Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Andrea Zonca <[email protected]> Co-authored-by: Theodore Kisner <[email protected]> Co-authored-by: Katie Harrington <[email protected]> Co-authored-by: Yilun Guan <[email protected]> Co-authored-by: Sigurd Kirkevold Næss <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Satoru Takakura <[email protected]> Co-authored-by: smsimon <[email protected]>

tskisner · 2021-02-03T17:32:05Z

closed by #96

* add the new SSO simulation operator to sotodlib.observe * Introduce core submodule, and core.AxisManager * data/load_observation: return an AxisManager * Changes to AxisManager... won't be the last... * metadata module: for now this just imports everything from sotoddb * Context: support for loading site/user/dataset-specific settings * axisman: Bridge missing intersect1d feature in numpy < 1.15 Also fix a data scrambling bug, in passing. Also catch bad (non-disjoint) requests in from_resultset. * Context.load_obs(obs_id) To get that working, added string substitution (from context "tags"), function to trigger database (re)load. obs_id strings can include detector identifiers such as 'f150'; such tags may be drawn from detdb fields listed in context['obs_colon_tags']. * data.load.load_observation: load directly into an AxisManager * OBSLOADER_REGISTRY tracks named observation loading functions And you tell it what loader you need through the context key "obs_loader_type". Both 'default' and 'pipe-s0001' point to data.load.load_observation. For now. * Context.get_meta, can operate independently of get_obs Also fix a couple of other bugs in passing * data.load.load_file, as alternative load_observation It can be used in cases where an ObsFileDb is not available, for example. * Context: load obsdb * Context: obsdb / detdb / obsfiledb load is optional * Whitespace fixes; no functional change * add simple map binning pipeline to visualize so3g data dumps * pass sinc-modulation argument to todground * Modify matplotlib backend to PDF inside plot function Closes #32 * docs: add docstring to set matplotlib backend function * Invert phi to get azimuth in export * Add pipeline for measuring the filter&bin transfer function * tweak the transfer function pipeline * The transfer function pipeline can now run with either Madam or the TOAST mapmaker * support cosecant modulation in SO sims * load_observation: permit user to request dets=[] Sometimes you just want to look at the pointing. This is not terribly time efficient, since the whole file has to be read regardless. But we could imagine superior data formats where the ancillary data is stored separately from the bolometers. * Rearrange existing files to match (as much as possible) the proposed layout in #53. Only delete files that were obviously empty or never used. No changes or cleanup to the code here, just moving things around and changing imports so that unit tests pass. * Fix imports in pipelines * Move context and g3_core into core * Adjust imports of sotoddb classes into new structure This imports ResultSet and the metadata *Db directly to core.metadata, renaming the *DB classes as *Db. The more stuff-on-disk code in sotodlib.simple is imported as io.metadata.simple. This has been tested on simons1 with the SO sim and ACT Uranus demo code. * Fix imports so all tests pass This includes a bug-fix in axisman that would only activate with old numpy. * Fix stray bad import * fix pipeline_tools import * fix small issues in TOAST load/export unit tests * fix broken imports (#67) * fix broken imports * use relative imports * more relative imports * Trivial fix to unit tests * Modifications to allow (#68) * Modifications to allow * Simulating atmosphere on top of loaded signal * writing TOD from a pipeline that begins with loading TOD * fix export when loading from file and using an export key * Time constant (#52) * Initial implementation of the operator and pipeline tools. Thanks to Katie Harrington for the code. * unit test now passes * fix name errors * add missing import * add unit test file * Add units [s] to time constant doc strings * Add parameters to leave a time constant residual * fix time constant unit test * Improve unit tests * move time_constant.py * fix import; add __init__.py * more import fixes * Fully functional time constant operator * fix issue in pipeline_tools; correct the order of time constant and noise simulation * remove obsolete directory * add FlagManager to core imports * add functionality to FlagManager * grammar * initial commits of TOD operations * automatically add FlagManager to tod * add 2 working flags * add __setitem__ for fields only * add simulation mechanim for glitches * fix logic error on intersects * flag updates & fixes * expand the comments! * updates for review * Fix trivial typo * use the new TOAST el-nod facility (#73) * docs: update module paths in API reference * context: fix a critical default behavior get_obs() should refer to obsfiledb if user does not specify detsets. Otherwise, all dets in the detdb will be try-loaded, and that's often too many. * context: accept a dict as obs_id argument to get_obs This is so that a single row from obsdb.get() can be used with convenience. * Context: get_obs accepts a list of ints for detsets Also get_meta is documented and accepts a simple string obs_id. * add copy constructor and update __repr__ to work for inheritance * sotodlib.coords: pointing and coordinate support Integrates so3g accelerated pointing code with tod-in-AxisManager. * fix error in thinning the focal plane for the second band * Make sure the observations get independent instances of the noise PSD (#75) * Run unit tests with github actions. This work: - Disables travis tests - Adds a simple github workflow that: - pulls the latest so3g image - pip-installs dependencies into the image and commits it to a local temporary image - runs the unit tests in this temporary image * Change name of test workflow. * Update test badge * tod_ops.filters: add docstr and signature for fft_filter decorator * Import of sotoddb code -- from sotodlib commit f4ee4167 docs and tests are still to come. * metadata: fix SuperLoader Hdf5Loader issue * metadata: rename DB to Db in main class definitions This doesn't change the external interface. * metadata: rename proddb.py to manifest.py * metadata: import tests from sotoddb * docs: in passing, re-fix name errors for sim/vis_hardware * metadata: import sotoddb docs * tod_ops: PCA module Provides a few functions for decomposing tod covariance into eigenbasis; reconstructing limited mode sets for signal cleaning; and cleaning (subtraction of dominant modes). Also an independent function to find and remove "trends", and return them like they were an set of modes with couplings to the signal. * tod_ops: basic docs for filters and pca. * docs: mock 'yaml', 'spt3g', 'pyfftw' if not found. * docs: mock 'toml', 'sqlite3', 'so3g', 'toast', 'pixell' if not found. * docs: rebrand SOTODLib -> sotodlib; set some autodoc options * tod_ops: gap-filling (#81) Introduces simple polynomial (linear, really) gap filling, as well as "model-based" gap-filling that can use the results of a PCA (or common mode). The Extract and ExtractMatrix classes (the later of which is not quite compatible with storage in AxisManager, but could be moved in that direction) help with storing sparse subsets of a TOD signal efficiently. * core.metadata: Clean up ObsDb code; document it. This includes a minimal API for creating a usable ObsDb. Common database i/o functions created in core/metadata/common.py; other Dbs will need to start using those. * core.metadata: add "tags" functionality to ObsDb; unittest for ObsDb Other minor cleanup as well. * context: remove use of contextlib.nullcontext, since that needs >=3.7 * context: modernize ManifestDb and ObsFileDb with common to_file, from_file * context: raise informative errors when failing to broadcast/match metadata * tests/test_metadata.py: tests for the metadata resolution system The test is especially needed now, since this system works but I want to refactor the HDF5 stuff and still have it work the same... * metadata: ManifestDb raises informative error when too many matches * metadata: ResultSet.asarray has hdf_compat option And that option causes strings to be coded as byte strings in the output array, so the result can be written as an HDF5 dataset. * io.metadata: ResultSetHdfLoader and write_dataset PerDetectorHdf5 is now deprecated, but left in place as a shell (probably not for long). The ResultSetHdfLoader is tuned for the context/metadata system, with a single API (the one hacked in through PerDetectorHdf5.loader_class). For writing datasets, a function makes more sense than having a write_dataset method, assuming we don't want to put HDF5 i/o right in the ResultSet class. Also added a test for the array string conversion function. * context/metadata: remove PerDetectorHdf5; ResultSetHdf is now default This still requires a Context user to import sotodlib.io.metadata before trying metdata loads. * context: remove abandoned code in ManifestDb * context: the Context can include a list of "imports" This is useful to cause registration of data and metadata loaders for Context to use. * WIP: updates to context documentation. * Avoid slow slice operation when merging axis managers * Simplifiy and slightly generalize simplify_slice. It can now simplify along each dimension independently. * Avoid confusing situations where attempted updates to members of the axismanager instead end up creating an entry in the axismanager object itself, that masks the actually managed one * Fixed oversimplification of slice in simplify_slice. * context: get_obs has a "logic_only" arg And it causes the function to return the load specification (including what dets) without loading any data or metadata. * context/metadata: better handling of obs_colon_tags Raises errors if parent field not found, duplicated. * Docstrings for Superloader; pass in CelestialSightline to get_footprint * coords.get_supergeom: get a supermap to hold submaps * obsfiledb: fix how prefix is set when loading from context * tod_ops: iir_filter function for readout filter conv/deconv This sort of IIR filter can express Smurf (and MCE) readout filters. * metadata: improve docstrings; create ABC for Metadata Loaders. * docs: more context/metadata. * context: tweaks to metadata loader - When loading ManifestDb, map sqlite files as readonly without making a copy. This is the fastest. - colon_obs_tags: prepend keys with 'dets:' before passing to loader. - SuperLoader caches ManifestDb connections. This is probably of modest value for .sqlite on disk, but it helps a lot for .gz->memory databases. * Add support for multiple common modes and randomized coupling strengths. Fix a recent bug preventing simulation of common modes. (#88) * When running setup.py test, propagate the exit status back to the shell. * Context: fix a test that broke due to SuperLoader db caching * AxisManager: fix in_place=False bug, add a bunch of documentation. * HWP demodulation (#87) * Initial implememtation * Add support for caching demodulated 2f component * to dump demodulated data Co-authored-by: Satoru Takakura <[email protected]> * Implement h_n map operator (#91) * initial implementation * use the correct angle for h_n; add missing files * Add cross-linking map support * Use stationary detector weights * h_n should be computed per detector * draft instrument model revisions * added new readout defaults * changed wafer/wafers fields to wafer_slot/wafer_slots * readout naming updates * update detector band naming convention * change detector naming scheme to detectors.w##_p###_bnd_A/B * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change tubes to tube_slots, add fields for tube_name and receiver_name (default blank) * Add field for platform_name * Update toast_so_example.py * updated lat tube names to cryo convention * SAT3 to SAT4 * update test naming conventions * Rotate the LAT boresight according to observing elevation * Move all toast-related source into a toast subpackage. * Fix imports * Fix default value * Reduce size of simulated hardware test. * Support recentering coordinates on given object or location * Remove import added by mistake * Support cuts in coords/pmat.py. Small rewrite in get_footprint for clarity. Handling cuts in the pointing matrix this way isn't enough to suppor them in the maximum-likelihood framework, but it makes the rest of the implementation a bit cleaner. However, right now enabling pmat cuts leads to some unexpected small changes in the result, which should be investigated before they are used properly.\n\nThe get_footprint change was made while I was debugging something else. What was there before was fine too, but I left these changes in because I think it makes it a bit easier to see the logic. While list comprehensions are very explicit, their verbosity can make it hard to see the forest for all the trees. * Hopefully clearer derivation of wcs in get_footprint * Fixed argument order bugs in get_gap_fill * coords: improve compatibility check in get_supergeom; add tests Our tests initially included some supergeom arguments that should have caused failure, but didn't... so now the supergeom compatibilty check is a bit more complete than the pixell one. Here we do not assume that all cylindrical proj must have crval[1]==0. * Add pixell to install_deps so test_coords can run * Move sim_sso * upgrade to new directory structure * complete sso sim integration * Add tool for measuring wafer and optics tube offsets * Remove stale folder Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Andrea Zonca <[email protected]> Co-authored-by: Theodore Kisner <[email protected]> Co-authored-by: Katie Harrington <[email protected]> Co-authored-by: Yilun Guan <[email protected]> Co-authored-by: Sigurd Kirkevold Næss <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Satoru Takakura <[email protected]> Co-authored-by: smsimon <[email protected]>

* Rotate the LAT focalplane based on elevation and encode the corotator angle in the so3g files * Check export key for validity * add TOAST MapMaker and FilterBin support to toast_so_sim.py (#102) * Fix a recent bug introduced in writing demodulated TOD (#104) * Initial implementation (#101) * Sso sim (#103) * add the new SSO simulation operator to sotodlib.observe * Introduce core submodule, and core.AxisManager * data/load_observation: return an AxisManager * Changes to AxisManager... won't be the last... * metadata module: for now this just imports everything from sotoddb * Context: support for loading site/user/dataset-specific settings * axisman: Bridge missing intersect1d feature in numpy < 1.15 Also fix a data scrambling bug, in passing. Also catch bad (non-disjoint) requests in from_resultset. * Context.load_obs(obs_id) To get that working, added string substitution (from context "tags"), function to trigger database (re)load. obs_id strings can include detector identifiers such as 'f150'; such tags may be drawn from detdb fields listed in context['obs_colon_tags']. * data.load.load_observation: load directly into an AxisManager * OBSLOADER_REGISTRY tracks named observation loading functions And you tell it what loader you need through the context key "obs_loader_type". Both 'default' and 'pipe-s0001' point to data.load.load_observation. For now. * Context.get_meta, can operate independently of get_obs Also fix a couple of other bugs in passing * data.load.load_file, as alternative load_observation It can be used in cases where an ObsFileDb is not available, for example. * Context: load obsdb * Context: obsdb / detdb / obsfiledb load is optional * Whitespace fixes; no functional change * add simple map binning pipeline to visualize so3g data dumps * pass sinc-modulation argument to todground * Modify matplotlib backend to PDF inside plot function Closes #32 * docs: add docstring to set matplotlib backend function * Invert phi to get azimuth in export * Add pipeline for measuring the filter&bin transfer function * tweak the transfer function pipeline * The transfer function pipeline can now run with either Madam or the TOAST mapmaker * support cosecant modulation in SO sims * load_observation: permit user to request dets=[] Sometimes you just want to look at the pointing. This is not terribly time efficient, since the whole file has to be read regardless. But we could imagine superior data formats where the ancillary data is stored separately from the bolometers. * Rearrange existing files to match (as much as possible) the proposed layout in #53. Only delete files that were obviously empty or never used. No changes or cleanup to the code here, just moving things around and changing imports so that unit tests pass. * Fix imports in pipelines * Move context and g3_core into core * Adjust imports of sotoddb classes into new structure This imports ResultSet and the metadata *Db directly to core.metadata, renaming the *DB classes as *Db. The more stuff-on-disk code in sotodlib.simple is imported as io.metadata.simple. This has been tested on simons1 with the SO sim and ACT Uranus demo code. * Fix imports so all tests pass This includes a bug-fix in axisman that would only activate with old numpy. * Fix stray bad import * fix pipeline_tools import * fix small issues in TOAST load/export unit tests * fix broken imports (#67) * fix broken imports * use relative imports * more relative imports * Trivial fix to unit tests * Modifications to allow (#68) * Modifications to allow * Simulating atmosphere on top of loaded signal * writing TOD from a pipeline that begins with loading TOD * fix export when loading from file and using an export key * Time constant (#52) * Initial implementation of the operator and pipeline tools. Thanks to Katie Harrington for the code. * unit test now passes * fix name errors * add missing import * add unit test file * Add units [s] to time constant doc strings * Add parameters to leave a time constant residual * fix time constant unit test * Improve unit tests * move time_constant.py * fix import; add __init__.py * more import fixes * Fully functional time constant operator * fix issue in pipeline_tools; correct the order of time constant and noise simulation * remove obsolete directory * add FlagManager to core imports * add functionality to FlagManager * grammar * initial commits of TOD operations * automatically add FlagManager to tod * add 2 working flags * add __setitem__ for fields only * add simulation mechanim for glitches * fix logic error on intersects * flag updates & fixes * expand the comments! * updates for review * Fix trivial typo * use the new TOAST el-nod facility (#73) * docs: update module paths in API reference * context: fix a critical default behavior get_obs() should refer to obsfiledb if user does not specify detsets. Otherwise, all dets in the detdb will be try-loaded, and that's often too many. * context: accept a dict as obs_id argument to get_obs This is so that a single row from obsdb.get() can be used with convenience. * Context: get_obs accepts a list of ints for detsets Also get_meta is documented and accepts a simple string obs_id. * add copy constructor and update __repr__ to work for inheritance * sotodlib.coords: pointing and coordinate support Integrates so3g accelerated pointing code with tod-in-AxisManager. * fix error in thinning the focal plane for the second band * Make sure the observations get independent instances of the noise PSD (#75) * Run unit tests with github actions. This work: - Disables travis tests - Adds a simple github workflow that: - pulls the latest so3g image - pip-installs dependencies into the image and commits it to a local temporary image - runs the unit tests in this temporary image * Change name of test workflow. * Update test badge * tod_ops.filters: add docstr and signature for fft_filter decorator * Import of sotoddb code -- from sotodlib commit f4ee4167 docs and tests are still to come. * metadata: fix SuperLoader Hdf5Loader issue * metadata: rename DB to Db in main class definitions This doesn't change the external interface. * metadata: rename proddb.py to manifest.py * metadata: import tests from sotoddb * docs: in passing, re-fix name errors for sim/vis_hardware * metadata: import sotoddb docs * tod_ops: PCA module Provides a few functions for decomposing tod covariance into eigenbasis; reconstructing limited mode sets for signal cleaning; and cleaning (subtraction of dominant modes). Also an independent function to find and remove "trends", and return them like they were an set of modes with couplings to the signal. * tod_ops: basic docs for filters and pca. * docs: mock 'yaml', 'spt3g', 'pyfftw' if not found. * docs: mock 'toml', 'sqlite3', 'so3g', 'toast', 'pixell' if not found. * docs: rebrand SOTODLib -> sotodlib; set some autodoc options * tod_ops: gap-filling (#81) Introduces simple polynomial (linear, really) gap filling, as well as "model-based" gap-filling that can use the results of a PCA (or common mode). The Extract and ExtractMatrix classes (the later of which is not quite compatible with storage in AxisManager, but could be moved in that direction) help with storing sparse subsets of a TOD signal efficiently. * core.metadata: Clean up ObsDb code; document it. This includes a minimal API for creating a usable ObsDb. Common database i/o functions created in core/metadata/common.py; other Dbs will need to start using those. * core.metadata: add "tags" functionality to ObsDb; unittest for ObsDb Other minor cleanup as well. * context: remove use of contextlib.nullcontext, since that needs >=3.7 * context: modernize ManifestDb and ObsFileDb with common to_file, from_file * context: raise informative errors when failing to broadcast/match metadata * tests/test_metadata.py: tests for the metadata resolution system The test is especially needed now, since this system works but I want to refactor the HDF5 stuff and still have it work the same... * metadata: ManifestDb raises informative error when too many matches * metadata: ResultSet.asarray has hdf_compat option And that option causes strings to be coded as byte strings in the output array, so the result can be written as an HDF5 dataset. * io.metadata: ResultSetHdfLoader and write_dataset PerDetectorHdf5 is now deprecated, but left in place as a shell (probably not for long). The ResultSetHdfLoader is tuned for the context/metadata system, with a single API (the one hacked in through PerDetectorHdf5.loader_class). For writing datasets, a function makes more sense than having a write_dataset method, assuming we don't want to put HDF5 i/o right in the ResultSet class. Also added a test for the array string conversion function. * context/metadata: remove PerDetectorHdf5; ResultSetHdf is now default This still requires a Context user to import sotodlib.io.metadata before trying metdata loads. * context: remove abandoned code in ManifestDb * context: the Context can include a list of "imports" This is useful to cause registration of data and metadata loaders for Context to use. * WIP: updates to context documentation. * Avoid slow slice operation when merging axis managers * Simplifiy and slightly generalize simplify_slice. It can now simplify along each dimension independently. * Avoid confusing situations where attempted updates to members of the axismanager instead end up creating an entry in the axismanager object itself, that masks the actually managed one * Fixed oversimplification of slice in simplify_slice. * context: get_obs has a "logic_only" arg And it causes the function to return the load specification (including what dets) without loading any data or metadata. * context/metadata: better handling of obs_colon_tags Raises errors if parent field not found, duplicated. * Docstrings for Superloader; pass in CelestialSightline to get_footprint * coords.get_supergeom: get a supermap to hold submaps * obsfiledb: fix how prefix is set when loading from context * tod_ops: iir_filter function for readout filter conv/deconv This sort of IIR filter can express Smurf (and MCE) readout filters. * metadata: improve docstrings; create ABC for Metadata Loaders. * docs: more context/metadata. * context: tweaks to metadata loader - When loading ManifestDb, map sqlite files as readonly without making a copy. This is the fastest. - colon_obs_tags: prepend keys with 'dets:' before passing to loader. - SuperLoader caches ManifestDb connections. This is probably of modest value for .sqlite on disk, but it helps a lot for .gz->memory databases. * Add support for multiple common modes and randomized coupling strengths. Fix a recent bug preventing simulation of common modes. (#88) * When running setup.py test, propagate the exit status back to the shell. * Context: fix a test that broke due to SuperLoader db caching * AxisManager: fix in_place=False bug, add a bunch of documentation. * HWP demodulation (#87) * Initial implememtation * Add support for caching demodulated 2f component * to dump demodulated data Co-authored-by: Satoru Takakura <[email protected]> * Implement h_n map operator (#91) * initial implementation * use the correct angle for h_n; add missing files * Add cross-linking map support * Use stationary detector weights * h_n should be computed per detector * draft instrument model revisions * added new readout defaults * changed wafer/wafers fields to wafer_slot/wafer_slots * readout naming updates * update detector band naming convention * change detector naming scheme to detectors.w##_p###_bnd_A/B * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change wafer naming to w##, card slot to card_slot##, crate slot crate_slot## * change tubes to tube_slots, add fields for tube_name and receiver_name (default blank) * Add field for platform_name * Update toast_so_example.py * updated lat tube names to cryo convention * SAT3 to SAT4 * update test naming conventions * Rotate the LAT boresight according to observing elevation * Move all toast-related source into a toast subpackage. * Fix imports * Fix default value * Reduce size of simulated hardware test. * Support recentering coordinates on given object or location * Remove import added by mistake * Support cuts in coords/pmat.py. Small rewrite in get_footprint for clarity. Handling cuts in the pointing matrix this way isn't enough to suppor them in the maximum-likelihood framework, but it makes the rest of the implementation a bit cleaner. However, right now enabling pmat cuts leads to some unexpected small changes in the result, which should be investigated before they are used properly.\n\nThe get_footprint change was made while I was debugging something else. What was there before was fine too, but I left these changes in because I think it makes it a bit easier to see the logic. While list comprehensions are very explicit, their verbosity can make it hard to see the forest for all the trees. * Hopefully clearer derivation of wcs in get_footprint * Fixed argument order bugs in get_gap_fill * coords: improve compatibility check in get_supergeom; add tests Our tests initially included some supergeom arguments that should have caused failure, but didn't... so now the supergeom compatibilty check is a bit more complete than the pixell one. Here we do not assume that all cylindrical proj must have crval[1]==0. * Add pixell to install_deps so test_coords can run * Move sim_sso * upgrade to new directory structure * complete sso sim integration * Add tool for measuring wafer and optics tube offsets * Remove stale folder Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Andrea Zonca <[email protected]> Co-authored-by: Theodore Kisner <[email protected]> Co-authored-by: Katie Harrington <[email protected]> Co-authored-by: Yilun Guan <[email protected]> Co-authored-by: Sigurd Kirkevold Næss <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Satoru Takakura <[email protected]> Co-authored-by: smsimon <[email protected]> * Add missing metadata required by atmospheric sims. * Swap Az/El and return the radius of the FOV * Rotate the LAT focalplane based on elevation and encode the corotator angle in the so3g files * write the corotator angle into so3g files * keep the cache object intact * Now receiver compensates for elevation rotation by default * Fix default corotator angle * Load the corotator angle when available * TOAST should also load the corotator angle Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Andrea Zonca <[email protected]> Co-authored-by: Theodore Kisner <[email protected]> Co-authored-by: Katie Harrington <[email protected]> Co-authored-by: Yilun Guan <[email protected]> Co-authored-by: Sigurd Kirkevold Næss <[email protected]> Co-authored-by: Matthew Hasselfield <[email protected]> Co-authored-by: Satoru Takakura <[email protected]> Co-authored-by: smsimon <[email protected]>

tskisner added the enhancement New feature or request label Apr 9, 2020

tskisner assigned tskisner and mhasself Apr 9, 2020

tskisner mentioned this issue Apr 16, 2020

Move existing source files to start reorganization #56

Closed

tskisner closed this as completed Feb 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revisit subpackage organization #53

Revisit subpackage organization #53

tskisner commented Apr 9, 2020

kmharrington commented Apr 9, 2020

tskisner commented Apr 9, 2020

tskisner commented Apr 16, 2020

tskisner commented Apr 16, 2020

tskisner commented Oct 28, 2020

kmharrington commented Oct 29, 2020

tskisner commented Oct 29, 2020

mhasself commented Oct 29, 2020

tskisner commented Oct 29, 2020

tskisner commented Feb 3, 2021

Revisit subpackage organization #53

Revisit subpackage organization #53

Comments

tskisner commented Apr 9, 2020

kmharrington commented Apr 9, 2020

tskisner commented Apr 9, 2020

tskisner commented Apr 16, 2020

tskisner commented Apr 16, 2020

tskisner commented Oct 28, 2020

kmharrington commented Oct 29, 2020

tskisner commented Oct 29, 2020

mhasself commented Oct 29, 2020

tskisner commented Oct 29, 2020

tskisner commented Feb 3, 2021