Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Further data reading cleanups #700

Merged
merged 1 commit into from
Sep 10, 2024
Merged

Further data reading cleanups #700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 2 additions & 38 deletions docs/source/user_manual/masking.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,6 @@
Masking
=======

The KBMOD algorithm uses a data mask to represent invalid pixel values that should be ignored during the search. Masking is applied by:
The KBMOD algorithm uses a data mask to represent invalid pixel values that should be ignored during the search. Masking is applied by in the standardizers.

* applying the per-image mask
* applying a computed a global mask
* growing the computed mask footprint

As stated previously (see :ref:`Input Files`), KBMOD expects Vera C. Rubin Science Pipelines calexp-style FITS files. Therefore each science image has an associated mask. Per-image masks are applied by checking whether the associated mask image contains one or more of the specified flags, see :py:attr:`~kbmod.run_search.run_search.default_flag_keys` attribute of :py:class:`~kbmod.run_search.run_search`.
These values can be overwritten by providing the :py:attr:`~kbmod.run_search.run_search.flag_keys` attribute of :py:class:`~kbmod.run_search.run_search`. Futhermore, pixels with values that exceed a threshold value will be masked if a flux threshold is provided, see ``mask_threshold`` key in :ref:`Search Parameters`.

Global mask is computed for all images by counting how many individual masks have flagged a pixel for **any** reason in the list of allowed global masking flags, and masking it out if that count surpasses a given threshold. The number of times a pixel has to be flagged to be masked can be set by ``mask_num_images`` parameter (see :ref:`Search Parameters`) and the list of flags to use in global masking can be modified via the ``repeated_flag_keys`` parameter and is by default set to :py:attr:`~kbmod.run_search.run_search.default_repeated_flag_keys`.
Global masking is not applied when the list of allowed masking flags is empty.

After the per-image and global masks are applied to every image, KBMOD grows the mask to nearby pixels. The parameters ``mask_grow`` (see :ref:`Search Parameters`) determines the ammount of the growth.

The provided pixel bitmask uses the following mapping between flag and bitmask values, which corresponds to the Rubin Science Pipelines mask values:

================== =====
Key Value
================== =====
BAD 0
CLIPPED 9
CR 3
CROSSTALK 10
DETECTED 5
DETECTED_NEGATIVE 6
EDGE 4
INEXACT_PSF 11
INTRP 2
NOT_DEBLENDED 12
NO_DATA 8
REJECTED 13
SAT 1
SENSOR_EDGE 14
SUSPECT 7
UNMASKEDNAN 15
================== =====


To overwrite the default mapping set ``mask_bits_dict`` parameter, see :ref:`Search Parameters`.
TODO: Add more detail.
30 changes: 1 addition & 29 deletions docs/source/user_manual/search_params.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,21 +99,6 @@ This document serves to provide a quick overview of the existing parameters and
| ``lh_level`` | 10.0 | The minimum computed likelihood for an |
| | | object to be accepted. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_bits_dict`` | default_mask_bits_dict | A dictionary indicating which masked |
| | | values to consider invalid pixels. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_bit_vector`` | None | A bit vector to use for masking. If |
| | | provided, takes precedence over |
| | | ``flag_keys``. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_grow`` | 10 | Size, in pixels, the mask will be grown|
| | | by. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_num_images`` | 2 | Threshold for number of times a pixel |
| | | needs to be flagged in order to be |
| | | masked in global mask. |
| | | See :ref:`Masking` for more. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_threshold`` | None | The flux threshold over which a pixel |
| | | is automatically masked. ``None`` |
| | | means no flux-based masking. |
Expand All @@ -123,10 +108,6 @@ This document serves to provide a quick overview of the existing parameters and
| | | computed likelihood above this |
| | | threshold are rejected. |
+------------------------+-----------------------------+----------------------------------------+
| ``mjd_lims`` | None | Limits the search to images taken |
| | | within the given range (or ``None`` |
| | | for no filtering). |
+------------------------+-----------------------------+----------------------------------------+
| ``mom_lims`` | [35.5, 35.5, 2.0, 0.3, 0.3] | Thresholds for the moments of a |
| | | Gaussian fit to the flux, specified as |
| | | ``[xx, yy, xy, x, y]``. |
Expand All @@ -147,10 +128,6 @@ This document serves to provide a quick overview of the existing parameters and
| ``psf_val`` | 1.4 | The value for the standard deviation of|
| | | the point spread function (PSF). |
+------------------------+-----------------------------+----------------------------------------+
| ``psf_file`` | None | The path and filename of a separate |
| | | file containing the per-image PSFs. |
| | | See :ref:`PSF File` for more. |
+------------------------+-----------------------------+----------------------------------------+
| ``repeated_flag_keys`` | default_repeated_flag_keys | The flags used when creating the global|
| | | mask. See :ref:`Masking`. |
+------------------------+-----------------------------+----------------------------------------+
Expand Down Expand Up @@ -182,12 +159,7 @@ This document serves to provide a quick overview of the existing parameters and
| | | if: |
| | | * ``sum`` - (default) Per pixel sum |
| | | * ``median`` - A per pixel median |
| | | * ``mean`` - A per pixel mean |
+------------------------+-----------------------------+----------------------------------------+
| ``time_file`` | None | The path and filename of a separate |
| | | file containing the time when each |
| | | image was taken. See :ref:`Time File` |
| | | for more. |
| | | * ``mean`` - A per pixel mean |\
+------------------------+-----------------------------+----------------------------------------+
| ``track_filtered`` | False | A Boolean indicating whether to track |
| | | the filtered trajectories. Warning |
Expand Down
3 changes: 0 additions & 3 deletions src/kbmod/configuration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,21 +41,18 @@ def __init__(self):
"known_obj_jpl": False,
"lh_level": 10.0,
"max_lh": 1000.0,
"mjd_lims": None,
"mom_lims": [35.5, 35.5, 2.0, 0.3, 0.3],
"num_obs": 10,
"output_suffix": "search",
"peak_offset": [2.0, 2.0],
"psf_val": 1.4,
"psf_file": None,
"res_filepath": None,
"result_filename": None,
"results_per_pixel": 8,
"save_all_stamps": False,
"sigmaG_lims": [25, 75],
"stamp_radius": 10,
"stamp_type": "sum",
"time_file": None,
"track_filtered": False,
"v_arr": [92.0, 526.0, 256],
"x_pixel_bounds": None,
Expand Down
8 changes: 8 additions & 0 deletions src/kbmod/standardizers/fits_standardizers/kbmodv05.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,9 +155,17 @@ def translateHeader(self):
standardizedHeader = {}
obs_datetime = Time(self.primary["DATE-AVG"], format="isot")
standardizedHeader["mjd"] = obs_datetime.mjd
standardizedHeader["mjd_mid"] = obs_datetime.mjd

# these are all optional things
standardizedHeader["filter"] = self.primary["FILTER"]

# If no observatory information is given, default to the Deccam data
# (Cerro Tololo Inter-American Observatory).
standardizedHeader["obs_lon"] = self.primary.get("OBS-LONG", 70.81489)
standardizedHeader["obs_lat"] = self.primary.get("OBS-LAT", -30.16606)
standardizedHeader["obs_elev"] = self.primary.get("OBS-ELEV", 2215.0)

return standardizedHeader

def _standardizeMask(self):
Expand Down
3 changes: 0 additions & 3 deletions tests/test_regression_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,8 +258,6 @@ def perform_search(im_stack, res_filename, default_psf):
"im_filepath": "./",
"res_filepath": None,
"result_filename": res_filename,
"time_file": None,
"psf_file": None,
"psf_val": default_psf,
"output_suffix": "",
"v_arr": v_arr,
Expand All @@ -268,7 +266,6 @@ def perform_search(im_stack, res_filename, default_psf):
"num_obs": num_obs,
"do_mask": True,
"lh_level": 25.0,
"mjd_lims": [52130.0, 62130.0],
"sigmaG_lims": [25, 75],
"mom_lims": [37.5, 37.5, 1.5, 1.0, 1.0],
"peak_offset": [3.0, 3.0],
Expand Down
10 changes: 0 additions & 10 deletions tests/test_work_unit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,6 @@ def setUp(self):
self.config = SearchConfiguration()
self.config.set("im_filepath", "Here")
self.config.set("num_obs", self.num_images)
self.config.set("mask_bits_dict", {"A": 1, "B": 2})
self.config.set("repeated_flag_keys", None)

# Create a fake WCS
self.wcs = make_fake_wcs(200.6145, -7.7888, 500, 700, 0.00027)
Expand Down Expand Up @@ -275,8 +273,6 @@ def test_save_and_load_fits(self):
# Check that we read in the configuration values correctly.
self.assertEqual(work2.config["im_filepath"], "Here")
self.assertEqual(work2.config["num_obs"], self.num_images)
self.assertDictEqual(work2.config["mask_bits_dict"], {"A": 1, "B": 2})
self.assertIsNone(work2.config["repeated_flag_keys"])

# We throw an error if we try to overwrite a file with overwrite=False
self.assertRaises(FileExistsError, work.to_fits, file_path)
Expand Down Expand Up @@ -333,8 +329,6 @@ def test_save_and_load_fits_shard(self):
# Check that we read in the configuration values correctly.
self.assertEqual(work2.config["im_filepath"], "Here")
self.assertEqual(work2.config["num_obs"], self.num_images)
self.assertDictEqual(work2.config["mask_bits_dict"], {"A": 1, "B": 2})
self.assertIsNone(work2.config["repeated_flag_keys"])

# We throw an error if we try to overwrite a file with overwrite=False
self.assertRaises(FileExistsError, work.to_fits, file_path)
Expand Down Expand Up @@ -365,8 +359,6 @@ def test_save_and_load_fits_shard_lazy(self):
# Check that we read in the configuration values correctly.
self.assertEqual(work2.config["im_filepath"], "Here")
self.assertEqual(work2.config["num_obs"], self.num_images)
self.assertDictEqual(work2.config["mask_bits_dict"], {"A": 1, "B": 2})
self.assertIsNone(work2.config["repeated_flag_keys"])
self.assertEqual(work2.im_stack.img_count(), 0)

work2.load_images()
Expand Down Expand Up @@ -413,8 +405,6 @@ def test_to_from_yaml(self):
# Check that we read in the configuration values correctly.
self.assertEqual(work2.config["im_filepath"], "Here")
self.assertEqual(work2.config["num_obs"], self.num_images)
self.assertDictEqual(work2.config["mask_bits_dict"], {"A": 1, "B": 2})
self.assertIsNone(work2.config["repeated_flag_keys"])

def test_image_positions_to_original_icrs_invalid_format(self):
work = WorkUnit(
Expand Down
Loading