Merge pull request #1666 from pypeit/lris_wave

LRIS jamboree

CHANGES.rst

@@ -42,6 +42,16 @@
 - HIRES wavelength solution improvements galor
 - Added `redo_slits` option
 - Refactored ``load_line_lists()`` yet again!
+- Improvements for keck/LRIS
+    - Generated wavelength templates for all the LRIS grism & grating
+    - Added FeAr line list
+    - Improved calibration association and frame typing
+    - Improved and added documentation
+    - Changes to ``metadata.py`` including commenting out, in the pypeit file,
+      files that have frametype None (this prevent ``run_pypeit`` to crash)
+    - Added a function ``check_spectrograph()`` (currently only defined for LRIS),
+      that checks (during ``pypeit_setup``) if the selected spectrograph is the
+      corrected one for the data used. 
 
 
 1.13.0 (2 June 2023)

doc/calibrations/slit_tracing.rst

@@ -19,12 +19,40 @@ task owing to the wide variety in:
 Developing a single algorithm to handle all of these edge cases (pun
 intended) is challenging if not impossible. Therefore, there are a number of
 user-input parameters that one may need to consider when running PypeIt (see
-below).
+below :ref:`slit_tracing_issues` and :ref:`slit_tracing_customizing`).
 
 Underlying the effort is the :class:`~pypeit.edgetrace.EdgeTraceSet` class; see
 :func:`~pypeit.edgetrace.EdgeTraceSet.auto_trace` for a description of the
 algorithm.
 
+Slit-mask design matching
+-------------------------
+
+PypeIt can incorporate information about the slit-mask design into the
+slit-tracing process. This is primarily performed by
+:func:`pypeit.edgetrace.EdgeTraceSet.maskdesign_matching`, which matches
+the slit edges traced by PypeIt to the slit edges predicted
+using the slit-mask design information stored in the observations.
+Moreover, :func:`~pypeit.edgetrace.EdgeTraceSet.maskdesign_matching`
+uses the predicted slit edges positions to add slit traces that have
+not been detected in the image.
+
+
+This functionality at the moment is implemented only for these
+:ref:`slitmask_info_instruments` and is switched on by setting
+``use_maskdesign`` flag in :ref:`edgetracepar` to True. Other parameters
+may need to be adjusted as well, depending on the instrument (see
+:ref:`slitmask_ids_report` and the relevant instrument documentation pages).
+
+.. _slitmask_info_instruments:
+
+Slit-mask design Spectrographs
+++++++++++++++++++++++++++++++
+- :doc:`../spectrographs/deimos`
+- :doc:`../spectrographs/mosfire`
+- :doc:`../spectrographs/lris` (limited)
+- :doc:`../spectrographs/gemini_gmos` (limited)
+
 Viewing
 =======
 
@@ -61,6 +89,7 @@ additional output that can be used to diagnose the parameterized fits to the
 edge traces and the PCA decomposition.  Fair warning that, for images with many
 slits, these plots can be laborious to wade through...
 
+.. _slit_tracing_issues:
 Known Slit Tracing Issues
 =========================
 
@@ -189,6 +218,7 @@ For example:
 This will remove any slit on detector 2 that contains ``x_spat=2121``
 at ``y_spec=2000`` and similarly for the slit on ``det=3``.
 
+.. _slit_tracing_customizing:
 Slit Tracing Customizing
 ========================
 

doc/calibrations/slits.rst

@@ -45,10 +45,10 @@ This will show a table that looks like this:
         322     699091   248.21048545837402 .. 278.352352142334   345.8145122528076 .. 376.4430561065674   251.7797389884793 .. 281.9216056724392   345.8145122528076 .. 376.4430561065674   297.0124988555908 .. 327.3977041244507         0     0    -inf     inf
         457     699080  350.76593017578125 .. 379.9788398742676   514.4993572235107 .. 544.0507583618164  355.0206685112119 .. 384.23357820969824    513.3126446738094 .. 542.864045812115     432.632643699646 .. 462.014799118042         0     0    -inf     inf
 
-In addition, if reducing :doc:`../spectrographs/deimos` or
-:doc:`../spectrographs/mosfire` data and slit-mask design matching is performed
-(see :ref:`deimos-mask-matching` for DEIMOS and :ref:`mosfire-edge-tracing` for
-MOSFIRE), a second `astropy.io.fits.BinTableHDU`_ is written to disk.
+In addition, if reducing data from these :ref:`slitmask_info_instruments`
+and slit-mask design matching is performed (see e.g., :ref:`deimos-mask-matching`
+for DEIMOS and :ref:`mosfire-edge-tracing` for MOSFIRE), a second
+`astropy.io.fits.BinTableHDU`_ is written to disk.
 
 .. code-block:: console
 

doc/calibrations/wave_calib.rst

@@ -81,8 +81,9 @@ HgI     2900-12000  28 February 2022
 KrI     4000-10000  3 May 2018
 NeI     5000-12000  3 May 2018
 XeI     4000-12000  3 May 2018
-ZnI     3000-5000   21 December 2016
+ZnI     3000-5000   6 Sep 2023
 ThAr    3000-11000  9 January 2018
+FeAr    3000-9000   6 Sep 2023
 ======  ==========  ================
 
 In the case of the ThAr list, all of the lines are taken from the NIST database,

doc/dev/add_missing_obj.rst

@@ -15,6 +15,7 @@ Version History
 1.1         Debora Pelliccia    28 Jul 2021  1.4.3dev
 1.3         Debora Pelliccia    21 Oct 2021  1.6.1dev
 1.4         J. Xavier Prochaska 25 Jan 2022  1.7.1dev
+1.5         Debora Pelliccia     6 Sep 2023  1.13.1dev
 =========   =================== =========== ===========
 
 ----
@@ -23,7 +24,7 @@ Basics
 ------
 
 The procedure to determine the location on the slit of undetected objects
-using the slitmask design information is currently available for Keck/DEIMOS and Keck/MOSFIRE only
+using the slitmask design information is currently available for these :ref:`slitmask_info_instruments` only
 and it is performed right after the object finding (see :ref:`object_finding`)
 and the RA, Dec and object name assignment procedures (see :ref:`radec_object_report`) have been completed.
 

doc/dev/lrisconfig.rst

@@ -0,0 +1,135 @@
+.. include:: ../include/links.rst
+
+.. _lris_config_report:
+
+Automated sorting of LRIS frames by instrument configuration
+==============================================================
+
+Version History
+---------------
+
+
+=========   ================   =========== ===========
+*Version*   *Author*           *Date*      ``PypeIt``
+=========   ================   =========== ===========
+1.0         Debora Pelliccia    6 Sep 2023 1.13.1.dev
+=========   ================   =========== ===========
+
+----
+
+Basics
+------
+
+To prepare for the data reduction, PypeIt, first, automatically associates fits
+files to specific :ref:`frame_types` (see :ref:`lris_frames_report`) and, then,
+collects groups of frames in unique instrument configurations (see below). This is performed
+by the :ref:`pypeit_setup` script, which sorts the frames and writes a
+:ref:`pypeit_file` for each unique configuration. See :ref:`setup_doc`.
+
+
+LRIS configuration identification
+---------------------------------
+
+The LRIS instrument configurations are determined, in the same
+way for ``keck_lris_red``, ``keck_lris_red_orig``, ``keck_lris_red_mark4``,
+``keck_lris_blue``, and ``keck_lris_blue_orig`` (unless otherwise noted),
+by the function :func:`pypeit.metadata.PypeItMetaData.unique_configurations`,
+which finds unique combinations of the following keywords:
+
+===============     ====================================================================
+``fitstbl`` key     Header Key
+===============     ====================================================================
+``dispname``        ``GRANAME`` (LRIS RED) or ``GRISNAME`` (LRIS BLUE)
+``dichroic``        ``DICHNAME``
+``decker``          ``SLITNAME``
+``binning``         ``BINNING``
+``amp``             ``NUMAMPS`` (``TAPLINES`` for ``keck_lris_red_mark4``)
+``dispangle``       ``GRANGLE`` (only LRIS RED)
+``cenwave``         ``WAVELEN`` (only LRIS RED)
+===============     ====================================================================
+
+The unique configurations are determined by collating the relevant metadata from the headers
+of all frames found by a run of :ref:`pypeit_setup`, *except* those that are designated as
+bias frames. The reason is that bias frames can have header data (e.g., ``dispangle``)
+that do not match the instrument configuration that an observer intended for their use;
+e.g., the frames were taken before the instrument was fully configured for the night's
+observations. Therefore, PypeIt uses the ``dateobs``, ``binning``, ``amp`` keys to match
+the bias frames to the configurations with frames taken on the same date, with
+the same binning and on the same amplifier.
+
+After that, :func:`pypeit.metadata.PypeItMetaData.set_configurations` associates each frame
+to the relevant unique configuration ("setup"), by assigning a setup identifier
+(e.g., A,B,C,D...) to every frames for which the values of the above keywords match the
+values of the specific unique configuration.
+
+LRIS calibration groups
+-----------------------
+
+PypeIt uses the concept of a "calibration group" to define a complete set of
+calibration frames (e.g., arcs, flats) and the science frames to which these calibration
+frames should be applied.
+
+By default, :ref:`pypeit_setup` uses the setup identifier to assign frames to a single
+calibration group. Frames that are in the same calibration group will have the same PypeIt
+keyword ``calib``. No automated procedure exists to do anything except this.
+However, the user can edit the :ref:`pypeit_file` to, within a given configuration, assign
+specific calibration frames to specific science frames using the data in the ``calib`` column
+of the :ref:`data_block`.
+
+Testing
+-------
+
+Requirement PLL-17 states: "As a user, I expect the pipeline to automatically
+and correctly associate calibrations with science frames."
+
+PypeIt meets this requirement in the majority of use cases, as shown by the tests below.
+
+Note that the tests described in :ref:`lris_frames_report` are also relevant here
+since they show that PypeIt correctly identifies LRIS data and associates
+them with a single configuration, all written to a single pypeit file.
+
+To test that PypeIt can successfully identify multiple
+configurations among a set of files, we have added five tests
+``${PYPEIT_DEV}/unit_tests/test_setups.py``. They are:
+
+- ``test_setup_keck_lris_blue_multiconfig()``,
+- ``test_setup_keck_lris_blue_orig_multiconfig()``
+- ``test_setup_keck_lris_red_multiconfig()``
+- ``test_setup_keck_lris_red_orig_multiconfig()``
+- ``test_setup_keck_lris_red_mark4_multiconfig()``
+
+Here is an example of how to run the tests:
+
+.. code-block:: bash
+
+    cd ${PYPEIT_DEV}/unit_tests
+    pytest test_setup.py::test_setup_keck_lris_blue_multiconfig -W ignore
+
+The tests require that you have downloaded the PypeIt
+:ref:`dev-suite` and defined the ``PYPEIT_DEV`` environmental
+variable that points to the relevant directory.
+
+The algorithm for all these tests is the same and is as follows:
+
+    1. Collect the names of all files in selected LRIS directories
+       (separately for ``keck_lris_blue``, ``keck_lris_blue_orig``, ``keck_lris_red``,
+       ``keck_lris_red_orig``, ``keck_lris_red_mark4``).
+
+    2. Use :class:`~pypeit.pypeitsetup.PypeItSetup` to automatically
+       identify the configurations for these files.
+
+    3. Check that the code found two configurations and wrote the
+       pypeit files for each.
+
+    4. For each configuration:
+
+        a. Read the pypeit file
+
+        b. Check that the name for the setup is correct ('A' or 'B')
+
+        c. Check that the calibration group is the same for all frames ('0' or '1')
+
+
+Because these tests are now included in the PypeIt
+:ref:`unit-tests`, these configuration checks are performed by the
+developers for every new version of the code.

doc/dev/lrisframes.rst

@@ -0,0 +1,136 @@
+.. include:: ../include/links.rst
+
+.. _lris_frames_report:
+
+Automated typing of LRIS (BLUE and RED) frames
+==============================================
+
+Version History
+---------------
+
+
+=========   ================   =========== ===========
+*Version*   *Author*           *Date*      ``PypeIt``
+=========   ================   =========== ===========
+1.0         Debora Pelliccia    6 Sep 2023 1.13.1.dev
+=========   ================   =========== ===========
+
+----
+
+Basics
+------
+
+The general procedure used to assign frames a given type is described
+here: :ref:`frame_types`.
+
+LRIS frame typing
+-----------------
+
+The primary typing of LRIS frames is performed by
+:func:`pypeit.spectrographs.keck_lris.KeckLRISSpectrograph.check_frame_type`.
+This function checks the values of various header keywords against a
+set of criteria used to classify the frame type. The same criteria are used for
+``keck_lris_red``, ``keck_lris_red_orig``, ``keck_lris_red_mark4``, ``keck_lris_blue``,
+and ``keck_lris_blue_orig``, unless otherwise noted.
+The header cards required for the frame-typing and their associated keyword in the
+:class:`~pypeit.metadata.PypeItMetaData` object are:
+
+===============     ======================================================
+``fitstbl`` key     Header Key
+===============     ======================================================
+``exptime``         ``ELAPTIME`` (``TELAPSE`` for ``keck_lris_red_mark4``)
+``hatch``           ``TRAPDOOR``
+``lampstat01``      See below
+===============     ======================================================
+
+``lampstat01`` is defined using a combination of header keywords, which include
+``LAMPS``, ``MERCURY``, ``NEON``, ``ARGON``, ``CADMIUM``, ``ZINC``, ``HALOGEN``,
+``KRYPTON``, ``XENON``, ``FEARGON``, ``DEUTERI``, ``FLAMP1``, ``FLAMP2``, ``FLIMAGIN``,
+``FLSPECTR``. Since LRIS header keywords have changed over time, the exact combination
+of keywords used to define ``lampstat01`` varies depending on the available header keywords.
+
+The criteria used to select each frame type are as follows:
+
+=============   ============   ======================================   ===========================================
+Frame           ``hatch``      ``lampstat01``                           ``exptime``
+=============   ============   ======================================   ===========================================
+``science``     ``'open'``     ``'off'``                                ``>61s``
+``standard``    ``'open'``     ``'off'``                                ``>1s`` & ``<61s``
+``bias``        ``'closed'``   ``'off'``                                ``<0.001s``
+``pixelflat``   ``'closed'``   ``'Halogen' or '2H'``                    ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE)
+``pixelflat``   ``'open'``     ``'on'``                                 ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE)
+``trace``       ``'closed'``   ``'Halogen'`` or ``'2H'``                ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE)
+``trace``       ``'open'``     ``'on'``                                 ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE)
+``illumflat``   ``'closed'``   ``'Halogen'`` or ``'2H'``                ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE)
+``illumflat``   ``'open'``     ``'on'``                                 ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE)
+``arc``         ``'closed'``   ``!= 'Halogen', '2H', 'on', 'off'``      Not used
+``tilt``        ``'closed'``   ``!= 'Halogen', '2H', 'on', 'off'``      Not used
+=============   ============   ======================================   ===========================================
+
+Note that PypeIt employs commonly used value of ``exptime`` to distinguish frame type;
+however, if needed, the user can specify a different value by
+using the ``exprng`` parameter in the :ref:`pypeit_file`; see also :ref:`frame_types`.
+
+The criteria used to select ``arc`` and ``tilt`` frames are identical; the same is true for
+``pixelflat``, ``trace``, and ``illumflat`` frames. However, it's important to note that
+PypeIt is able to correctly assign the ``pixelflat``, ``trace``, and ``illumflat`` types
+to the internal and dome flat frames, but the twilight flats will generally have the
+``science`` or ``standard`` type. Therefore, the user should manually change their frame type
+in the :ref:`pypeit_file`.
+
+Finally, note that a LRIS frame is never given a ``pinhole`` or ``dark`` type.
+
+
+Testing
+-------
+
+Requirement PLL-16 states: "As a user, I expect the pipeline to automatically classify my data."
+
+``PypeIt`` meets this requirement as demonstrated by the tests at
+``${PYPEIT_DEV}/unit_tests/test_frametype.py``. There is one test
+per spectrograph:
+
+- ``test_lris_blue()``
+- ``test_lris_blue_orig()``
+- ``test_lris_red()``
+- ``test_lris_red_orig()``
+- ``test_lris_red_mark4()``
+
+Here is an example of how to run the tests:
+
+.. code-block:: bash
+
+    cd ${PYPEIT_DEV}/unit_tests
+    pytest test_frametype.py::test_lris_blue -W ignore
+
+The tests requires that you have downloaded the PypeIt
+:ref:`dev-suite` and defined the ``PYPEIT_DEV`` environmental
+variable that points to the relevant directory. The algorithm for
+all these tests is the same and is as follows:
+
+    1. Find the directories in the :ref:`dev-suite` with Keck
+       LRIS data (separately for ``keck_lris_blue``,
+       ``keck_lris_blue_orig``, ``keck_lris_red``,
+       ``keck_lris_red_orig``, ``keck_lris_red_mark4``).
+
+    2. For each directory (i.e., instrument setup):
+
+        a. Make sure there is a "by-hand" version of the pypeit file
+           for this setup where a human (one of the pypeit
+           developers) has ensured the frame types are correct.
+
+        b. Effectively run :ref:`pypeit_setup` on each of the
+           instrument setups to construct a new pypeit file with the
+           automatically generated frame types.
+
+        c. Read both the by-hand and automatically generated frame
+           types from these two pypeit files and check that they are
+           identical. This check is *only* performed for the
+           calibration frames, not any ``science`` or ``standard``
+           frames.
+
+Because this test is now included in the ``PypeIt``
+:ref:`unit-tests`, this frame-typing check is performed by the
+developers for every new version of the code.
+
+