Merge branch 'develop' into cf_naming

doc/cookbook.rst

@@ -13,6 +13,10 @@ should adhere to the following approach.
 
 Note that:
 
+  - Essentially all PypeIt reduction steps are done via executing command-line
+    scripts using a terminal window.  We provide specific commands below and
+    throughout our documentation.
+
   - This cookbook provides minimal detail, but serves as a basic introduction to
     the primary steps used to reduce your data with PypeIt.  We guide you to
     other parts of our documentation that explain specific functionality in more
@@ -27,8 +31,9 @@ Note that:
   - Specific advice on :doc:`spectrographs/spectrographs` is provided in their own doc page
     (although not every supported spectrograph has stand-alone documentation).
 
-  - Invariably something will be out of date.  When you see an egregious
-    example, please holler on GitHub or Slack.
+  - Invariably something will be out of date in our doc pages.  When you see an
+    egregious example, please holler on `GitHub
+    <https://github.com/pypeit/PypeIt>`__ or `Slack <pypeit-users.slack.com>`__.
 
 Finally, note that before you keep going, you should have already done the following:
 

doc/include/shane_kast_blue_A.pypeit.rst

@@ -33,7 +33,6 @@
     b11.fits.gz | pixelflat,illumflat,trace |            144.955 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 |  57162.07897476852 |            1.0 |    15.0 |      d55 |     0
     b12.fits.gz | pixelflat,illumflat,trace |  145.0908333333333 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 | 57162.079351388886 |            1.0 |    15.0 |      d55 |     0
     b13.fits.gz | pixelflat,illumflat,trace | 145.22791666666666 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 | 57162.079728240744 |            1.0 |    15.0 |      d55 |     0
-     b2.fits.gz | pixelflat,illumflat,trace | 143.36208333333335 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 |  57162.07473645834 |            1.0 |    30.0 |      d55 |     0
      b3.fits.gz | pixelflat,illumflat,trace | 143.86791666666667 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 |  57162.07596400463 |            1.0 |    15.0 |      d55 |     0
      b4.fits.gz | pixelflat,illumflat,trace | 144.00458333333333 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 | 57162.076341782406 |            1.0 |    15.0 |      d55 |     0
      b5.fits.gz | pixelflat,illumflat,trace | 144.14041666666665 |  37.43222222222222 |  Dome Flat | 600/4310 | 2.0 arcsec |     1,1 |  57162.07671956019 |            1.0 |    15.0 |      d55 |     0

doc/tutorials/kast_howto.rst

@@ -11,7 +11,7 @@ Overview
 ========
 
 This doc goes through a full run of PypeIt on one of the Shane Kast *blue*
-datasets in the `PypeIt Development Suite`_.
+datasets in the :ref:`dev-suite`.
 
 Setup
 =====
@@ -26,11 +26,11 @@ Place all of the files in a single folder. Mine is named
 .. code-block:: bash
 
     $ ls
-    b10.fits.gz  b15.fits.gz  b1.fits.gz   b24.fits.gz  b4.fits.gz  b9.fits.gz
-    b11.fits.gz  b16.fits.gz  b20.fits.gz  b27.fits.gz  b5.fits.gz
-    b12.fits.gz  b17.fits.gz  b21.fits.gz  b28.fits.gz  b6.fits.gz
-    b13.fits.gz  b18.fits.gz  b22.fits.gz  b2.fits.gz   b7.fits.gz
-    b14.fits.gz  b19.fits.gz  b23.fits.gz  b3.fits.gz   b8.fits.gz
+    b1.fits.gz   b14.fits.gz  b19.fits.gz  b24.fits.gz  b5.fits.gz
+    b10.fits.gz  b15.fits.gz  b20.fits.gz  b27.fits.gz  b6.fits.gz
+    b11.fits.gz  b16.fits.gz  b21.fits.gz  b28.fits.gz  b7.fits.gz
+    b12.fits.gz  b17.fits.gz  b22.fits.gz  b3.fits.gz   b8.fits.gz
+    b13.fits.gz  b18.fits.gz  b23.fits.gz  b4.fits.gz   b9.fits.gz
 
 Run ``pypeit_setup``
 --------------------
@@ -48,10 +48,12 @@ Here is my call for these data:
     cd folder_for_reducing   # this is usually *not* the raw data folder
     pypeit_setup -r ${RAW_PATH}/b -s shane_kast_blue -c A
 
-This creates a :doc:`../pypeit_file` in the folder named
-``shane_kast_blue_A`` beneath where the script was run.
-Note that ``$RAW_PATH`` should be the *full* path, i.e. including a /
-at the start.
+This creates a :doc:`../pypeit_file` in the folder named ``shane_kast_blue_A``
+beneath where the script was run.  Note that ``$RAW_PATH`` should be the *full*
+path (i.e., as given in the example above).  Note that I have selected a single
+configuration (using the ``-c A``) option.  There is only one instrument
+configuration for this dataset, meaning using ``--c A`` and ``-c all`` are
+equivalent; see :doc:`../setup`.
 
 The ``shane_kast_blue_A.pypeit`` files looks like this:
 
@@ -60,7 +62,15 @@ The ``shane_kast_blue_A.pypeit`` files looks like this:
 For some instruments (especially Kast), it is common for frametypes to be
 incorrectly assigned owing to limited or erroneous headers.  However, in this
 example, all of the frametypes were accurately assigned in the
-:doc:`../pypeit_file`, so there are no edits to be made.
+:doc:`../pypeit_file`.
+
+.. note::
+
+    This is the rare case when the observation of a standard star is correctly
+    typed.  Generally, it will be difficult for the automatic frame-typing code
+    to distinguish standard-star observations from science targets, meaning that
+    you'll need to edit the pypeit file directly to designate standard-star
+    observations as such.
 
 Main Run
 ========
@@ -71,10 +81,13 @@ simply:
 .. code-block:: bash
 
     cd shane_kast_blue_A
-    run_pypeit shane_kast_blue_A.pypeit -o
+    run_pypeit shane_kast_blue_A.pypeit
 
-The ``-o`` indicates that any existing output files should be overwritten.  As
-there are none, it is superfluous but we recommend (almost) always using it.
+If you find you need to re-run the code, you can use the ``-o`` option to ensure
+the code overwrites any existing output files (excluding processed calibration
+frames).  If you find you need to re-build the calibrations, it's best to remove
+the relevant (or all) files from the ``Calibrations/`` directory **instead** of
+using the ``-m`` option.
 
 The :doc:`../running` doc describes the process in some
 more detail.
@@ -100,12 +113,13 @@ with `ginga`_:
 
 .. code-block:: bash
 
-    ginga Calibrations/Bias_A_1_01.fits
+    ginga Calibrations/Bias_A_0_DET01.fits
 
 As typical of most bias images, it is featureless
 (effectively noise from the readout).
 
-.. image:: ../figures/kastb_bias_image.png
+.. figure:: ../figures/kastb_bias_image.png
+   :width: 40%
 
 See :doc:`../calibrations/bias` for further details.
 
@@ -117,12 +131,13 @@ with `ginga`_:
 
 .. code-block:: bash
 
-    ginga Calibrations/Arc_A_1_01.fits
+    ginga Calibrations/Arc_A_0_DET01.fits
 
 As typical of most arc images, one sees a series
 of arc lines, here oriented horizontally (as always in PypeIt).
 
-.. image:: ../figures/kastb_arc_image.png
+.. figure:: ../figures/kastb_arc_image.png
+   :width: 30%
 
 See :doc:`../calibrations/arc` for further details.
 
@@ -140,9 +155,10 @@ the :ref:`pypeit_chk_edges` script, with this explicit call:
 
 .. code-block:: bash
 
-    pypeit_chk_edges Calibrations/Edges_A_1_01.fits.gz
+    pypeit_chk_edges Calibrations/Edges_A_0_DET01.fits.gz
 
-.. image:: ../figures/kastb_edges_image.png
+.. figure:: ../figures/kastb_edges_image.png
+   :width: 40%
 
 The data is the combined flat images and the green/red
 lines indicate the left/right slit edges (green/magenta in more recent versions).  The S174 label
@@ -161,13 +177,15 @@ calibration.  These are PNGs in the ``QA/PNG/`` folder.
 ::
 
 Here is an example of the 1D fits, written to
-the ``QA/PNGs/Arc_1dfit_A_1_01_S0175.png`` file:
+the ``QA/PNGs/Arc_1dfit_A_0_DET01_S0175.png`` file:
 
-.. image:: ../figures/kastb_arc1d.png
+.. figure:: ../figures/kastb_arc1d.png
+   :width: 90%
 
 What you hope to see in this QA is:
 
- - On the left, many of the blue arc lines marked with green IDs
+ - On the left, many of the blue arc lines marked with *green* IDs
+ - That the green IDs span the full spectral range.
  - In the upper right, an RMS < 0.1 pixels
  - In the lower right, a random scatter about 0 residuals
 
@@ -177,16 +195,32 @@ See :doc:`../calibrations/wvcalib` for further details.
 ::
 
 There are several QA files written for the 2D fits.
-Here is ``QA/PNGs/Arc_tilts_2d_A_1_01_S0175.png``:
+Here is ``QA/PNGs/Arc_tilts_2d_A_0_DET01_S0175.png``:
 
-.. image:: ../figures/kastb_arc2d.png
+.. figure:: ../figures/kastb_arc2d.png
+   :width: 50%
 
 Each horizontal line of circles traces the arc line centroid as a function of
 spatial position along the slit length.  These data are used to fit the tilt in
 the spectral position.  "Good" measurements included in the parametric trace are
 shown as black points; rejected points are shown in red.  Provided most were not
 rejected, the fit should be good.  An RMS<0.1 is also desired.
 
+We also provide a script so that the arcline traces can be assessed against the
+image using `ginga`_, similar to checking the slit edge tracing.
+
+.. code-block:: bash
+
+    pypeit_chk_tilts Calibrations/Tilts_A_0_DET01.fits.gz
+
+.. figure:: ../figures/kastb_ginga_tilts.png
+   :width: 40%
+
+   Main `ginga`_ window produced by ``pypeit_chk_tilts``.  The arc image is
+   shown in gray scale, the slit edges are shown in green/magenta, masked pixels
+   are highlighted in red, good centroids are shown in blue, and centroids
+   rejected during the fit are shown in yellow.
+
 See :doc:`../calibrations/wvcalib` for further details.
 
 Flatfield
@@ -201,25 +235,29 @@ window (``pixflat_norm``) after using
 
 .. code-block:: bash
 
-    pypeit_chk_flats Calibrations/Flat_A_1_01.fits
+    pypeit_chk_flats Calibrations/Flat_A_0_DET01.fits
 
-.. image:: ../figures/kastb_flat.png
+.. figure:: ../figures/kastb_flat.png
+   :width: 40%
 
 One notes the pixel-to-pixel variations;  these are
 at the percent level.
 The slit edges defined by the code
 are also plotted (green/red lines; green/magenta in more recent versions).
 The region of the detector beyond these images
-has been set to unit value.
+has been set to unity.
 
 See :doc:`../calibrations/flat` for further details.
 
 Spectra
 -------
 
-Eventually (be patient), the code will start
-generating 2D and 1D spectra outputs.  One per standard
-and science frame, located in the ``Science/`` folder.
+Eventually (be patient), the code will start generating 2D and 1D spectra
+outputs.  One per standard and science frame, located in the ``Science/``
+folder.
+
+For reference, full processing of this dataset on my circa 2020 Macbook
+Pro took a little more than 2 minutes.
 
 Spec2D
 ++++++
@@ -230,9 +268,10 @@ window (``sky_resid-det01``) after using
 
 .. code-block:: bash
 
-    pypeit_show_2dspec Science/spec2d_b27-J1217p3905_KASTb_2015may20T045733.560.fits
+    pypeit_show_2dspec Science/spec2d_b27-J1217p3905_KASTb_20150520T045733.560.fits
 
-.. image:: ../figures/kastb_spec2d.png
+.. figure:: ../figures/kastb_spec2d.png
+   :width: 40%
 
 The green/red lines are the slit edges (green/magenta in more recent versions).
 The brighter pixels down the center of the slit is the object.  The orange line
@@ -250,26 +289,46 @@ Here is a screen shot from the GUI showing the
 
 .. code-block:: bash
 
-    pypeit_show_1dspec Science/spec1d_b27-J1217p3905_KASTb_2015may20T045733.560.fits
+    pypeit_show_1dspec Science/spec1d_b27-J1217p3905_KASTb_20150520T045733.560.fits
 
-.. image:: ../figures/kastb_spec1d.png
+.. figure:: ../figures/kastb_spec1d.png
+   :width: 70%
 
-This uses the
-`XSpecGUI <https://linetools.readthedocs.io/en/latest/xspecgui.html>`_
-from the `linetools`_ package.
+This uses the `XSpecGUI
+<https://linetools.readthedocs.io/en/latest/xspecgui.html>`_ from the
+`linetools`_ package.  With your mouse hovering over the window, type ``?`` to
+open a webpage with the set of available commands used to interact with the
+plot.  The spectrum can also be ingested into a `specutils.Spectrum1D`_ object
+using our :ref:`spec1D-specutils`.
 
 See :doc:`../out_spec1D` for further details.
 
 Fluxing
 =======
 
+This dataset includes observations of a spectrophotometric standard, Feige 66,
+which is reduced alongside the science target observations.
+
 Now that we have a reduced standard star spectrum, we can
 use that to generate a sensitivity file.  Here is the
-call for this example, which I run in the ``Science/`` folder:
+call for this example:
 
 .. code-block:: bash
 
-    pypeit_sensfunc spec1d_b24-Feige66_KASTb_2015may20T041246.960.fits -o Kastb_feige66_sens.fits
+    pypeit_sensfunc Science/spec1d_b24-Feige66_KASTb_20150520T041246.960.fits -o Kastb_feige66_sens.fits
+
+This produces the sensitivity function (saved to ``Kastb_feige66_sens.fits``)
+and three QA (pdf) plots.  The main QA plot looks like this:
+
+.. figure:: ../figures/kastb_sens.png
+   :width: 60%
+
+   QA plot from the sensitivity calculation.  Black is the observed zeropoints,
+   red is the best-fit model, orange are points masked *before* fitting, blue
+   are points masked *during* fitting.
+
+The other two plots show the flux-calibrated standard-star spectrum against the
+archived spectrum and the full system (top of atmosphere) throughput.
 
 See :doc:`../fluxing` for further details.
 

doc/tutorials/tutorials.rst

@@ -5,14 +5,16 @@
 Tutorials
 =========
 
+If you've landed here without first reading through the :ref:`cookbook`, you're
+encouraged to start there and come back.
+
 If this is your **first time using PypeIt**, you're encouraged to read through
 the :doc:`Shane Kast<kast_howto>` tutorial as a general example of how to use
-PypeIt; see also the :ref:`cookbook`.  **You are also encouraged to pull example
-data from the DevSuite for your instrument when learning how to use the
-software**; see :ref:`dev-suite`.
+PypeIt.  **You are also encouraged to pull example data from the DevSuite for
+your instrument when learning how to use the software**; see :ref:`dev-suite`.
 
-For examples of reductions for different types of data (long-slit, echelle,
-etc), we recommend the following starting points:
+For examples of reductions for different types of data, we recommend the
+following starting points:
 
     - **Long-slit data**: :doc:`Shane Kast<kast_howto>`
 

pypeit/par/pypeitpar.py

@@ -3771,7 +3771,7 @@ def __init__(self, trace_npoly=None, snr_thresh=None, find_trim_edge=None,
 
         defaults['find_min_max'] = None
         dtypes['find_min_max'] = list
-        descr['find_min_max'] = 'It defines the minimum and maximum of your object in the spectral direction on the ' \
+        descr['find_min_max'] = 'It defines the minimum and maximum of your object in pixels in the spectral direction on the ' \
                                 'detector. It only used for object finding. This parameter is helpful if your object only ' \
                                 'has emission lines or at high redshift and the trace only shows in part of the detector.'
 

pypeit/scripts/view_fits.py

@@ -25,8 +25,7 @@ def get_parser(cls, width=None):
                             help='List the extensions only?')
         parser.add_argument('--proc', default=False, action='store_true',
                             help='Process the image (i.e. orient, overscan subtract, multiply by '
-                                 'gain) using pypeit.images.buildimage. Note det=mosaic will not '
-                                 'work with this option')
+                                 'gain) using pypeit.images.buildimage.')
         parser.add_argument('--bkg_file', type=str, default=None, help='FITS file to be subtracted from the image in file.'
                             '--proc must be set in order for this option to work.')
 
@@ -66,9 +65,6 @@ def main(args):
 
         if args.proc and args.exten is not None:
             msgs.error('You cannot specify --proc and --exten, since --exten shows the raw image')
-#        if args.proc and args.det == 'mosaic':
-#            msgs.error('You cannot specify --proc and --det mosaic, since --mosaic can only '
-#                       'display the raw image mosaic')
         if args.exten is not None and args.det == 'mosaic':
             msgs.error('You cannot specify --exten and --det mosaic, since --mosaic displays '
                        'multiple extensions by definition')
@@ -97,7 +93,6 @@ def main(args):
                 mosaic = len(_det) > 1
                 if not mosaic:
                     _det = _det[0]
-
             if args.proc:
                 # Use the biasframe processing parameters because processing
                 # these frames is independent of any other frames (ie., does not