Merge branch 'master' into catsAndCrowdedFields

.github/pull_request_template.md

@@ -0,0 +1,18 @@
+<!-- If this PR closes a JIRA ticket, make sure the title starts with the JIRA issue number,
+for example HLA-1234: <Fix a bug> -->
+Resolves [HLA-nnnn](https://jira.stsci.edu/browse/HLA-nnnn)
+
+<!-- If this PR closes a GitHub issue, reference it here by its number -->
+Closes #
+
+<!-- describe the changes comprising this PR here -->
+This PR addresses ...
+
+**Checklist for maintainers**
+- [ ] added entry in `CHANGELOG.rst` within the relevant release section
+- [ ] updated or added relevant tests
+- [ ] updated relevant documentation
+- [ ] added relevant milestone
+- [ ] added relevant label(s)
+- [ ] ran regression tests, post a link to the Jenkins job below.
+      [How to run regression tests on a PR](https://github.com/spacetelescope/jwst/wiki/Running-Regression-Tests-Against-PR-Branches)

.github/workflows/ci.yml

@@ -55,8 +55,6 @@ jobs:
       cache-path: ${{ needs.crds.outputs.path }}
       cache-key: crds-${{ needs.crds.outputs.context }}
       envs: |
-        - linux: test-xdist
-          python-version: 3.8
         - linux: test-xdist
           python-version: 3.9
         - linux: test-xdist

.github/workflows/ci_cron.yml

@@ -43,8 +43,6 @@ jobs:
       cache-path: ${{ needs.crds.outputs.path }}
       cache-key: crds-${{ needs.crds.outputs.context }}
       envs: |
-        - macos: test-xdist
-          python-version: 3.8
         - macos: test-xdist
           python-version: 3.9
         - macos: test-xdist

.readthedocs.yaml

@@ -16,9 +16,15 @@ formats:
   - htmlzip
   - pdf
 
-# Optionally set the version of Python and requirements required to build your docs
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "mambaforge-4.10"
+
+conda:
+  environment: doc/.rtd-environment.yml
+
 python:
-  version: 3.8
   install:
     - method: pip
       path: .

CHANGELOG.rst

@@ -18,22 +18,55 @@ number of the code change for that issue.  These PRs can be viewed at:
     https://github.com/spacetelescope/drizzlepac/pulls
 
 
-3.6.1 (unreleased)
-==================
+3.6.2rc0 (unreleased)
+=====================
 
-- Fixed an incompatibility in the ``minmed`` code for cosmic ray rejection
-  with the ``numpy`` version ``>=1.25``. [#1573]
+- Removed the version restriction on matplotlib. [#1649]
 
+- Forced a preferential order on the final selection of the WCS solution 
+  from the common pool of solutions among all input exposurea.  All input images 
+  need to have the same WCSNAME (same WCS solution) when performing pipeline 
+  alignment to avoid imprinting differences from one catalog to another on the 
+  final fit and destroying the relative alignment. [#1645, #1638]
 
-3.6.1rc0 (15-Jun-2023)
-======================
+- Redesigned the overall structure of the documentation, readthedocs, for the
+  package. [#1620]
+
+- Addressed a bug in the calculation of measurements for each detected source
+  in the filter catalogs. The detection catalog, based upon the "total" image,
+  is now used in the correct manner to define the source centroids and shape 
+  properties.  In addition, these properties are used to perform aperture 
+  photometry. [#1614]
+
+- Updated the HAP drizzle parameters for WFPC2. The primary change includes 
+  changing skymethod='localmin' from the prior 'match' which did not work well 
+  for the overlapping chips. [#1617]
+
+- Corrected reference catalog weights from being proportional to sigma to
+  the proper 1/sigma**2. [#1616]
+
+- Removed the use of the shadow mask as an initial step in addressing the WFPC2 
+  chip gaps [#1551]
+
+- Fixed a bug in processing of the ``group`` argument due to which the code
+  would crash when ``group`` would be an integer number or a list of numbers.
+  Also, added support for specifying extensions as tuples of
+  ``(extname, extver)``. [#1612]
+
+
+3.6.1 (15-Jun-2023)
+===================
+
+- Fixed an incompatiblity in the ``minmed`` code for cosmic ray rejection
+  with the ``numpy`` version ``>=1.25``. [#1573]
+
+- Fixed projection cell identification in overlapping regions. [#1572]
 
 - Force the version of matplotlib to be <= 3.6.3 as the newer versions of
-  the library cause problems with the calcloud preview generation.  This
-  is a temporary restriction.
+  the library cause problems with the calcloud preview generation. [#1571] 
 
 3.6.0 (12-Jun-2023)
-======================
+===================
 
 - Modified the pyproject.toml file to ensure the tweakwcs version is greater
   than 0.8.2 as the issue of taking a very long time to compute the bounding

JenkinsfileRT

@@ -9,49 +9,47 @@ jobconfig = new JobConfig()
 jobconfig.credentials = [
         ['drizzlepac_codecov', 'CODECOV_TOKEN']
     ]
-// jobconfig.post_test_summary = true
+jobconfig.post_test_summary = true
 
 // Configure artifactory ingest
 data_config = new DataConfig()
 data_config.server_id = 'bytesalad'
-data_config.root = 'tests_output'
+data_config.root = 'clone/tests_output'
 data_config.match_prefix = '(.*)_result' // .json is appended automatically
 
 bc1 = new BuildConfig()
 bc1.nodetype = 'linux'
 bc1.env_vars = ['TEST_BIGDATA=https://bytesalad.stsci.edu/artifactory']
 bc1.name = '3.9'
 bc1.conda_packages = ['python=3.9']
-bc1.build_cmds = ["pip install numpy astropy codecov pytest-cov ci-watson==0.5",
+bc1.build_cmds = ["pip install numpy astropy codecov pytest-cov ci-watson",
 		 "pip install --upgrade -e '.[test]'",
                  "pip freeze"]
-bc1.test_cmds = ["pytest --cov=./ --basetemp=tests_output --bigdata",
+bc1.test_cmds = ["pytest --cov=./ --basetemp=tests_output --junitxml=results.xml --bigdata",
                 "codecov"]
 bc1.test_configs = [data_config]
-
+bc1.failedFailureThresh = 0
 
 bc2 = utils.copy(bc1)
 bc2.name = '3.9-dev'
 bc2.build_cmds[1] = "pip install -r requirements-dev.txt --upgrade -e '.[test]'"
 
 bc3 = new BuildConfig()
+bc3.runtime.add('CFLAGS=-std=gnu99')
 bc3.nodetype = 'linux'
 bc3.env_vars = ['TEST_BIGDATA=https://bytesalad.stsci.edu/artifactory']
-bc3.name = '3.10-dev'
+bc3.name = '3.10'
 bc3.conda_packages = ['python=3.10']
-bc3.build_cmds = ["pip install numpy astropy codecov pytest-cov ci-watson==0.5",
-		 "pip install -r requirements-dev.txt --upgrade -e '.[test]'",
+bc3.build_cmds = ["pip install numpy astropy ci-watson",
+		 "pip install --upgrade -e '.[test]'",
                  "pip freeze"]
-bc3.test_cmds = ["pytest --cov=./ --basetemp=tests_output --bigdata",
-                "codecov"]
 bc3.test_configs = [data_config]
 
 bc4 = utils.copy(bc3)
-bc4.name = '3.11-dev'
+bc4.name = '3.11'
 bc4.conda_packages = ['python=3.11']
 
-
 // Iterate over configurations that define the (distributed) build matrix.
 // Spawn a host (or workdir) for each combination and run in parallel.
 // Also apply the job configuration defined in `jobconfig` above.
-utils.run([bc1, bc3, bc4, jobconfig])
+utils.run([bc1, bc2, bc3, bc4, jobconfig])

doc/.rtd-environment.yml

@@ -1,4 +1,4 @@
-name: drizzlepac
+name: rtd311
 channels:
     - defaults
 dependencies:
@@ -12,7 +12,7 @@ dependencies:
     - pip:
       - -e ..
       - sphinx
-      - sphinx_rtd_theme
+      - "sphinx_rtd_theme>1.2.0"
       - sphinx_automodapi
       - matplotlib
       - "spherical_geometry>=1.2.22"

doc/source/CHANGELOG.rst

@@ -1,2 +1,4 @@
+Release Notes
+-------------
 
 .. include:: ../../CHANGELOG.rst

doc/source/DEPENDENCIES.rst

@@ -0,0 +1,5 @@
+Requirements
+-------------
+
+.. include:: ../../requirements-dev.txt
+    :literal:

doc/source/LICENSE.rst

@@ -1,5 +1,6 @@
-*******
-LICENSE
-*******
+License
+-------
+.. LICENSE:
 
 .. include:: ../../LICENSE.txt
+   :literal:

doc/source/conf.py

@@ -46,7 +46,7 @@ def setup(app):
 # sys.path.insert(0, os.path.abspath('exts/'))
 
 # If your documentation needs a minimal Sphinx version, state it here.
-needs_sphinx = '1.3'
+# needs_sphinx = '1.3'
 
 on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
 
@@ -132,7 +132,7 @@ def check_sphinx_version(expected_version):
 
 # General information about the project.
 project = u'DrizzlePac'
-copyright = u'2021, Warren Hack, Nadia Dencheva, Chris Sontag, Megan Sosey, Michael Droettboom, Mihai Cara, Michael Dulude, Michele De La Pena'
+copyright = u'2021, Warren Hack, Nadia Dencheva, Chris Sontag, Megan Sosey, Michael Droettboom, Mihai Cara, Michael Dulude, Michele De La Pena, Steve Goldman'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

doc/source/explanation.rst

@@ -0,0 +1,57 @@
+Explanation
+===========
+.. _explanation:
+
+Interactive Image Alignment
+---------------------------
+.. _tweakreg-tweakback:
+
+.. toctree:: 
+    :maxdepth: 1
+
+    explanation/image_registration
+
+Pipeline processing
+-------------------
+.. _pipeline-processing:
+
+.. toctree:: 
+    :maxdepth: 1
+
+    reference/astrodrizzle
+
+Hubble Archival Products
+------------------------
+.. _hubble-archival-products:
+
+.. toctree:: 
+    :maxdepth: 1
+
+    explanation/hap_introduction
+    explanation/astrometry
+    Improvements to HST astrometry<https://outerspace.stsci.edu/pages/viewpage.action?spaceKey=HAdP&title=Improvements+in+HST+Astrometry>
+    reference/catalogs
+
+SVM
+---
+
+.. _SVM:
+
+.. toctree:: 
+    :maxdepth: 1
+
+    explanation/singlevisit
+    explanation/catalog_generation
+
+MVM
+---
+
+.. _MVM:
+
+.. toctree:: 
+    :maxdepth: 1
+
+    explanation/mvmutilities
+    explanation/multivisit
+    reference/runmultihap
+    explanation/multivisit_products

doc/source/astrometry.rst → doc/source/explanation/astrometry.rst

@@ -1,8 +1,8 @@
 .. _astrometry:
 
-===================================================
-Headerlets and You: Astrometry in Drizzle Products
-===================================================
+=========================
+Astrometry and headerlets
+=========================
 
 The astrometry for any given observation relies upon accurate pointing information from the telescope.   However, HST has evolved over time since it was put into orbit, with the focus changing over time as the telescope desorbs.  The instruments have also changed positions over time relative to the FGS guides, and the coordinates for the guide stars were orginally determined using ground-based information.  All this has limited the calculation of the pointing of any given observation on the sky (absolute astrometry) to no better than 1-2 arc-seconds.
 
@@ -143,8 +143,8 @@ The terms are defined as:
 
   * **`FIT`**
 
-    - This term refers to the fact that sources from the image were identified, cross-matched and fit to sources from an astrometric catalog to create an *a posteriori* WCS solution.  
-        
+    - This term refers to the fact that sources from the image were identified, cross-matched and fit to sources from an astrometric catalog to create an *a posteriori* WCS solution.
+
   * **`<REL|IMG|EVM|SVM>`**
 
     - ``REL`` : This term denotes the fact that all images were aligned relative (REL) to each other and then aligned to an astrometric catalog.  This attempts to maintain the original relative alignment between the images in a given visit.
@@ -175,33 +175,33 @@ These separate terms provide as succinct a description of the solution determine
 Pipeline Processing
 -------------------
 All HST observations get processed in an automated environment using standard
-parameters for the calibration code, including the alignment and combination of 
+parameters for the calibration code, including the alignment and combination of
 individual exposures into undistorted products.  The standard pipeline processing
-to create the undistorted drizzled images (drc.fits or drz.fits) gets performed 
-using the 'runastrodriz' task in this package.  This same processing can be 
+to create the undistorted drizzled images (drc.fits or drz.fits) gets performed
+using the 'runastrodriz' task in this package.  This same processing can be
 run at any time using:
 
 .. code-block:: bash
 
     runastrodriz j8cw03010_asn.fits
-    
+
     runastrodriz j8cw03f6q_raw.fits
-    
+
 The files which need to be present are:
 
     * RAW files (\*raw.fits)
     * FLT files (\*flt.fits)
     * FLC files (\*flc.fits, if any were created by the pipeline)
     * ASN file  (\*asn.fits, if applicable)
-    
-This processing includes a lot of logic intended to not only apply pre-defined (apriori) 
-WCS solutions, but also to try and determine a new aposteriori solution then 
+
+This processing includes a lot of logic intended to not only apply pre-defined (apriori)
+WCS solutions, but also to try and determine a new aposteriori solution then
 verify which solution (default pipeline, apriori or aposteriori) actually provides
-the WCS which comes closest to the GAIA astrometric frame.  
-The :ref:`runastrodriz-description` of the runastrodriz task provides 
+the WCS which comes closest to the GAIA astrometric frame.
+The :ref:`runastrodriz-description` of the runastrodriz task provides
 the full discussion of the logic used to define the
 defined 'active' WCS that gets used to create the products which get archived.
- 
+
 
 Choosing a WCS
 ---------------