Merge branch 'main' into mike/style-guide-retry

docs/source/contributing.rst

@@ -158,7 +158,10 @@ NOTE:  If you have added or modified an example script, see `Adding Usage Exampl
 Adding Usage Examples
 ---------------------
 
-New usage examples should adhere to the same general structure as the existing examples to ensure that the mechanism for automatically generating corresponding Jupyter notebooks functions correctly. In particular:
+New usage examples should adhere to the same general structure as the
+existing examples to ensure that the mechanism for automatically
+generating corresponding Jupyter notebooks functions correctly. In
+particular:
 
 1. The initial lines of the script should consist of a comment block, followed by a blank line, followed by a multiline string with an RST heading on the first line, e.g.
 
@@ -193,47 +196,37 @@ New usage examples should adhere to the same general structure as the existing e
 Adding new examples
 ^^^^^^^^^^^^^^^^^^^
 
-The following steps show how to add a new example, ``new_example.py``, to the packaged usage
-examples. We assume the SCICO repository has been cloned to ``scico/``.
+The following steps show how to add a new example, ``new_example.py``,
+to the packaged usage examples. We assume the SCICO repository has
+been cloned to ``scico/``.
 
-Note that the ``.py`` scripts are included in ``scico/examples/scripts``, while the compiled
-Jupyter Notebooks are located in the scico-data submodule, which is symlinked to ``scico/data``.
-When adding a new usage example, both the scico and scico-data repositories must be updated and
-kept in sync.
+Note that the ``.py`` scripts are included in
+``scico/examples/scripts``, while the compiled Jupyter Notebooks are
+located in the scico-data submodule, which is symlinked to
+``scico/data``.  When adding a new usage example, both the ``scico``
+and ``scico-data`` repositories must be updated and kept in sync.
 
 .. warning::
-   Ensure that all binary data (including raw data, images, ``.ipynb`` files) are added to scico-data, not the base ``scico`` repo.
-
+   Ensure that all binary data (including raw data, images, ``.ipynb`` files) are added to ``scico-data``, not the main ``scico`` repo.
 
 
 1. Add the ``new_example.py`` script to the ``scico/examples/scripts`` directory.
 
-2. Add the basename of the script (i.e., without the pathname or ``.py`` extension; in this case,
-   ``new_example``) to ``examples/notebooks/examples.rst``.
-
-3. Convert your new example to a Jupyter notebook by navigating the ``scico/examples`` directory and performing
-
-::
-
-   make notebooks/new_example.ipynb
-
-Alternatively, all examples can be run by calling
-
-::
-
-   make
+2. Add the basename of the script (i.e., without the pathname; in this case,
+``new_example.py``) to the appropriate section of
+``examples/scripts/index.rst``.
 
-from ``scico/examples``.
+3. Convert your new example to a Jupyter notebook by changing directory to the ``scico/examples`` directory and following the instructions in ``scico/examples/README.rst``.
 
-4.  Navigate to the ``data`` directory and add/commit the new Jupyter Notebook
+4.  Change directory to the ``data`` directory and add/commit the new Jupyter Notebook
 
 ::
 
    cd scico/data
    git add notebooks/new_example.ipynb
    git commit -m "Add new usage example"
 
-5.  Return to the base SCICO repository, ensure the ``main`` branch is checked out, add/commit the new script and updated submodule:
+5.  Return to the main SCICO repository, ensure the ``main`` branch is checked out, add/commit the new script and updated submodule:
 
 ::
 
@@ -254,9 +247,9 @@ Adding New Data
 
 The following steps show how to add new data, ``new_data.npz``, to the packaged data. We assume the SCICO repository has been cloned to ``scico/``.
 
-Note that the data is located in the scico-data submodule, which is symlinked to ``scico/data``.
-When adding new data, both the scico and scico-data repositories must be updated and
-kept in sync.
+Note that the data is located in the scico-data submodule, which is
+symlinked to ``scico/data``.  When adding new data, both the scico and
+scico-data repositories must be updated and kept in sync.
 
 
 1. Add the ``new_data.npz`` file to the ``scico/data`` directory.

docs/source/examples.rst

@@ -0,0 +1,57 @@
+.. _example_notebooks:
+
+Usage Examples
+--------------
+
+.. toctree::
+   :maxdepth: 1
+
+
+Computed Tomography
+===================
+
+.. toctree::
+   :maxdepth: 1
+
+   examples/admm_tv_ct
+   examples/admm_tv_ct_weighted
+   examples/pcg_ct
+   examples/admm_ppp_bm3d_svmbir_cg
+   examples/admm_ppp_bm3d_svmbir_prox
+
+
+Deconvolution
+=============
+
+.. toctree::
+   :maxdepth: 1
+
+   examples/admm_ppp_bm3d_deconvolution
+   examples/admm_ppp_dncnn_deconvolution
+   examples/admm_tv_deconvolution
+   examples/admm_tv_circ_deconvolution
+   examples/pgm_ppp_bm3d_deconvolution
+   examples/deconv_microscopy
+
+
+Sparse Coding
+=============
+
+.. toctree::
+   :maxdepth: 1
+
+   examples/admm_nonnegative_sparse_coding
+   examples/pgm_sparse_coding
+
+
+Miscellaneous
+=============
+
+.. toctree::
+   :maxdepth: 1
+
+   examples/admm_ppp_bm3d_demosaicing
+   examples/admm_tv_isotropic
+   examples/pgm_stepsize_blockarray
+   examples/pgm_stepsize_poisson
+   examples/pgm_tv_isotropic

docs/source/index.rst

@@ -10,7 +10,7 @@ SCICO Documentation
    install
    important_classes
    notes
-   examples/examples
+   examples
    API Reference <_autosummary/scico.rst>
    zreferences
 

examples/README.rst

@@ -4,22 +4,50 @@ SCICO Usage Examples
 This directory contains usage examples for the SCICO package. The primary form of these examples is the Python scripts in the directory ``scripts``. A corresponding set of Jupyter notebooks, in the directory ``notebooks``, is auto-generated from these usage example scripts.
 
 
+Building Notebooks
+------------------
+
+The scripts for building Jupyter notebooks from the source example scripts are currently only supported under Linux. All scripts described below should be run from this directory, i.e. ``[repo root]/examples``.
+
+
+The procedure for adding a adding a new notebook to the documentation is:
+
+1. Add an entry for the source file in ``scripts/index.rst``. Note that a script that is not listed in this index will not be converted into a notebook.
+
+2. Run ``makeindex.py`` to update the example scripts README file and the notebook index file in the docs.
+
+3. Run ``makejnb.py`` to build the new notebook, as well as any other notebooks that are out of date with respect to their source scripts, as determined by the respective file timestamps.
+
+4. Add and commit the new script, the ``scripts/index.rst`` script index file, the auto-generated
+``scripts/README.rst`` file and ``docs/source/examples.rst`` index file, and the new or updated notebooks (following the submodule handling procedure as described in the developer docs).
+
+
+The procedure for rebuilding notebook(s) after the source file(s) have been modified is:
+
+1. Run ``makejnb.py`` to build the new notebook, as well as any other notebooks that are out of date with respect to their source scripts, as determined by the respective file timestamps. Note that timestamps for files retrieved from version control may not be meaningful for this purpose. In such cases, ``touch`` the relevant source scripts to force updating on the next run of ``makejnb.py``.
+
+2. Add and commit the modified script(s), and the updated notebooks (following the submodule handling procedure as described in the developer docs).
+
+
 Management Utilities
 --------------------
 
-A number of additional files in this directory assist in the mangement of the usage examples:
+A number of files in this directory assist in the mangement of the usage examples:
 
 `examples_requirements.txt <examples_requirements.txt>`_
    Requirements file (as used by ``pip``) listing additional dependencies for the usage examples.
 
-`Makefile <Makefile>`_
-   A makefile allowing use of the command ``make`` to update auto-generated Jupyter notebooks. Run as ``make no-execute=true`` to update the notebooks without executing them.
-
 `makejnb.py <makejnb.py>`_
    An alternative to the makefile for updating the auto-generated Jupyter notebooks. Requires package ``ray`` to be installed. Notebooks are executed in parallel.
 
-`pytojnb <pytojnb>`_
+`makeindex.py <makeindex.py>`_
+   Auto-generate the docs example index ``docs/source/examples.rst`` from the example scripts index ``scripts/index.rst``.
+
+`Makefile <Makefile>`_
+   A makefile allowing use of the command ``make`` to update auto-generated Jupyter notebooks. Run as ``make no-execute=true`` to update the notebooks without executing them. Use of `makejnb.py` rather than this makefile is recommended.
+
+`pytojnb.sh <pytojnb.sh>`_
    Low-level python to Jupyter notebook conversion script. Used by both the makefile and `makejnb.py <makejnb.py>`_.
 
-`scriptcheck <scriptcheck>`_
+`scriptcheck.sh <scriptcheck.sh>`_
    Run all example scripts with a reduced number of iterations as a rapid check that they are functioning correctly.

examples/makeindex.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+
+import re
+from pathlib import Path
+
+src = "scripts/index.rst"
+
+# Make dict mapping script names to docstring header titles
+titles = {}
+scripts = list(Path("scripts").glob("*py"))
+for s in scripts:
+    prevline = None
+    with open(s, "r") as sfile:
+        for line in sfile:
+            if line[0:3] == "===":
+                titles[s.name] = prevline.rstrip()
+                break
+            else:
+                prevline = line
+
+
+# Build README in scripts directory
+dst = "scripts/README.rst"
+with open(dst, "w") as dstfile:
+    with open(src, "r") as srcfile:
+        for line in srcfile:
+            # Detect lines containing script filenames
+            m = re.match(r"(\s+)- ([^\s]+.py)", line)
+            if m:
+                prespace = m.group(1)
+                name = m.group(2)
+                title = titles[name]
+                print(
+                    "%s`%s <%s>`_\n%s   %s" % (prespace, name, name, prespace, title), file=dstfile
+                )
+            else:
+                print(line, end="", file=dstfile)
+
+
+# Build examples index for docs
+dst = "../docs/source/examples.rst"
+prfx = "examples/"
+with open(dst, "w") as dstfile:
+    print(".. _example_notebooks:\n", file=dstfile)
+    with open(src, "r") as srcfile:
+        for line in srcfile:
+            # Detect lines containing script filenames
+            m = re.match(r"(\s+)- ([^\s]+).py", line)
+            if m:
+                print(m.group(1) + prfx + m.group(2), file=dstfile)
+            else:
+                print(line, end="", file=dstfile)
+                # Add toctree statements after section headings
+                if line[0:3] == line[0] * 3 and line[0] in ["-", "="]:
+                    print("\n.. toctree::\n   :maxdepth: 1", file=dstfile)

examples/makejnb.py

@@ -1,5 +1,6 @@
 #!/usr/bin/env python
 
+import re
 from pathlib import Path
 
 import nbformat
@@ -11,19 +12,32 @@
     raise RuntimeError("The ray package is required to run this script")
 import os
 
-# source scripts
-scripts = list(Path("scripts").glob("*py"))
+# Read script names from index file
+scriptnames = []
+srcidx = "scripts/README.rst"
+with open(srcidx, "r") as idxfile:
+    for line in idxfile:
+        m = re.match(r"(\s+)- ([^\s]+.py)", line)
+        if m:
+            scriptnames.append(m.group(2))
+# Ensure list entries are unique
+scriptnames = list(set(scriptnames))
+
+# Construct script paths
+scripts = [Path("scripts") / Path(s) for s in scriptnames]
+
 notebooks = []
-# construct list of scripts that are have no corresponding notebook or
+# Construct list of scripts that are have no corresponding notebook or
 # are more recent than corresponding notebook
 for s in scripts:
     nb = Path("notebooks") / (s.stem + ".ipynb")
     if not nb.is_file() or s.stat().st_mtime > nb.stat().st_mtime:
-        # make notebook file
-        os.popen(f"./pytojnb {s} {nb}")
-        # add it to the list for execution
+        # Make notebook file
+        os.popen(f"./pytojnb.sh {s} {nb}")
+        # Add it to the list for execution
         notebooks.append(nb)
 
+
 ray.init()
 
 ngpu = 0
@@ -36,12 +50,12 @@
     ngpu = 0
     ar = ray.available_resources()
     if "GPU" in cr:
-        ngpu = ar["GPU"]
+        ngpu = int(ar["GPU"])
     if ngpu < 2:
         print("Warning: host has fewer than two GPUs available")
     print(f"Executing on {ngpu} GPUs")
 
-# function to execute each notebook with one gpu each
+# Function to execute each notebook with one GPU
 @ray.remote(num_gpus=1)
 def run_nb(fname):
     with open(fname) as f:

examples/scripts/README.rst

@@ -0,0 +1,58 @@
+Usage Examples
+--------------
+
+
+Computed Tomography
+===================
+
+   `admm_tv_ct.py <admm_tv_ct.py>`_
+      Few-View CT (ADMM w/ Total Variation)
+   `admm_tv_ct_weighted.py <admm_tv_ct_weighted.py>`_
+      Low-Dose CT (ADMM w/ Total Variation)
+   `pcg_ct.py <pcg_ct.py>`_
+      CT with Preconditioned Conjugate Gradient
+   `admm_ppp_bm3d_svmbir_cg.py <admm_ppp_bm3d_svmbir_cg.py>`_
+      CT Reconstruction (ADMM Plug-and-Play Priors w/ BM3D, SVMBIR+CG)
+   `admm_ppp_bm3d_svmbir_prox.py <admm_ppp_bm3d_svmbir_prox.py>`_
+      CT Reconstruction (ADMM Plug-and-Play Priors w/ BM3D, SVMBIR+Prox)
+
+
+Deconvolution
+=============
+
+   `admm_ppp_bm3d_deconvolution.py <admm_ppp_bm3d_deconvolution.py>`_
+      Image Deconvolution (ADMM Plug-and-Play Priors w/ BM3D)
+   `admm_ppp_dncnn_deconvolution.py <admm_ppp_dncnn_deconvolution.py>`_
+      Image Deconvolution (ADMM Plug-and-Play Priors w/ DnCNN)
+   `admm_tv_deconvolution.py <admm_tv_deconvolution.py>`_
+      Image Deconvolution (ADMM w/ Total Variation)
+   `admm_tv_circ_deconvolution.py <admm_tv_circ_deconvolution.py>`_
+      Image Deconvolution (ADMM w/ Total Variation and Circulant Blur)
+   `pgm_ppp_bm3d_deconvolution.py <pgm_ppp_bm3d_deconvolution.py>`_
+      Image Deconvolution (PGM Plug-and-Play Priors w/ BM3D)
+   `deconv_microscopy.py <deconv_microscopy.py>`_
+      Deconvolution Microscopy
+
+
+Sparse Coding
+=============
+
+   `admm_nonnegative_sparse_coding.py <admm_nonnegative_sparse_coding.py>`_
+      Non-negative Basis Pursuit DeNoising (ADMM)
+   `pgm_sparse_coding.py <pgm_sparse_coding.py>`_
+      Basis Pursuit DeNoising (Accelerated PGM)
+
+
+Miscellaneous
+=============
+
+   `admm_ppp_bm3d_demosaicing.py <admm_ppp_bm3d_demosaicing.py>`_
+      Image Demosaicing (ADMM Plug-and-Play Priors w/ BM3D)
+   `admm_tv_isotropic.py <admm_tv_isotropic.py>`_
+      Isotropic Total Variation
+   `pgm_stepsize_blockarray.py <pgm_stepsize_blockarray.py>`_
+      Non-negative Poisson Loss Reconstruction (APGM w/ adaptive PGMStepSize)
+   `pgm_stepsize_poisson.py <pgm_stepsize_poisson.py>`_
+      Non-negative Poisson Loss Reconstruction (APGM w/ adaptive PGMStepSize)
+   `pgm_tv_isotropic.py <pgm_tv_isotropic.py>`_
+      Isotropic Total Variation (Accelerated PGM)