New optimization algorithms (#117)

* Add Linearized ADMM and PDHG solvers * Add proximal map initialization * Improve parameters * Update submodule to include new notebooks * Improve test coverage * Respect style guidelines * Fix some style violations and other minor changes * Style compliance fix * Simplify step implementation * Bug fix
lanl · Dec 7, 2021 · 2f28bfc · 2f28bfc
1 parent 50e4935
commit 2f28bfc
Show file tree

Hide file tree

Showing 10 changed files with 1,501 additions and 3 deletions.
diff --git a/data b/data
diff --git a/docs/source/optimizer.rst b/docs/source/optimizer.rst
@@ -87,8 +87,8 @@ the :math:`\mb{x}`-update,
     .. math::
        :label: eq:admm_x_step
 
-        \argmin_{\mb{x}} \; f(\mb{x}) + \sum_i \frac{\rho_i}{2}
-        \norm{\mb{z}^{(k)}_i - \mb{u}^{(k)}_i - C_i \mb{x}}_2^2 \;.
+	\argmin_{\mb{x}} \; f(\mb{x}) + \sum_i \frac{\rho_i}{2}
+	\norm{\mb{z}^{(k)}_i - \mb{u}^{(k)}_i - C_i \mb{x}}_2^2 \;.
 
 
 The available solvers for this problem are:
@@ -119,6 +119,56 @@ reference page for :mod:`scico.admm`.
 
 
 
+Linearized ADMM
+---------------
+
+Linearized ADMM algorithm :cite:`yang-2012-linearized`
+:cite:`parikh-2014-proximal` (Sec. 4.4.2) is an algorithm for solving
+problems of the form
+
+    .. math::
+	\argmin_{\mb{x}} \; f(\mb{x}) + g(C \mb{x}) \;,
+
+where :math:`f` and :math:`g` are are convex (but not necessarily smooth)
+functions. Although convergence per iteration is typically significantly
+worse than that of ADMM, the :math:`\mb{x}`-update,
+
+    .. math::
+
+       \mathrm{prox}_{\mu f} \left( \mb{x}^{(k)} - (\mu / \nu) C^T
+       \left(C \mb{x}^{(k)} - \mb{z}^{(k)} + \mb{u}^{(k)} \right) \right)
+
+is can be much cheaper than that of ADMM, giving Linearized ADMM competitive
+time convergence performance.
+
+The SCICO Linearized ADMM solver, :class:`.LinearizedADMM`,
+requires :math:`f` and :math:`g` to be instances of :class:`.Functional`,
+and to have a proximal operator defined (:meth:`.Functional.prox`), and
+:math:`C` is required to be an instance of :class:`.LinearOperator`.
+
+
+
+PDHG
+----
+
+The Primal–Dual Hybrid Gradient (PDHG) algorithm
+:cite:`esser-2010-general` :cite:`chambolle-2010-firstorder`
+:cite:`pock-2011-diagonal` solves problems of the form
+
+    .. math::
+	\argmin_{\mb{x}} \; f(\mb{x}) + g(C \mb{x}) \;,
+
+where :math:`f` and :math:`g` are are convex (but not necessarily smooth)
+functions. The algorithm has similar advantages over ADMM to those of Linearized ADMM, but typically exhibits better convergence properties.
+
+The SCICO PDHG solver, :class:`.PDHG`,
+requires :math:`f` and :math:`g` to be instances of :class:`.Functional`,
+and to have a proximal operator defined (:meth:`.Functional.prox`), and
+:math:`C` is required to be an instance of :class:`.LinearOperator`.
+
+
+
+
 PGM
 ---
 

diff --git a/docs/source/references.bib b/docs/source/references.bib
@@ -97,6 +97,19 @@ @Article {buzzard-2018-plug
   doi =		 {10.1137/17M1122451}
 }
 
+@Article {chambolle-2010-firstorder,
+  author =	 {Antonin Chambolle and Thomas Pock},
+  title =	 {A First-Order Primal-Dual Algorithm for Convex
+                  Problems with~Applications to Imaging},
+  journal =      {Journal of Mathematical Imaging and Vision},
+  doi =		 {10.1007/s10851-010-0251-1},
+  year =	 2010,
+  month =	 Dec,
+  volume =	 40,
+  number =	 1,
+  pages =	 {120--145}
+}
+
 @Article {clinthorne-1993-preconditioning,
   author =	 {Clinthorne, Neal H. and Pan, Tin-Su and Chiao,
                   Ping-Chun and Rogers, W. Leslie and Stamos, John A.},
@@ -128,6 +141,20 @@ @InProceedings{dabov-2008-image
   doi =		 {10.1117/12.766355}
 }
 
+@Article {esser-2010-general,
+  author =	 {Ernie Esser and Xiaoqun Zhang and Tony F. Chan},
+  title =	 {A General Framework for a Class of First Order
+                  Primal-Dual Algorithms for Convex Optimization in
+                  Imaging Science},
+  journal =	 {SIAM Journal on Imaging Sciences},
+  doi =	 {10.1137/09076934x},
+  year =	 2010,
+  month =	 Jan,
+  volume =	 3,
+  number =	 4,
+  pages =	 {1015--1046}
+}
+
 @Article {daubechies-2004-iterative,
   title =	 {An iterative thresholding algorithm for linear
                   inverse problems with a sparsity constraint},
@@ -278,6 +305,19 @@ @Misc {pfister-2021-scico
   year =	 2021
 }
 
+@InProceedings {pock-2011-diagonal,
+  author =	 {Thomas Pock and Antonin Chambolle},
+  title =	 {Diagonal preconditioning for first order primal-dual
+                  algorithms in convex optimization},
+  booktitle =	 {Proceedings of the International Conference on
+                  Computer Vision (ICCV)},
+  doi =	 {10.1109/iccv.2011.6126441},
+  pages =	 {1762--1769},
+  year =	 2011,
+  month =	 Nov,
+  address =	 {Barcelona, Spain}
+}
+
 @Article {sauer-1993-local,
   title =	 {A local update strategy for iterative reconstruction
                   from projections},
@@ -352,6 +392,19 @@ @Book {voelz-2011-computational
   isbn =	 9780819482044,
 }
 
+@article{yang-2012-linearized,
+  author =	 {Junfeng Yang and Xiaoming Yuan},
+  title =	 {Linearized augmented {L}agrangian and alternating
+                  direction methods for nuclear norm minimization},
+  journal =	 {Mathematics of Computation},
+  doi =	 {10.1090/s0025-5718-2012-02598-1},
+  year =	 2012,
+  month =	 mar,
+  volume =	 82,
+  number =	 281,
+  pages =	 {301--329}
+}
+
 @Article {zhang-2017-dncnn,
   author =	 {Kai Zhang and Wangmeng Zuo and Yunjin Chen and Deyu
                   Meng and Lei Zhang},