Documentation for optimizer

.github/workflows/Documenter.yml

@@ -4,35 +4,35 @@ on:
   push:
     branches:
       - main
-    tags: '*'
   pull_request:
 
 jobs:
   build:
     name: Documentation
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
-      - uses: julia-actions/setup-julia@v1
-        with:
-          version: '1'
-      - name: Install dependencies
-        run: |
+      - uses: actions/checkout@v3
+      - run: |
+          sudo apt-get install imagemagick
+          sudo apt-get install poppler-utils
+          sudo apt-get install texlive-xetex
+          sudo apt-get install texlive-science
+          mkdir docs/src/assets
+          make all -C docs/src/tikz
+      - uses: julia-actions/setup-julia@latest
+      - run: |
           julia --project=docs -e '
             using Pkg
             Pkg.develop(PackageSpec(path=pwd()))
             Pkg.instantiate()
             Pkg.build()
-            Pkg.precompile()'
-      - name: Run doctests
-        run: |
-          julia --project=docs -e '
+            Pkg.precompile()
             using Documenter: doctest
             using GeometricMachineLearning
             doctest(GeometricMachineLearning)'
-      - name: Build and deploy Documentation
-        run: julia --project make.jl
-        working-directory: docs
+          julia --project=docs docs/make.jl
+      - uses: julia-actions/julia-buildpkg@v1
+      - uses: julia-actions/julia-docdeploy@v1
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}
+          DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}

README.md

@@ -1,4 +1,9 @@
-# Geometric Machine Learning
+<picture>
+  <source media="(prefers-color-scheme: light)" srcset="https://github.com/JuliaGNI/GeometricMachineLearning.jl/assets/55493704/8d6d1410-b857-4e0f-8609-50e43be9a268">
+  <source media="(prefers-color-scheme: dark)" srcset="https://github.com/JuliaGNI/GeometricMachineLearning.jl/assets/55493704/014929d1-2297-4b2c-9359-58cadbb03a0e">
+  <img alt="Shows a black logo in light color mode and a white one in dark color mode.">
+</picture>
+
 
 [![Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://juliagni.github.io/GeometricMachineLearning.jl/stable)
 [![Latest](https://img.shields.io/badge/docs-latest-blue.svg)](https://juliagni.github.io/GeometricMachineLearning.jl/latest)
@@ -52,4 +57,4 @@ plot(trajectory_to_plot)
 The optimization of the first layer is done on the Stiefel Manifold $St(n, N)$, and the optimizer used is the manifold version of Adam (see (Brantner, 2023)).
 
 ## References
-- Brantner B. Generalizing Adam To Manifolds For Efficiently Training Transformers[J]. arXiv preprint arXiv:2305.16901, 2023.
+- Brantner B. Generalizing Adam To Manifolds For Efficiently Training Transformers[J]. arXiv preprint arXiv:2305.16901, 2023.

docs/Makefile

@@ -0,0 +1,15 @@
+
+.PHONY: documenter images
+
+all: images documenter
+
+documenter:
+	julia --color=yes --project=. make.jl
+
+images:
+	$(MAKE) all   -C src/tikz
+
+clean:
+	$(MAKE) empty -C src/tikz
+	rm -Rf build
+	rm -Rf src/tutorial

docs/Project.toml

@@ -1,4 +1,21 @@
 [deps]
+AbstractNeuralNetworks = "60874f82-5ada-4c70-bd1c-fa6be7711c8a"
+BandedMatrices = "aae01518-5342-5314-be14-df237901396f"
+ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
+Distances = "b4f34e82-e78d-54a5-968a-f98e89d6e8f7"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
+DocumenterCitations = "daee34ce-89f3-4625-b898-19384cb65244"
+ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
+GeometricBase = "9a0b12b7-583b-4f04-aa1f-d8551b6addc9"
+GeometricEquations = "c85262ba-a08a-430a-b926-d29770767bf2"
+GeometricIntegrators = "dcce2d33-59f6-5b8d-9047-0defad88ae06"
 GeometricMachineLearning = "194d25b2-d3f5-49f0-af24-c124f4aa80cc"
+InteractiveUtils = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
+KernelAbstractions = "63c18a36-062a-441e-b654-da1e3ab1ce7c"
+LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
+NNlib = "872c559c-99b0-510c-b3b7-b6c96a88d5cd"
 Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
+ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
+Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+TimerOutputs = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
+Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f"

docs/make.jl

@@ -1,8 +1,15 @@
 using GeometricMachineLearning
 using Documenter
+using DocumenterCitations
+# using Weave
 
+# this is necessary to avoid warnings. See https://documenter.juliadocs.org/dev/man/syntax/
+ENV["GKSwstype"] = "100"
+
+bib = CitationBibliography(joinpath(@__DIR__, "src", "GeometricMachineLearning.bib"))
 
 makedocs(;
+    plugins=[bib],
     modules=[GeometricMachineLearning],
     authors="Michael Kraus, Benedikt Brantner",
     repo="https://github.com/JuliaGNI/GeometricMachineLearning.jl/blob/{commit}{path}#L{line}",
@@ -11,28 +18,39 @@ makedocs(;
         prettyurls=get(ENV, "CI", "false") == "true",
         canonical="https://juliagni.github.io/GeometricMachineLearning.jl",
         assets=String[],
+        # specifies that we do not display the package name again (it's already in the logo)
+        sidebar_sitename=false,
     ),
     pages=[
         "Home" => "index.md",
         "Architectures" => [
             "SympNet" => "architectures/sympnet.md",
         ],
         "Manifolds" => [
+            "Concepts from General Topology" => "manifolds/basic_topology.md",
+            "General Theory on Manifolds" => "manifolds/manifolds.md",
+            "The Inverse Function Theorem" => "manifolds/inverse_function_theorem.md",
+            "The Submersion Theorem" => "manifolds/submersion_theorem.md",
             "Homogeneous Spaces" => "manifolds/homogeneous_spaces.md",
             "Stiefel" => "manifolds/stiefel_manifold.md",
             "Grassmann" => "manifolds/grassmann_manifold.md",
+            "Differential Equations and the EAU theorem" => "manifolds/existence_and_uniqueness_theorem.md",
             ],
         "Arrays" => [
             "Global Tangent Space" => "arrays/stiefel_lie_alg_horizontal.md",
         ],
-        "Optimizer Framework" => "Optimizer.md",
+        "Optimizer Framework" => [
+            "Optimizers" => "Optimizer.md",
+            "General Optimization" => "optimizers/general_optimization.md",
+        ],
         "Optimizer Functions" => [
             "Horizontal Lift" => "optimizers/manifold_related/horizontal_lift.md",
             "Global Sections" => "optimizers/manifold_related/global_sections.md",
             "Retractions" => "optimizers/manifold_related/retractions.md",
             "Geodesic Retraction" => "optimizers/manifold_related/geodesic.md",
             "Cayley Retraction" => "optimizers/manifold_related/cayley.md",
             "Adam Optimizer" => "optimizers/adam_optimizer.md",
+            "BFGS Optimizer" => "optimizers/bfgs_optimizer.md",
             ],
         "Special Neural Network Layers" => [
             "Attention" => "layers/attention_layer.md",
@@ -49,9 +67,11 @@ makedocs(;
             "Projection and Reduction Error" => "reduced_order_modeling/projection_reduction_errors.md",
         ],
         "Tutorials" =>[
+            "Sympnets" => "tutorials/sympnet_tutorial.md",
             "Linear Wave Equation" => "tutorials/linear_wave_equation.md",
             "MNIST" => "tutorials/mnist_tutorial.md",
         ],
+        "References" => "references.md",
         "Library" => "library.md",
     ],
 )

docs/src/GeometricMachineLearning.bib

@@ -0,0 +1,153 @@
+@article{brantner2023generalizing,
+  title={Generalizing Adam To Manifolds For Efficiently Training Transformers},
+  author={Brantner, Benedikt},
+  journal={arXiv preprint arXiv:2305.16901},
+  year={2023}
+}
+
+@article{jin2020sympnets,
+  title={SympNets: Intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems},
+  author={Jin, Pengzhan and Zhang, Zhen and Zhu, Aiqing and Tang, Yifa and Karniadakis, George Em},
+  journal={Neural Networks},
+  volume={132},
+  pages={166--179},
+  year={2020},
+  publisher={Elsevier}
+}
+
+@article{jin2022optimal,
+  title={Optimal unit triangular factorization of symplectic matrices},
+  author={Jin, Pengzhan and Lin, Zhangli and Xiao, Bo},
+  journal={Linear Algebra and its Applications},
+  year={2022},
+  publisher={Elsevier}
+}
+
+@book{hairer2006geometric,
+  title={Geometric Numerical integration: structure-preserving algorithms for ordinary differential equations},
+  author={Hairer, Ernst and Lubich, Christian and Wanner, Gerhard},
+  year={2006},
+  publisher={Springer}
+}
+
+
+@book{leimkuhler2004simulating,
+  title={Simulating hamiltonian dynamics},
+  author={Leimkuhler, Benedict and Reich, Sebastian},
+  number={14},
+  year={2004},
+  publisher={Cambridge university press}
+}
+
+@book{lang2012fundamentals,
+  title={Fundamentals of differential geometry},
+  author={Lang, Serge},
+  volume={191},
+  year={2012},
+  publisher={Springer Science \& Business Media}
+}
+
+@book{lipschutz1965general,
+    title={General Topology},
+    author={Seymour Lipschutz},
+    year={1965},
+    publisher={McGraw-Hill Book Company},
+    location={New York City, New York}
+}
+
+@book{bishop1980tensor,
+    title={Tensor Analysis on Manifolds},
+    author={Richard L. Bishop, Samuel I. Goldberg},
+    year={1980},
+    publisher={Dover Publications},
+    location={Mineola, New York}
+}
+
+@book{wright2006numerical,
+  title={Numerical optimization},
+  author={Stephen J. Wright, Jorge Nocedal},
+  year={2006},
+  publisher={Springer Science+Business Media},
+  location={New York, NY}
+}
+
+@article{fresca2021comprehensive,
+  title={A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs},
+  author={Fresca, Stefania and Dede’, Luca and Manzoni, Andrea},
+  journal={Journal of Scientific Computing},
+  volume={87},
+  pages={1--36},
+  year={2021},
+  publisher={Springer}
+}
+
+@article{buchfink2023symplectic,
+  title={Symplectic model reduction of Hamiltonian systems on nonlinear manifolds and approximation with weakly symplectic autoencoder},
+  author={Buchfink, Patrick and Glas, Silke and Haasdonk, Bernard},
+  journal={SIAM Journal on Scientific Computing},
+  volume={45},
+  number={2},
+  pages={A289--A311},
+  year={2023},
+  publisher={SIAM}
+}
+
+@article{peng2016symplectic,
+  title={Symplectic model reduction of Hamiltonian systems},
+  author={Peng, Liqian and Mohseni, Kamran},
+  journal={SIAM Journal on Scientific Computing},
+  volume={38},
+  number={1},
+  pages={A1--A27},
+  year={2016},
+  publisher={SIAM}
+}
+
+@article{luong2015effective,
+  title={Effective approaches to attention-based neural machine translation},
+  author={Luong, Minh-Thang and Pham, Hieu and Manning, Christopher D},
+  journal={arXiv preprint arXiv:1508.04025},
+  year={2015}
+}
+
+@article{bahdanau2014neural,
+  title={Neural machine translation by jointly learning to align and translate},
+  author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua},
+  journal={arXiv preprint arXiv:1409.0473},
+  year={2014}
+}
+
+@article{greif2019decay,
+  title={Decay of the Kolmogorov N-width for wave problems},
+  author={Greif, Constantin and Urban, Karsten},
+  journal={Applied Mathematics Letters},
+  volume={96},
+  pages={216--222},
+  year={2019},
+  publisher={Elsevier}
+}
+
+@article{blickhan2023registration,
+  title={A registration method for reduced basis problems using linear optimal transport},
+  author={Blickhan, Tobias},
+  journal={arXiv preprint arXiv:2304.14884},
+  year={2023}
+}
+
+@article{lee2020model,
+  title={Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders},
+  author={Lee, Kookjin and Carlberg, Kevin T},
+  journal={Journal of Computational Physics},
+  volume={404},
+  pages={108973},
+  year={2020},
+  publisher={Elsevier}
+}
+
+@article{vaswani2017attention,
+  title={Attention is all you need},
+  author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Lukasz and Polosukhin, Illia},
+  journal={Advances in neural information processing systems},
+  volume={30},
+  year={2017}
+}

docs/src/Optimizer.md

@@ -4,7 +4,9 @@ In order to generalize neural network optimizers to [homogeneous spaces](manifol
 
 Starting from an element of the tangent space $T_Y\mathcal{M}$[^1], we need to perform two mappings to arrive at $\mathfrak{g}^\mathrm{hor}$, which we refer to by $\Omega$ and a red horizontal arrow:
 
-![](images/general_optimization_with_boundary.png)
+[^1]: In practice this is obtained by first using an AD routine on a loss function $L$, and then computing the Riemannian gradient based on this. See the section of the [Stiefel manifold](manifolds/stiefel_manifold.md) for an example of this.
+
+![](tikz/general_optimization_with_boundary.png)
 
 Here the mapping $\Omega$ is a [horizontal lift](optimizers/manifold_related/horizontal_lift.md) from the tangent space onto the **horizontal component of the Lie algebra at $Y$**. 
 
@@ -13,6 +15,10 @@ The red line maps the horizontal component at $Y$, i.e. $\mathfrak{g}^{\mathrm{h
 The $\mathrm{cache}$ stores information about previous optimization steps and is dependent on the optimizer. The elements of the $\mathrm{cache}$ are also in $\mathfrak{g}^\mathrm{hor}$. Based on this the optimer ([Adam](optimizers/adam_optimizer.md) in this case) computes a final velocity, which is the input of a [retraction](optimizers/manifold_related/retractions.md). Because this *update* is done for $\mathfrak{g}^{\mathrm{hor}}\equiv{}T_Y\mathcal{M}$, we still need to perform a mapping, called `apply_section` here, that then finally updates the network parameters. The two red lines are described in [global sections](optimizers/manifold_related/global_sections.md).
 
 ## References 
-- Brantner B. Generalizing Adam To Manifolds For Efficiently Training Transformers[J]. arXiv preprint arXiv:2305.16901, 2023.
 
-[^1]: In practice this is obtained by first using an AD routine on a loss function $L$, and then computing the Riemannian gradient based on this. See the section of the [Stiefel manifold](manifolds/stiefel_manifold.md) for an example of this.
+```@bibliography
+Pages = []
+Canonical = false
+
+brantner2023generalizing
+```