Skip to content

Commit

Permalink
Remake Halley example
Browse files Browse the repository at this point in the history
  • Loading branch information
tansongchen committed Nov 26, 2024
1 parent 998db88 commit 35a20ff
Show file tree
Hide file tree
Showing 8 changed files with 137 additions and 369 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
<br />
<a href="https://github.com/SciML/ColPrac"><img src="https://img.shields.io/badge/contributor's%20guide-ColPrac-blueviolet" alt="ColPrac: Contributor's Guide on Collaborative Practices for Community Packages" /></a>
<a href="https://github.com/SciML/SciMLStyle"><img src="https://img.shields.io/badge/code%20style-SciML-blueviolet" alt="SciML Code Style" /></a>
<a href="https://doi.org/10.5281/zenodo.14226430"><img src="https://zenodo.org/badge/563952901.svg" alt="DOI" /></a>
</p>

<p align=center>
Expand Down
3 changes: 3 additions & 0 deletions docs/Project.toml
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
[deps]
Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
TaylorDiff = "b36ab563-344f-407b-a36a-4f200bebf99c"
TaylorIntegration = "92b13dbe-c966-51a2-8445-caca9f8a7d42"
TaylorSeries = "6aa5eb33-94cf-58f4-a9d0-e4b2c4fc25ea"
6 changes: 5 additions & 1 deletion docs/make.jl
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,13 @@ makedocs(;
prettyurls = get(ENV, "CI", "false") == "true",
canonical = "https://juliadiff.org/TaylorDiff.jl",
edit_link = "main",
assets = String[]),
assets = String[]
),
pages = [
"Home" => "index.md",
"Examples" => [
"Efficient Halley's method for nonlinear solving" => "examples/halley.md"
],
"Theory" => "theory.md",
"API" => "api.md"
])
Expand Down
76 changes: 76 additions & 0 deletions docs/src/examples/halley.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Efficient Halley's method for nonlinear solving

## Introduction

Say we have a system of $n$ equations with $n$ unknowns $f(x)=0$, and $f\in \mathbb R^n\to\mathbb R^n$ is sufficiently smooth.

Given a initial guess $x_0$, Newton's method finds a solution by iterating like

$$x_{i+1}=x_i-J(x_i)^{-1}f(x_i)$$

and this method converges quadratically.

We can make it converge faster using higher-order derivative information. For example, Halley's method iterates like

$$x_{i+1}=x_i-(a_i\odot a_i)\oslash(a_i-b_i/2)$$

where the vector multiplication and division $\odot,\oslash$ are defined element-wise, and term $a_i$ and $b_i$ are defined by $J(x_i)a_i = f(x_i)$ and $J(x_i)b_i = H(x_i)a_ia_i$.

Halley's method is proved to converge cubically, which is faster than Newton's method. Here, we demonstrate that with TaylorDiff.jl, you can compute the Hessian-vector-vector product $H(x_i)a_ia_i$ very efficiently, such that the Halley's method is almost as cheap as Newton's method per iteration.

## Implementation

We first define the two iteration schemes mentioned above:

```@example 1
using TaylorDiff, LinearAlgebra
import ForwardDiff
function newton(f, x, p; tol = 1e-12, maxiter = 100)
fp = Base.Fix2(f, p)
for i in 1:maxiter
fx = fp(x)
error = norm(fx)
println("Iteration $i: x = $x, f(x) = $fx, error = $error")
error < tol && return
J = ForwardDiff.jacobian(fp, x)
a = J \ fx
@. x -= a
end
end
function halley(f, x, p; tol = 1e-12, maxiter = 100)
fp = Base.Fix2(f, p)
for i in 1:maxiter
fx = f(x, p)
error = norm(fx)
println("Iteration $i: x = $x, f(x) = $fx, error = $error")
error < tol && return
J = ForwardDiff.jacobian(fp, x)
a = J \ fx
hvvp = derivative(fp, x, a, Val(2))
b = J \ hvvp
@. x -= (a * a) / (a - b / 2)
end
end
```

Note that in Halley's method, the hessian-vector-vector product is computed with `derivative(fp, x, a, Val(2))`. It is guaranteed that asymptotically this is only taking 2x more time compared to evaluating `fp(x)` itself.

Now we define some test function:

```@example 1
f(x, p) = x .* x - p
```

The Newton's method takes 6 iterations to converge:

```@example 1
newton(f, [1., 1.], [2., 2.])
```

While the Halley's method takes 4 iterations to converge:

```@example 1
halley(f, [1., 1.], [2., 2.])
```
264 changes: 0 additions & 264 deletions examples/halley.ipynb

This file was deleted.

Loading

0 comments on commit 35a20ff

Please sign in to comment.