Skip to content

Latest commit

 

History

History
284 lines (204 loc) · 7.24 KB

README.md

File metadata and controls

284 lines (204 loc) · 7.24 KB
Raw

ParameterSchedulers

Dev Build Status

ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:

using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Momentum, Exp(start = 1e-2, decay = 0.8))

Available Schedules

This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. The higher order schedules should make it so that you will rarely need to write a schedule from scratch.

You can read this paper for more information on the schedules below.

Schedule Description Type Example

Step(; start, decay, step_sizes)

Exponential decay by decay every step in step_sizes

 Decay 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Step(start = 1.0, decay = 0.8, step_sizes = [2, 3, 2]) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Exp(start, decay)

Exponential decay by decay every iteration

 Decay 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Exp(start = 1.0, decay = 0.5) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

CosAnneal(;l0, l1, period)

Cosine annealing

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = CosAnneal(l0 = 0.0, l1 = 1.0, period = 4) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

OneCycle(nsteps, maxval)

One cycle cosine

 Complex 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = OneCycle(10, 1.0) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Triangle(l0, l1, period)

Triangle wave function

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Triangle(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

TriangleDecay2(l0, l1, period)

Triangle wave function with half the amplitude every period

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = TriangleDecay2(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

TriangleExp(l0, l1, period, decay)

Triangle wave function with exponential amplitude decay at rate decay

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = TriangleExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Poly(start, degree, max_iter)

Polynomial decay at degree degree.

 Decay 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Poly(start = 1.0, degree = 2, max_iter = t[end]) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Inv(start, decay, degree)

Inverse decay at rate (1 + t * decay)^degree

 Decay 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Inv(start = 1.0, degree = 2, decay = 0.8) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

Sin(;l0, l1, period)

Sine function

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Sin(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

SinDecay2(l0, l1, period)

Sine function with half the amplitude every period

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = SinDecay2(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide

SinExp(l0, l1, period)

Sine function with exponential amplitude decay at rate decay

 Cyclic 
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = SinExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide