Skip to content

Commit

Permalink
move a sentence
Browse files Browse the repository at this point in the history
  • Loading branch information
mcabbott committed Nov 29, 2022
1 parent 259e663 commit 5e62649
Showing 1 changed file with 5 additions and 3 deletions.
8 changes: 5 additions & 3 deletions docs/src/training/training.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ for data in train_set
end
```

It is important that every `update!` step receives a newly gradient computed gradient.
This loop can also be written using the function [`train!`](@ref Flux.Train.train!),
but it's helpful to undersand the pieces first:

Expand All @@ -43,8 +42,8 @@ end

Fist recall from the section on [taking gradients](@ref man-taking-gradients) that
`Flux.gradient(f, a, b)` always calls `f(a, b)`, and returns a tuple `(∂f_∂a, ∂f_∂b)`.
In the code above, the function `f` is an anonymous function with one argument,
created by the `do` block, hence `grads` is a tuple with one element.
In the code above, the function `f` passed to `gradient` is an anonymous function with
one argument, created by the `do` block, hence `grads` is a tuple with one element.
Instead of a `do` block, we could have written:

```julia
Expand All @@ -58,6 +57,9 @@ structures are what Zygote calls "explicit" gradients.
It is important that the execution of the model takes place inside the call to `gradient`,
in order for the influence of the model's parameters to be observed by Zygote.

It is also important that every `update!` step receives a newly gradient computed gradient,
as this will be change whenever the model's parameters are changed, and for each new data point.

!!! compat "Explicit vs implicit gradients"
Flux ≤ 0.13 used Zygote's "implicit" mode, in which `gradient` takes a zero-argument function.
It looks like this:
Expand Down

0 comments on commit 5e62649

Please sign in to comment.