Add jacobian, at last?

[mcabbott]

This adds a Jacobian function.

Compared to #747 this one:

• has tests
• accepts multiple arguments
• shouldn't fail if back returns nothing
• inserts vec a few more places
• has a method which accepts implicit Params, and returns Grads. (This was the only part I actually needed in real life.)
• now works on the GPU too!

Compared to #414 this one:

• always inserts vec, never makes higher-dimensional arrays
• runs on current Zygote
• has tests.

Compared to #235 this one:

• doesn't try to provide numerical jacobian
• doesn't alter testing infrastructure
• doesn't provide jacobian!, nor any code for structured matrices.

This does not address #564's concerns about functions which return a tuple of arrays. Only functions returning an array (or a scalar) are permitted. Similar considerations might give sensible jacobians when the argument of a function is a tuple, or some other struct, but for now these are handled by putting up a giant warning sign.

Nothing in the file utils.jl seems to have any tests at all. So I also added tests for hessian.

And, while I was there, made hessian actually accept a real number like its docstring promises. (Hence this closes #891.) And, made a version that is reverse-over-reverse, using this jacobian, which works less well of course but may as well exist to test things. (See for example #865.) Ideally there would be a pure-Zygote version using its own forward mode, but I didn't write that.

Fixes #51, fixes #98, fixes #413. Closes #747.

[DhairyaLGandhi]

I'd prefer something that is somewhat more generic rather than using if blocks for type checking. Not a big deal, but the approach in #747 seems better suited to be extended using our existing tools.

[mcabbott]

approach in #747

Can you say what differences you see in approach? I tried other things and we can discuss them, but circled back to pretty small adjustments.

seems better suited to be extended using our existing tools

Which tools are you referring to?

[CarloLucibello]

looks very good, had only minor comments

[DhairyaLGandhi]

Why do we need to explicitly copy?

[CarloLucibello]

Julia 1.3 tests failing due to CUDA compatibility issues is annoying. Maybe we should drop < 1.5 support

[mcabbott]

I believe that if we move tests to 1.4 they will get the new resolver, maybe that's the minimal step? Maybe not this PR though.

What are the rules here, do all PRs need at least one approval? And is Bors expected to obey commands from me?

[mcabbott]

This is not quite true, we do deal with non real output in several cases in sciml

MWE?

This docstring just matches the errors:

julia> gradient(identity, [1,2])
ERROR: Output should be scalar; gradients are not defined for output [1, 2]

julia> gradient(identity, 1+im)
ERROR: Output is complex, so the gradient is not defined.

[DhairyaLGandhi]

https://github.com/SciML/DiffEqSensitivity.jl/blob/e04f8a1a300ed2dbc2e0e8da520bb3ef6d9548a2/src/concrete_solve.jl#L279 outputs a vector in the forward pass.

[mcabbott]

https://github.com/SciML/DiffEqSensitivity.jl/blob/e04f8a1a300ed2dbc2e0e8da520bb3ef6d9548a2/src/concrete_solve.jl#L279 outputs a vector in the forward pass.

Sure. But I meant an example for gradient, clearly.

[DhairyaLGandhi]

I mean, I've shown you it working in the ecosystem

[mcabbott]

I mean, I've shown you it working in the ecosystem

Sorry, what does "it" refer to here?

The file you linked to does not seem to contain the function gradient. It contains many other functions, and I totally agree that other functions can accept vectors. In fact, in my example, there is also such a function!

[CarloLucibello]

bors r+

[bors]

Build succeeded:

• buildkite/zygote-dot-jl