Add the common interface code to ODEInterface?

[ChrisRackauckas]

Hey,
I was wondering if you would like to include the code to plug into the JuliaDiffEq "common interface". This common interface is a type-based interface which makes all of the differential equations solvers have a common solver signature. The full discussion is here:

SciML/Roadmap#5

Since the discussion is long, let me give a quick summary. There is a problem type which defines the Cauchy problem to solve:

type ODEProblem{uType,tType,isinplace,F} <: AbstractODEProblem{uType,tType,isinplace,F}
  f::F
  u0::uType
  tspan::Vector{tType}
end

https://github.com/JuliaDiffEq/DiffEqBase.jl/blob/common_interface/src/problems.jl#L26

Each algorithm is a subtype of AbstractODEAlgorithm and thus the command solve(prob,alg;kwargs....) dispatches to the appropriate solvers from the appropriate packages. For example, I have the following preliminary code in OrdinaryDiffEq.jl which implements this interface over ODEInterface:

The Algorithm types (may need to be renamed due to the name clashes with the functions):
https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/common_interface/src/algorithms.jl#L141

The solve dispatch
https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/common_interface/src/solve/ode_solve.jl#L191

A user can then seamlessly call ODEInterface's algorithms from this interface, as seen in these tests:
https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/common_interface/test/ode/ODEInterface_tests.jl

The tests show that even the common interface's keyword arguments will work, which is currently done by aliasing:
https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/common_interface/src/solve/ode_constants.jl#L57
https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/common_interface/src/solve/ode_solve.jl#L224

But if this code is in ODEInterface, it can more easily be tailored to get rid of this extra step.

This code needs to be removed from OrdinaryDiffEq.jl to get rid of the conditional dependency usage initialize_backend(:ODEInterface) (ODEInterface is not a full dependency because of the possibility of compilation issues).

What a user gets out of that is a solution object which acts very nicely:

sol[i] # Gets the ith value of u, the saved timeseries
sol.t[i] # Gets the timepoint of the ith save

A separate solution subtype ODEInterfaceSolution <: AbstractODESolution might be necessary to implement the interpolation part of the interface, which allows a user to:

sol(t) # Gets the interpolated solution at `t`

Thus without any prior knowledge of the solvers, users are able to easily and uniformly use advanced features like dense output from many different packages.

One benefit from joining the common interface is that there are plenty of tools which have been built or are being built on this interface. For example, DiffEqParamEstim.jl performs parameter estimation, and can use any of the backend ODE solver algorithms which are part of the common interface for the solving step. DiffEqDevTools.jl gives benchmarking which works on all algorithms in the common interface, and while the results show I've been able to beat the Fortran code in performance in nonstiff equations, that Radau solver is still one hell of an efficient stiff solving method:

https://github.com/JuliaDiffEq/DiffEqBenchmarks.jl

The amount of tools which are part of this are seemingly endless, and it would be nice for ODEInterface to join the group in a more formal way rather than being tagged onto OrdinaryDiffEq.jl as a conditional dependency. Thanks!

[ChrisRackauckas]

Any update on whether you would accept this code, or whether it should be its own package?

[luchr]

Sorry, a longer illness "knocked me out".

I'll guess the new location where I have to look is at https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/odeinterface_solve.jl , right?

I'll try to include the code. Is there a need to add an extra Package-Requirement to ODEInterface or is it possible to do this in an optional way?

[ChrisRackauckas]

Is there a need to add an extra Package-Requirement to ODEInterface or is it possible to do this in an optional way?

DiffEqBase should be all that's required. It's a small package which just includes the types for the problem and solution. When extern modules are a thing this could be done conditionally (or you can hack it using something like this: https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/backends.jl . But this is ill-advised since it doesn't play well with precompilation among other things (but you do have precompilation turned off)).

[ChrisRackauckas]

I'll guess the new location where I have to look is at https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/odeinterface_solve.jl , right?

Yes, with also this set of algorithms:

https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/blob/master/src/algorithms.jl#L116

Not sure how to name it since that clashes with the function names. Maybe just capitalize them like a type? Dop853 etc.?

[luchr]

Looking at https://github.com/JuliaDiffEq/DiffEqBase.jl/blob/master/REQUIRE
This adds the packages RecipesBase, Ranges, Parameters, RecursiveArrayTools, SimpleTraits to the requirements and julia 0.5.
At the moment ODEInterface was a (simple) standalone package.

I'll try to understand the DiffEqBase documentation to see if there is some way to detect, if this package was loaded before ODEInterface and do the DiffEqBase-neccessary stuff only if the package is detected.

[ChrisRackauckas]

This adds the packages RecipesBase, Ranges, Parameters, RecursiveArrayTools, SimpleTraits to the requirements and julia 0.5.

True. Those are small, but it might be good to keep this isolated.

I'll try to understand the DiffEqBase documentation to see if there is some way to detect, if this package was loaded before ODEInterface and do the DiffEqBase-neccessary stuff only if the package is detected.

There isn't really a safe way to do this. There's a bunch of discussions in Base conditional modules for doing this kind of thing, but as of now there's no real solution to the conditional dependency problem (in fact, the reason why this should leave OrdinaryDiffEq.jl is because it's being treated as a conditional dependency and the METADATA gatekeepers have asked for this to end because of the problems it can cause).

Maybe it would be best to have it as a separate package then? It's such a small interface, but maybe the LSODA.jl common interface parts could be there as well, depending on the author's wishes there.

[luchr]

I see the discussions at Julia Issue 15705 and Julia Issue 6195.

I want to have the "pros": Using the common DiffEqBase Interface without the "cons" (claiming to require 5 packages, that ODEInterface does not require).

I see two ways (at the moment):

1. Doing a "glue-package" (DiffEqBase-ODEInterface or whatever name)
2. Doing the very fragile and unsafe DiffEqBase-existence test inside ODEInterface and wait for further improvement (OPTIONAL file, etc.)

Do I miss one or more possibilities?

[mauro3]

Yep, it's not ideal. The current best-practice, namely either your point 1 or just straight inclusion, both have their down-sides. The former causes bloat in the package-system, the latter causes bloat in the package itself (through extra requirements).

[ChrisRackauckas]

I see the discussions at Julia Issue 15705 and Julia Issue 6195.

Oh there are plenty more. But yes, it's all the same discussion of don't do what you have as option 2, and sure there will be a way to do conditional modules happening sometime in the future, but there is no clear indication as to when it will ever happen.

So those are the two ways of doing it. One is to have another package for each add-on. Examples being UnitfulPlots.jl, StatPlots.jl, the JuliaOpt "JuMP" ecosystem, the way DiffEq is now structured, etc. which all add to methods of a base package. The other way is the old DifferentialEquations.jl / Plots.jl method using @eval for imports (see the backend link above). This is very fragile and very not recommended, but if you really want conditional modules it does (in certain cases) work.

[ChrisRackauckas]

Pkg3 has a solution to this "glue package" problem because you can optionally include other (unregistered) packages as requirements.

So since the Pkg3 solution will be to have this functionality in another repo or module, I think what I'll do is create a ODEInterfaceDiffEq with this glue functionality for now and register that, and then when Pkg3 comes around we can find out how to set the two up together the Pkg3 way.