Introduce Constraints objects

[anriseth]

This needs

• Tests (may be some in IPNewtons we can use?)
• A review to check whether all the code I put here is necessary

As a future PR, we should introduce value! et al. to the NDifferentiableConstraints objects as well. But let's deal with that after we have moved IPNewton into Optim.

• Add Issue on value! et al. for Constraints

[codecov]

Codecov Report

Merging #38 into master will increase coverage by 6.46%.
The diff coverage is 97.82%.

@@            Coverage Diff             @@
##           master      #38      +/-   ##
==========================================
+ Coverage   77.38%   83.84%   +6.46%     
==========================================
  Files           9       10       +1     
  Lines         199      291      +92     
==========================================
+ Hits          154      244      +90     
- Misses         45       47       +2

Impacted Files	Coverage Δ
src/NLSolversBase.jl	100% <ø> (ø)	⬆️
src/interface.jl	81.63% <ø> (ø)	⬆️
src/objective_types/constraints.jl	97.82% <97.82%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e75b91...bfeb496. Read the comment docs.

[anriseth]

This is ready to go. Can someone review the changes, please?

[pkofod]

Looks good. I'm wondering if we should simply use the existing NDifferentiable setup for the constraints, and have the constraints be a wrapper around an NDifferentiable and ConstraintsBounds? Would buy us AD, counters, etc automatically. We can always iterate on this in the future...

[pkofod]

👍

[anriseth]

Looks good. I'm wondering if we should simply use the existing NDifferentiable setup for the constraints, and have the constraints be a wrapper around an NDifferentiable and ConstraintsBounds? Would buy us AD, counters, etc automatically. We can always iterate on this in the future...

Yes, if that's possible it will be very useful :)

[pkofod]

I don't see why it wouldn't? Currently a constraint is simply a function and it's derivative + it's bounds. What I suggest is basically

struct Constraints{Td, Tb}
    c::Td
    b::Tb
end

where c is then either a NonDifferentiable, a OnceDifferentiable or a TwiceDifferentiable. This seems much simpler than "duplicating" these types with a Constraints suffix.

I know where this comes from, I just figured I'd mention it.

[pkofod]

another relevant thing is to consider things like restricting parameters to a Ball for example. In https://github.com/cortner/ConstrainedOptim.jl we had a BallConstraint type used to project back into the ball.

[anriseth]

Great, it seems like we can quickly improve the current interface then 👍

Ah, and BallConstraint projection works better than supplying a constraint c(x) = sum(abs2, x) with c(x) \leq C ?

[pkofod]

Ah, and BallConstraint projection works better than supplying a constraint c(x) = sum(abs2, x) with c(x) \leq C ?

I guess it can be simpler in some cases, but "better" is sort of a multivalued thing here.

[longemen3000]

adding to this, one problem is the existence of the second order differenciable constraints, because that hessian is not of any NDifferentiable type. as my current understanding