uncertainties computation for non-minuit backends

[lukasheinrich]

We need to decide how we should report

• parameter uncertainties
• parameter covarianecs

in backends that are not minuit. There are two options

1. try to be compatible with minuit

we can do this see notebooks in #1233 his makes it easy to compare to minuit b/c we're computing the same thing. This deppens on internals in minuit like it's map to "internal" unbounded parameters etcc.

2. just go with standard numbers

e.g. parameter covariance is computed from the hessian (via inverse) and parameter uncertainties form the diag elements of the covarianec. This seems more principled but ignores boundary effects (though it's unclear to me that what MINUIT is doing is "more" correct. There might be an argument that both are not hyper-correct anyways and the "real" error comes from a MINOS lilke line search. In that case we can just do the more principled version and support an API for lhood scans

The two approaches differ numerically whenever a minimum is close to a parameter boundary. In the "bulk" they should be quite similar.

I personally lean towards 2) with the knowledge that we can in principle reproduce 1) if necessary. However this will require more communication when comparing results.

Thoughts @alexander-held @matthewfeickert @kratsg @cranmer @HDembinski (for possible comment on minuit philoosophy)

[alexander-held]

I am leaning towards 2, mostly because I think that is the implementation most users will expect. I will have to go through the notebook in more detail to understand the cases where this distinction matters. @nsmith- might also have some input based on earlier discussions.

[nsmith-]

I would go with option 2 myself, i.e. sqrt(diag(linalg.inv(hessian))). After all, most likely such numbers would not be the final story for uncertainties: usually you would need to choose which subset of parameters are to be reported and do a line search on each parameter of interest while re-fitting the remaining nuisance parameters ("profiling")

[matthewfeickert]

I personally lean towards 2) with the knowledge that we can in principle reproduce 1) if necessary. However this will require more communication when comparing results.

These are my thoughts as well, @lukasheinrich. I think between your very nice learn notebook you have going in PR #1233 and additional clarification and notes in the docs this should be reasonable to achieve.

[HDembinski]

Minos is not more correct than Hesse in theory, but the Minos interval is invariant to monotonic transformations, which means in practice that you may get a better error estimate when you set limits on your parameter. Both Minos and Hesse are theoretically wrong when evaluated near a boundary, since this violates the assumptions under which these error estimates were derived.

[HDembinski]

Other minimisers do not use the transformation technique when they limit a parameter, I only ever saw that technique used in MINUIT. That is probably why you get a different result near the boundary. I am not sure that the answer of just inverting the Hessian near the boundary is more or less meaningful than the MINUIT one. I personally would not mirror the MINUIT behavior, I would just do the standard inversion of the Hessian for other minimisers and add a note about this in the docs.

[HDembinski]

Thinking more about it, it is possible to do a numerical test to decide which version is more correct.

In repeated identical experiments, the interval [value - error, value + error] should contain the true value in 68 % of the cases. This can be checked for both the MINUIT interval and the one where you just compute and invert the Hesse matrix and then compute error from sqrt(diagonal).