Ola @joaogvcarneiro! I have a test failing for this branch. It's related to the variable integrator scenario, which I believe you wrote?

I made some changes to the adaptive integrators. The math should be the same, but the numerics have shifted around. I think the failing test is testing to an accuracy that is within the integrator error, so the updated numerics trigger the comparison failure. That's only my intuition, so I'd appreciate your take on it if you have a minute.

Ola @joaogvcarneiro! I have a test failing for this branch. It's related to the variable integrator scenario, which I believe you wrote?

I made some changes to the adaptive integrators. The math should be the same, but the numerics have shifted around. I think the failing test is testing to an accuracy that is within the integrator error, so the updated numerics trigger the comparison failure. That's only my intuition, so I'd appreciate your take on it if you have a minute.

I think your intuition is correct. You could vary the timestep to see if the results improve. Further, all integrators have an upper bound on the error that depends on the timestep, so you could do some correlation to determine if the errors are changing as expected. This link is a good resource. I hope this helps!

Ola @joaogvcarneiro! I have a test failing for this branch. It's related to the variable integrator scenario, which I believe you wrote?
I made some changes to the adaptive integrators. The math should be the same, but the numerics have shifted around. I think the failing test is testing to an accuracy that is within the integrator error, so the updated numerics trigger the comparison failure. That's only my intuition, so I'd appreciate your take on it if you have a minute.

I think your intuition is correct. You could vary the timestep to see if the results improve. Further, all integrators have an upper bound on the error that depends on the timestep, so you could do some correlation to determine if the errors are changing as expected. This link is a good resource. I hope this helps!

The integrators do behave as expected:

The evolution of position error and integration time evolve as I would expect as a function of the time step, order, and tolerance.

I will try lowering the time-step of the scenario as you suggested and see if that helps. I suspect that doing so will produce results closer to the real truth, but maybe not closer to the "truth" value used in the test (since the test "truth" could have been obtained with integration error larger than the test accuracy). If that's the case, is it ok if I update the truth value in the test and change the accuracy to reflect the expected integrator error?

Yes, it is ok to update the truth value for this test. In the commit just explain why this is being updated, and how the new truth value was computed. Thanks for doing the time step refinement test to validate the expected behavior.

If we're changing the truth values, then I'd argue for removing this "magic number" approach all together. We're seeing precisely why hardcoding truth values can be so confusing: we don't know where these results come from, what integrator was used, what the tolerance was, etc. If we change these numbers to some others with a lower tolerance, we're going to end up in the same situation a few years down the line.

The test checks the position for some orbits defined in the variable integrator scenario. Couldn't we find the analytical value for the position at any point in the orbit? That would be the actual truth value. It would obviously take some time to do the math, but once it's done, we wouldn't have to change it ever again and we'd avoid all these issues with magic numbers.

I haven’t looked at the details of the test yet. In many cases we had separate code that computed the numerical truth values. That is a documented approach. In other cases the test was rather a consistency check. In this branch case, if we assume Keplerian motion and we solve Kepler’s equation to sufficient accuracy, we could use that as a truth value. We check if we are close to that value.

However, note that this truth isn’t the correct truth for an integrator. Rather, the better truth would be to duplicate the integration method to have consistent math. But, I think this is overkill for this branch.

I haven’t looked at the details of the test yet. In many cases we had separate code that computed the numerical truth values. That is a documented approach. In other cases the test was rather a consistency check. In this branch case, if we assume Keplerian motion and we solve Kepler’s equation to sufficient accuracy, we could use that as a truth value. We check if we are close to that value.

However, note that this truth isn’t the correct truth for an integrator. Rather, the better truth would be to duplicate the integration method to have consistent math. But, I think this is overkill for this branch.

For posterity: I ended up computing a reference analytical solution for this test (it's a simple 2body problem). I used some test tolerances that make sense and should be robust to numeric jitter.