Control of Assertion Level when running tests

[stevelinton]

It would be nice to be able to set the assertion level via an option in Test and TestDirectory. Level 0 makes it a more reasonable benchmark and lower levels might help tests run faster (for instance in continuous integration or git bisect).

It is also theoretically possible to have a bug that only shows up at lower assertion levels (when an assertion has side-effects that the code actually depends on).

This arises from issue #524

[fingolfin]

Fully agreed.

And somewhat orthogonal to this: I can remember several instances where I tried to manually debug a test failure, then couldn't reproduce it -- and only after some time realized that I forgot to increment the assertion level, which was needed to reproduce the bug.
So, perhaps we can print the used assertion level when we start running the tests? E.g. "Running tests at assertion level 2"

[frankluebeck]

The change of the assertion level and printing of GAP stones and resetting of the main random source are not done by Test but by the START_TEST and STOP_TEST functions.

I never understood why the assertion level is changed by these functions at all. Shouldn't the frequent tests for correct functionality and runtime regressions be made in the state which most GAP users will use?

If one would remove the change of the assertion level from these functions one could still easily run all tests with higher assertion level (i.e., change the level and then run all tests) occasionally.

[stevelinton]

I have a very vague memory of this being added, and of someone having a reason, but it was probably last century and I don't recall exactly why it was done or by whom. Possibly to detect broken code IN Assert statements.

Anyway I support Frank's proposal.

[fingolfin]

One reason why one might want to run the tests at a high assertion level is because this way, one may catch more bugs by running the tests -- bugs which might not be easily noticed w/o an assertion catching them.

[fingolfin]

While this would be nice to have, I don't think it's mandatory for 4.9.0, hence I just removed the milestone. Perhaps we can try to get this into 4.10.0.