Implement reversing the dependencies for tests #491
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This works for the
--reset
option, but after trying it out I see severalproblems with this approach: we should probably think a bit more about it.
Dependencies for tests are now defined from the outside. This means that the
result of a single test may become inconsistent if included from different
places.
This makes it pretty hard to run a single test (one idea is to allow mutual
includes, and silently ignore loops; but that may cause a risk for
inconsistenncies and would need support from Catala itself)
Another issue comes from the testing engine:
clerk runtest
has a two modes:original file with an external tool)
--reset
, the source file is updated with the new output of theinline tests
Now the tests may stretch across different files. The
--reset
case ishandled and will update all included files. This works but might be a problem
because the targets of the build rule become hard to trace. For the first case,
on the other hand, the interface no longer makes sense with multiple files --
the only consistent option would be not to run included tests there.
With all this in mind, we should either rethink the testing scheme, or find a
different way to provide the end-user feature planned in #485.
Some ideas to work around this:
and would make
pdf
generation possible without affecting the current test runprocess.
--include bundle.catala_fr
flag with specialhandling for avoiding re-including themselves
for each test included in
bundle.catala_fr
. It might be possible toimplement some safeguards against 1. above.
Not sure if the 2nd idea is worth the trouble, but it might actually be
interesting if combined with modules: the bundle compiled to a module would
naturally exclude any inline tests, and the tests would just need the flag to
link that module upon execution. If we go that route, this PR can probably be
discarded but the code for detecting inclusions (which is the main part) can be
kept (and moved to Clerk proper instead of the specific
clerk runtest
command)