Report "direct" pseudo tested tests differently than "indirect" ones

[vmassol]

Right now pseudo tested tests are reported whether the method that is pseudo tested is the direct method called by the test or if it's an indirect method called by a method called by the test.

In most cases, indirect pseudo tested methods are not a problem. It's just that the test writer didn't want to assert the indirectly-called method.

OTOH if Descartes notices a pseudo tested method that is directly called by the test then you can be sure it's a real problem that needs to be fixed (since it means the test is not testing anything).

I guess the hard part is to find out the real tested methods from the test.

WDYT?

[oscarlvp]

So, you propose to measure the relevance of pseudo-tested methods using their distance to the test cases in the call graph. This is something we have been discussing for a while.
For example, suppose that you have a private method that is pseudo-tested. It is never the intention to directly test this method, nor you can do it. However, if this method is pseudo-tested it means that its behaviors, that is, the functionalities this method implements, are not verified. On the other hand, a log method might have a shorter distance to the tests, even called directly, yet it is not really important.
Also, the same methods can have different distances to different test cases. Remember that the classification of pseudo-tested methods takes into account the entire test suite.
Another scenario, suppose you have methods A and B. A is directly invoked by a test case and A invokes B. Lets suppose that B is pseudo-tested and this makes also A pseudo-tested. The actual finding here is that B is pseudo-tested, A is pseudo-tested only as a consequence, therefore the test improvement should be to better verify the behavior of B, which will produce a better verification for A in turn.

Technically, I don't think it is possible to get this information in the scope that Descartes handles as a plugin for PIT. We do have another tool, Reneri that works with this information, the distance to the tests, to generate the suggestions for improvement. This tool is still a prototype, but we plan to develop it further. This type of analysis is better suited for an external tool, like Reneri, than Descartes.

[vmassol]

Thanks for the reply and the information @oscarlvp . Cool about Reneri.