Tree format 'Fixture list' + grouping 'By Category' + categories on test case level => no categories are displayed

[rowo360]

Describe the bug
This issue describes a similar use case like issue #1106. But I expect it's best to keep the discussion and solution separate, so I create a new issue for it.

If I define the test categories on test case level and I use tree format 'Fixture list' and grouping 'By category' the system fails to find the categories:

Expected behavior
I expect that all the categories are found and displayed in the tree regardless on which level they are defined (test fixture or test case). Each category should contain a node for the TestFixture and the appropriate test cases. Here are two screenshots how Visual Studio test explorer and Resharper handle this use case. Although their tree display + grouping is not 100% comparable to TestCentric, it's my expected behavior for this use case.

Environment (please complete the following information):

• TestCentric Version - latest developer build
• OS: Windows 11

[rowo360]

From technical point of view it can be explained easily why these categories are currently not found:
The CategoryGrouping class only inspects the XML nodes for the TestFixture node itself, but it doesn't inspect the XML nodes of any children. However in this case the TestCategory is only present at a XML node of child, so it's currently not found.

I could imagine that this can be solved by extending the XPath expression in the CategoryGrouping class. I didn't look it up: but something like 'descendant-or-self'. So overall the TestCategory in a child node will be found.

But there is more to consider here:
So far the TestFixture is only added to exactly one category. This must be changed, so that it's possible to add to several categories.

[CharliePoole]

This was by design although it may not be what people expect. My logic was as follows:

1. FixtureList is a list of fixtures. The list only contains fixtures, although you can expand the fixtures and see tests.
2. If you group the list by categories, you should still see the same fixtures listed as you did when it was ungrouped

So, if the user wanted to see test cases in a category, I assume they would use the TestList strategy

AFAIK, it's possible to put a fixture into multiple categories and they should appear in multiple groups in that case. If it's not working, I think that's a bug.

[rowo360]

Yes, I can confirm that all groups are already created when I define multiple categories on a fixture. That's fine!

[rowo360]

I have the feeling that the discussion might be similar to our conversation in #1106 . Or at least my expectation and argumentation is similar :-)
In that issue we talked about the 'Test list' view and if test categories defined on Fixture level were not found. I mentioned that I think that users don't distinguish between defining the test category on test fixture or test case level although you explained internally there is definitely a difference. And after we have exchanged ideas we finally decided to consider the test categories on Test Fixture level also in the 'Test list' view.

Now, my argumentation for the 'Fixture list' view is similar:
If I switch to the category grouping, I want to see all my categories so that I can easily execute all tests of one category. Or one step further: I want to execute all tests of one category in a Fixture. In this case too, I believe that the user does not know the subtle differences between categories on the fixture and test case. He just defined some test categories and expect to see them in the tree.
Actually there should be no difference using the 'category' grouping between 'Test list' and 'Fixture list' view besides that in the later case the test cases are grouped by their fixture and not displayed in a plain list. And for example if I switch between 'Test list' and 'Fixture list' view, the test cases in the categories remain identically, they are only grouped differently.
Ok, that's at least my expectation...

[CharliePoole]

If we decide to make the two of them work the same, then do we need both of them? For me, that's the key question. We have about a dozen ways to display tests, if you take all the strategies together with the groupings. I didn't put a lot of thought into which we actually needed, but just took the existing groups from VS a long time ago.

So, dealing just with categories, can we reason about whether and why we need to see both test list by category and a fixture list by category? When would you use one or the other. My thinking when we changed how test list works was that the typical user would not look at the fixture list by category at all. It's only useful for somebody who understands the distinction.

Additionally, I don't think the two cases are symmetrical because of how categories act. A category on a test fixture acts the same as if all the test cases had the category. Therefore, I agree that it makes sense to show the test cases that way. But the reverse doesn't seem to be true, i.e., a category on a single test method is NOT the same as putting the category on the test suite.

Slightly off topic for this issue: NUnit V2 allowed selecting categories in the NUnit Tree, which responded by greying out the un-selected nodes. Is this something to (separately) consider?

[rowo360]

I have just realized that I am actually thinking in a different direction in my arguments and thoughts than our current implementation. Perhaps this is the reason for the confusion?
In our implementation we have a DisplayStrategy for either Fixtures or TestCases. And then we are grouping them in categories, duration or outcome.

But actually I have something different in mind. If I am interested in test categories, I would first of all like to see all these categories for my test cases. The second question is how I want these test cases to be grouped:

• No grouping: that's the test list
• Fixture grouping: that's the fixture list
• Namespace grouping: just to mention something else ;-)

So I don't want a list of fixtures and group them in categories. Instead, I want to have a list of categories and group the associated test cases by their fixtures. It was now quick to write and even quicker to read, but it took me a little longer to notice the difference. :-)

I think the focus is always on the test cases and I can either group them in Fixtures or Namespaces (just to mention something different again) or do no grouping. Thinking about the implementation, I wonder if this means having a CategoryDisplayStrategy and a FixtureListGrouping/NoGrouping/(NamespaceGrouping) instead? I hope that doesn't sound too wild now. But wouldn't this also simplify our discussion about categories on TestCase vs Fixture level? The starting point for grouping is always the test cases. And a test case gets its category either from itself or its fixture (or both).

Now you have already asked whether these different groupings are needed at all. And I think it's a hard question! I myself find it better when I can see a bit of structure in the tree and not just a (long) list of test cases. But if I only want to run all the tests in a category, then the further subdivision within the category is irrelevant. If, on the other hand, I only want to execute the tests of one category within a fixture, then I need the FixtureList grouping. And the same argument can be made with a namespace.
Therefore, there will probably be a use case for all groupings and we can't just throw one out.

I hope I haven't gone too far with these considerations, but it has helped me.
So far I have only described these considerations for the test category, but actually this also applies to 'duration' and 'outcome'. (That's the related discussion in #1135)

• Outcome: I want to a list of all 'passed', 'failed', 'ignored' test cases grouped by fixtures, (namespace) or not grouped.
• Duration: I want a list list of all 'fast', 'slow' test cases grouped by fixtures, (namespace) or not grouped.
  (Having a OutcomeDisplayStrategy/DurationDisplayStrategy and apply the FixtureListGrouping/NoGrouping/(NamespaceGrouping))

Do these considerations make sense to you?

[CharliePoole]

It's interesting that you view the FixtureList as a grouping of tests by categories. I think of it as a list of those tests of type TestFixture. Visual Studio's Test Explorer has made people think that only methods are tests, and we have gone along with that to a certain extent. But in NUnit Tests can contain other tests (we call those TestSuites). In NUnit a simple method is a test and also a test case. A parameterized method is a test (suite) containing test cases. A TestFixture is represented by a class containing both of the preceding kinds of tests. A namespace, ditto. An assembly, ditto.

So when we "group" test cases by Fixture or Assembly, it's not arbitrary but based on an underlying reality. It's similar to "grouping" children by their parents as opposed to grouping people by hair color or, even better, by whether we like them. The latter is arbitrary while the parental grouping is intrinsic. Grouping by Fixture or Assembly is intrinsic. Categories are arbitrary in the sense that we can make up whatever categories are important to us. I think this is so even though categories are expressed in the code. Properties are similar to categories - I think of categories as degenerate properties without a value.

Duration and Outcome are both extrinsic and dynamic. They are more like properties of test runs than of tests. I'm stressing how different in nature I believe these different "groupings" are because I'd like to see us deal with each one of them separately across all three strategies at least in the design stage if not the implementation stage.

That's my view. And yet I think it's OK for a user to think of a Fixture as simply a way to group tests. Also, I think you are suggesting that we start talking about Categories rather than strategies and groupings. So, using categories as an example, we can ask "What are they for?" "How do people need to use them?" "How can that be reflected in our different ways of showing the tree?"

So, proceeding, what do we need to reach a plateau where we can close this issue? It doesn't have to involve solving every possible platform. Then, how can we get organized to design what we want to do about categories overall? Maybe a separate design issue? Similarly, how can we do the same for the other "groupings" one at a time. What do you think?

Off topic... is there still a failing pull request you need me to merge manually?

[rowo360]

It seems that I have also been influenced by the Test Explorers views without such a deep understanding and knowledge of the NUnit internas. However, like you, I immediately understand and see the difference between the intrinsic, extrinic and dynamic properties - yes, I agree, the Duration and Outcome are properties of the test run.

But I'll stick to my view about a test case for a moment, just to comment on my considerations: :-)
My idea was to start with test cases and group them according to some criteria. I noticed a flaw about this idea regarding the Duration grouping now: in our discussion in #1135 we came to the conclusion that it's beneficial to group Durations by Fixture, because the Fixture itself provides also some duration. I guess this won't be possible with my idea right away - but well, nothing is perfect from the start :-)
But you also mentioned an idea about viewing the Duration not as a grouping at all, so that might help.

And sticking with my view about test cases another moment:
I could imagine that there's no separation in DisplayStrategies and Groupings with this approach anymore. But there's just some Grouping functionality which receives a list of test cases and provides a grouped list of test cases as output (for example the output: IDictionary<string, IList<TestNode>> or IList<TestGroup>). And now the clue: this output (the list of test cases of one group) can be passed to the input of a different Grouping. So we can plug groupings in different order by this approach. This will provide some flexiblity.
You could even view the NUnit Tree as first a grouping of namespaces and afterwards a grouping of Fixtures (I don' t want to change the NUnit Tree - it's just an example).
I certainly haven't considered everything, but I would like the flexiblity.

Now something different:
Until now, I have never worked with multiple test assemblies in a TestCentric project. But that's possible now and I have to give it a try. Does this introduce new use cases? At least there are more possibilites:
maybe one user wants to view the outcome of all tests in his complete TestCentric project and another user wants to view the outcome of all his tests grouped by assemblies...
Oh, it seems to me that it won't get any easier :-)