Think about hierarchies for test items

[davidanthoff]

At the moment the hierarchy for test items is essentially the file system, and then a flat list of @testitem per file.

Is that good enough? One really nice thing about this is that it is simple, plus we reuse an existing well established hierarchical structure, the file system, to order things. So right now I'm thinking that we probably don't want anything beyond that, but who knows...

One could also consider something like @testitem "Level 1/Level 2/My test" begin end, and then the hierarchy would be constructed out of these paths of names...

[davidanthoff]

Another idea would be to add something like a @testfolder macro. Things might look like this:

@testfolder "Foo" begin
    @testfolder "Bar" begin
        @testitem "Something begin
        end
    end
end

The rule would presumably be that a @testfolder can only contain other @testfolders and @testitems.

Could also be named @testcollection...

[jondea]

First let me say thank you for developing this package. It's one of those packages where you immediately think: I didn't know I needed this but I really do!

Reusing the file system does have a certain elegance and robustness.

You may have already considered this, but as an alternative could the existing @testset be repurposed for this? It is very natural to refer to a set of items, and the hierarchy in this system would reflect the default test runner.

[davidanthoff]

New idea: we conceptually change things so that each test item has a name/id and a label. The name/id can be hierarchical. If no label is specified, the name automatically becomes the label. The name/id can be specified either as a String or as a normal Julia identifier.

Implications would be this:

@testitem "Foo" begin
end

Here "Foo" is the id/name, and because no label is specified it also is the label.

@testitem Foo label="Something else" begin
end

Name is Foo, but label is "Something else"

@testitem Bar.Foo begin
end

The name is Foo, and the hierarchy will now show the file name, then a folder Bar, and Foo will appear inside the folder `Bar.

@testitem Bar.Foo label="Something else" begin
end

Would be the obvious thing.

And we would of course allow arbitrarily deep nesting:

@testitem Foobar.Bar.Foo begin
end

[lmiq]

Currently I have not felt the need of another level of nesting besides that of the files, and actually having the files as the top labels for the test makes it very easy to visualize where the tests are.

However, I do know that other packages have large files, and the list of tests can become complicated, thus I like the idea of allowing nesting by labeling.

My doubt would be: If the @testitem is not nested, will it be at the top level, or as subitem of the "file"? Having to necessarily nest all testitems with a redundant filename information maybe a downgrade of usability.

ps: Another syntax alternative could be:

@testgroup "Tests1"      # (actually `@testset` sounds nicer, but may cause conflicts).

@testitem "Foo" <: Tests1 begin 
end

That would not allow arbitrary nesting, but neither does our type system :-D.

[davidanthoff]

Ah, I should clarify: my idea is that the file/folder type nesting we have already stays in place, this would just be an optional way to add more nesting below the file system nesting we already have. I completely agree, I really like having the file system structure in place.

So this would also mean that if one has two files, fileA.jl and fileB.jl, and they have the following respective @testitems in them:

In fileA.jl

@testitem Foo.Testitem1 begin
end

In fileB.jl`

@testitem Foo.Testitem2 begin
end

Then the hierarchy would look like this:

fileA.jl
  Foo
    Testitem1
fileB.jl
  Foo
    Testitem2

So the two Foo "folders" wouldn't be the same because they have different parents.

[cmichelenstrofer]

Maybe unrelated, but I think it would be great if the Pkg.test also showed even the current hierarchy. It currently lumps all @testitems as a single testset named Package.

[NHDaly]

+1 for doing the nesting by directory/file structure. this maps to what they do in golang, and that seems to work well enough for them! :)

[NHDaly]

I'm ambivalent on the sub-file nesting, but it does seem like it could be nice, yeah! Either by having yet another grouping macro, or using nested names as you suggested.

[quinnj]

my idea is that the file/folder type nesting we have already stays in place, this would just be an optional way to add more nesting below the file system nesting we already have. I completely agree, I really like having the file system structure in place.

Note that we don't seem to be doing actual nested folder -> nested tests yet; for example, if I have 2 subdirectories in my test directory, each w/ their own @testitems, it shows up like:

i.e. each individual file w/ @testitems is listed as a top-level "set" of tests

Whereas in go (which @NHDaly was referring to), has:

so here we have the std directory, in which we have the bytes directory, and finally the boundary_test.go file has a nested list of individual tests.

As far as I can tell, we need to restructure how we're providing testitems to the vscode request by utilizing the children field to have the appropriate nesting show up.

If I can figure out how to get a dev env setup to hack on the julia extension, I'll see if I can take a crack at making this change.

[davidanthoff]

Note that we don't seem to be doing actual nested folder -> nested tests yet

Yep, I was more describing an aspiration ;) Not yet implemented, and definitely the go style is better as one can then run all tests in a given folder.

I think this folder hierarchy thing could be implemented purely in the VS Code extension, essentially https://github.com/julia-vscode/julia-vscode/blob/d0d8c6b43fc7cf3f96cd41493e7a943829c421e1/src/testing/testFeature.ts#L295-L328 would just need to parse params.uri into the individual folder parts and then construct the correct hierarchy of TestItems. The implementation might be a bit annoying because the publishTestitemsHandler notification is per file, so one then needs to go into the tree of TestItems and only modify parts of it. I'm also not entirely sure whether we handle deletion of files properly (even in the current implementation).

And we might then also need to check https://github.com/julia-vscode/julia-vscode/blob/d0d8c6b43fc7cf3f96cd41493e7a943829c421e1/src/testing/testFeature.ts#L414 to make sure it can deal with TestItems that are folders, but I think that should be fairly straightforward.

[MilesCranmer]

Any progress on this? I wanted to do

@testitem "Set of tests" begin
    f(x) = x
    @testitem "Single part" begin
        import ..f
        f(1)
    end
end

but this didn't seem to work. I think according to https://en.wikipedia.org/wiki/Principle_of_least_astonishment it would be nice to have this since it's possible with testset, and intuitively I think of the @testitem as defining little modules.

[davidanthoff]

So, I don't think we would ever allow nesting of @testitem, they are really the atomic leaf unit that one can execute. If we were to enable some in-file nesting it would presumably use one of the solutions that we discussed above.

But for now I think the low hanging fruit is to just properly implement a tree view for the folder/file hierarchy in the extension, that seems the first step.