Move the test runner to use TestItems.jl

[thofma]

Closes #328

Some comments:

• Sometimes I had to encapsulat the whole test file in a @testitem. This lead to some indendation changes. Best to look at the PR without whitespace changes: https://github.com/QuantumSavory/QuantumClifford.jl/pull/329/files?diff=unified&w=1.
• TestItems.jl already runs @testitem in their own module making the use of SafeTestSet unnecessary (see https://discourse.julialang.org/t/prerelease-of-new-testing-framework-and-test-run-ui-in-vs-code/86355/32).

[thofma]

The error with the first commit is due to the following:

julia> begin
         tgate = pcT
         state = StabMixture(S"X")

         apply!(state, tgate)
         apply!(state, tgate)
         apply!(state, tgate)
         apply!(state, tgate)
         state.destabweights |> values |> collect
       end
1-element Vector{ComplexF64}:
 0.9999999999999998 + 0.0im

and the fact that the tests in https://github.com/QuantumSavory/QuantumClifford.jl/blob/master/test/test_nonclifford.jl are not run with the current setup on master.

Edit: I change the test to use isapprox in f5dfe72. Now it passes.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.12%. Comparing base (f5b8b47) to head (056442e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #329      +/-   ##
==========================================
- Coverage   82.32%   82.12%   -0.20%     
==========================================
  Files          61       61              
  Lines        4067     4067              
==========================================
- Hits         3348     3340       -8     
- Misses        719      727       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Krastanov]

This is wonderful, thank you! And thanks for noticing the testing bug! I will need a day or so to get back to the office and check how this runs. The reason for the occasional spurious import or ## is to make these easy to run interactively in VSCode, but I imagine with the VSCode integration for TestItems.jl, this would not be necessary anymore.

[thofma]

I just installed vscode and checked that it works. It does! If I were a vscode user, I would change all my packages to use this!

One downside is that the summary is less helpful:

Test Summary: |  Pass  Total     Time
Package       | 36020  36020  9m18.9s
     Testing QuantumClifford tests passed 

But I think this is something that is worked on upstream, see julia-vscode/TestItemRunner.jl#48. Although this is stalling for over a year now.

[Krastanov]

Thanks for taking care of all of this. By any chance, did you have doctest failures inside of the VSCode testrunner (but no doctest failures outside of it)? I seem to be getting such a failure, something to do with how things are sandboxed in VSCode. I left a comment to seemingly related issue julia-vscode/TestItemRunner.jl#49 in the commit I just pushed.

Please message me at [email protected] with your name, email, and link to this PR so I can start processing the bounty (I do them in bulk at the end of each month -- you will get a email by numfocus to fill out a webform with payment details). Thanks for all the help!

[thofma]

Yes, I am seeing this too. There are a few instances of CurrentModule = QuantumClifford.Exper[...] in the documentation markdown source files. If I remove those, then the tests pass. I could open a PR, but not sure if it breaks the documentation in some other way?

[Krastanov]

If you could open that PR, that would be great, thanks! I thought they are necessary for the autodocs, but on actually reading through the corresponding doc files, they seem a bit superfluous... maybe just a remnant of some old documentation.