Cancelled tests not reported as failures

[thomhurst]

See this run: https://github.com/thomhurst/ModularPipelines/actions/runs/12072550368/job/33666771665?pr=787#step:9:3473

Cancelled tests are marked as failures.

I'd argue they shouldn't be counted as failures as that's misleading. They haven't failed per se, they were just terminated.

If the overall test run was cancelled it should fail overall, but not each cancelled test imo.

What do you think?

[Evangelink]

Is this functionally expected to have a test being cancelled? For me, it is an error even if that's the result of a user requesting cancellation.

We cannot say the test is ok as it didn't reach the end of its action.

[thomhurst]

We cannot say the test is ok as it didn't reach the end of its action.

This is similar to #3699 I think.
It's more inconclusive than failed. If the test framework marks then as cancelled, I'd say it's inconclusive. If it didn't cancel, it might have passed, or it might have failed, we don't know.

The overall run should still fail, but it can create noise in the logs or test explorer when trying to find actual failures.

[thomhurst]

My use case is I have exposed a --fail-fast flag in TUnit. So if you've got a large test suite that takes ages to run, there's the option to fail quickly on the first failure. The first failure will cancel the test session's cancellation token, and so all pending tests would then be marked as cancelled. Sifting through the output/test explorer to find that failure is harder if all the cancellations look like failures too.

[Evangelink]

I understand the TE situation, let's put it on the side for a second to focus on CLI experience, what would be your expectations there?

[thomhurst]

Personally I wouldn't count them under the failure counter, and I find the stacktrace clutter that makes real failures harder to find

E.g.

failed (canceled) Has_Not_Errored (0ms)
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at TUnit.Engine.Framework.TUnitTestFramework.ExecuteRequestAsync(ExecuteRequestContext context)
failed Unhandled exception - TaskCanceledException: A task was canceled. (0ms)
  System.Threading.Tasks.TaskCanceledException: A task was canceled.
    at TUnit.Engine.Framework.TUnitTestFramework.ExecuteRequestAsync(ExecuteRequestContext context)

[nohwnd]

Sounds reasonable to me to not report it as error (there is no additional info in the error anyway?) as long as we guarantee that the whole run will finish with error code.