Fake5 / Fake cli : XUnit2 helper should check is project is an xunit one to run the executable

[tebeco]

see : https://github.com/FakeBuild/fake-netcore-sample/blob/08a725d7f40869af6f72d042b0ad7978cc009efc/Console/build.fsx#L38-L46

Description

How should we use the new Xunit2 api

Repro steps

$> mkdir test
$> cd test
$> dotnet new xunit -n Foo.Tests
$> cd ..
$> touch build.fsx

2. in the build.fsx

Target.create "Test" (fun _ ->
  Trace.log " --- Testing the app --- "

  let setXUnitParams (defaultXUnitParams :XUnit2.XUnit2Params) = 
    let xUnitParams =
      { defaultXUnitParams with
          Parallel = XUnit2.ParallelMode.All
          WorkingDir = Some "test\\Foo.Tests"
      }
      
    printfn "%A" xUnitParams
    xUnitParams

  XUnit2.run setXUnitParams [||]
)

3. run it using fake-cli
fake run build.fsx --target Test

Expected behavior

Execute the tests since it is an xunit dotnet default template

Actual behavior

Crash while looking for xunit.console.exe

Starting target 'Test'
NoDescription
 --- Testing the app ---
Starting task 'xUnit2':
{ToolPath =
  "C:\Workspace\GitHub\TeBeCo\fake-netcore-sample\Console\tools\xunit.runner.console\xunit.console.exe";
 NoAppDomain = false;
 Parallel = NoParallelization;
 MaxThreads = Default;
 HtmlOutputPath = None;
 XmlOutputPath = None;
 XmlV1OutputPath = None;
 NUnitXmlOutputPath = None;
 WorkingDir = Some "test\Poc.Cli.Tests";
 ShadowCopy = true;
 Silent = false;
 TimeOut = 00:05:00;
 ErrorLevel = Error;
 IncludeTraits = [];
 ExcludeTraits = [];
 ForceTeamCity = false;
 ForceAppVeyor = false;
 Wait = false;
 Namespace = None;
 Class = None;
 Method = None;}
C:\Workspace\GitHub\TeBeCo\fake-netcore-sample\Console\tools\xunit.runner.console\xunit.console.exe -parallel none
Finished (Failed) 'xUnit2' in 00:00:00.0816028
Finished (Failed) 'Test' in 00:00:00.0855364

---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
Target     Duration
------     --------
Clean      00:00:00.5258420
Build      00:00:01.7127578
Test       00:00:00.0846504   (Start of process 'C:\Workspace\GitHub\TeBeCo\fake-netcore-sample\Console\tools\xunit.runner.console\xunit.console.exe' failed.)
Publish    00:00:00           (skipped)
Deploy     00:00:00           (skipped)
Total:     00:00:02.3968859
Status:    Failure
---------------------------------------------------------------------
-> BuildFailedException: Target 'Test' failed.
-> One or more errors occurred. (Start of process 'C:\Workspace\GitHub\TeBeCo\fake-netcore-sample\Console\tools\xunit.runner.console\xunit.console.exe' failed.)
-> Start of process 'C:\Workspace\GitHub\TeBeCo\fake-netcore-sample\Console\tools\xunit.runner.console\xunit.console.exe' failed.
-> Win32Exception: The system cannot find the file specified

Known workarounds

Running an exec on dotnet CLI and passing xunit to it like this :

Target.create "Test" (fun _ ->
  Trace.log " --- Testing the app --- "

  let setDotNetOptions (projectDirectory:string) : (DotNet.Options-> DotNet.Options)=
    fun (dotNetOptions:DotNet.Options) -> { dotNetOptions with WorkingDirectory = projectDirectory}

  !!("test/**/*.Tests.csproj")
  |> Seq.toArray
  |> Array.Parallel.map Path.GetDirectoryName
  |> Array.Parallel.map (fun projectDirectory -> DotNet.exec (setDotNetOptions projectDirectory) "xunit" "")
  |> ignore
)

[matthid]

Honestly I have no idea how to solve or detect this. My first feeling is that this needs to be handled by the user and we need to provide a new task for dotnet xunit. In fact we have a similar problem for expecto #1880 and probably lots of others in the future

[tebeco]

something that just run dotnet xunit
for example instead of having to
dotnet.exec “xunit” “”
something like
dotnet.xunit folderPath

the more i this no about it the more i realize i have no good idea in fact for the api or naming ^^

[kblohm]

The console runner just does not work with dotnet core. I had the same issues at work where i had to use both runners because of WPF projects ;). Would a new Task be in the XUnit module or in DotNet or in a new one?

[matthid]

Would a new Task be in the XUnit module or in DotNet or in a new one?

Yes exactly, those kind of questions arise ;)
I'd assume they would be part of XUnit but XUnit would probably depend on DotNet.Cli module

[kblohm]

XUnit is dropping support for dotnet-xunit and will release a global tool. So i do not think it is worth to support this before the global tool is released.

[tebeco]

yes and no
it will still be dotnet xunit to run right ?
that what i would expect for a fake Api and that for either global tool or not ^^

it is still a “dotnet cli” stuff

[kblohm]

Yes, but the global tool will probably support multiple projects and parallelism etc. just like the consoleRunner. You will probably have to provide the path to dlls and not run it from the the project folder and probably more changes. The fake api will likely look quite different for a global tool compared to dotnet-xunit.

[tebeco]

providing dll to a tool is a hell to my point of view ...
i would prefer starts from csproj as this is tests projects as this is easy to do something like

!!(“./test/**/**.Tests.csproj”)
|> Seq.map GetParentFolder
|> DotnetCli.xunit
|> ignoreOrFail

Building each csproj like VsDoes in csproj respective folder is a lot preferred i really hate moving everything to a single folder as you could end up with tests stuff or “fake backend” (for devci) in the output folder

also if someone référence a testProject into another and you got the dll copied you run test twice etc ....

soooo csproj ARE test project ... this is perfect and really simple entry point

[matthid]

I think I'm closing this for now. I feel like the preferred way for "new style" projects is to call dotnet test or dotnet run, which are both supported. Maybe we just need XUnit.buildArgs (just like we added for other modules) then the user can build arguments if he needs to.
PRs which improve things are welcome

[mastoj]

@matthid is it documented anywhere how dotnet test with XUnit is supposed to run.

I'm trying this:

XUnit2.run (fun p -> { p with HtmlOutputPath = Some (testDir @@ "xunit.html") })

But that seems to run mono. I see that it is a toolpath, but that seems to reference to xunit.console.exe and not dotnet test.