Generate xml file of test results as specified in .runsettings file

[PriyankaDhamdhere]

I am using Nunit adapter nuget package to integrate Nunit and visual studio. It is not generating TestResults.xml when i run nunit tests from test explorer.

I can do the same when i run it from Nunit3-console.exe.

[CharliePoole]

You are correct. The adapter does not create an XML result file.

If we want it to do so, it would help to have some kind of spec. What should be the name, location and content of such a file. How will the user tell nunit to save the results.

[rprouse]

Yet another reason to create a settings UI for the extension 😄

[CharliePoole]

Have we talked about such a ui elsewhere? I searched but found nothing. For us to have a ui, we would need to stop being just an adapter and replace at least some of the functionality of the test window.

[rprouse]

I'm not sure if it would work for the nuget package, but a VSIX can add a settings node to the Visual Studio options.

[CharliePoole]

Right. But then we become an extension in our own right and not an adapter. Or some kind of hybrid.

It's not impossible but it is a major escalation. I've thought about doing that as a commercial product.

[PriyankaDhamdhere]

@CharliePoole

Test results that I am looking for could be same as the one generated by console runner but more specifically want numbers like passed, failed, run, time took to run tests.

ways that I can think of for doing this –

1.   To import a setting file from VS menu, same like mstest. This could allow us to set if we want to generate XML, its path etc.
   

2.   To have config by default placed at root level of solution, which adapter can identify (something like NunitSetting.config) which will have filepath to generate results.
   

3.   To have an UI for settings in Test Explorer
   

[CharliePoole]

@PriyankaDhamdhere Thanks for the suggestions...

1. NUnit already has a runsettings file, which allows you to set a number of things. However it doesn't allow specifying where the XML should go and it doesn't save to TestResult.xml by default. It's definitely possible to add this. It could either be a single setting, formatted like the console command-line argument, or be divided into multiple settings. This is probably the most direct approach and easiest to understand because it is how we handle other options.
2. I'd rather not proliferate config files - we already have an xml file for lasting settings and the runsettings file for those that pertain to a particular project.
3. See my earlier answer. This would mean expanding the adapter to be a true VS extension. In addition, I think Settings files are more appropriate for genral, cross-project settings, rather than for a single project.

[LirazShay]

Hi, I'm also need this functionality
I'm running SpecFlow tests with NUnit framework
There is a famous extension for BDD, it's called pickles -see http://www.picklesdoc.com/
It has msbuild task that create nice html to show the spec of the system, based of the feature files.
It can combine the nunit test result inside the html if a path was supplied.
It would be great to save each test run from VS in folder called test-results, then that extension will use it on build

[CharliePoole]

@LirazShay User can control which path is used to save output under the console by specifying --work with a directory. I'm not sure if this is ported to the runsettings file, however.

[LirazShay]

It's not required that the exact path will be determined by the settings. Even if it will be saved in default location - it will help me

[OsirisTerje]

@CharliePoole We're setting the WorkDirectory (https://github.com/nunit/docs/wiki/Tips-And-Tricks ) from runsettings to the engine:
package.Settings[PackageSettings.WorkDirectory] = workDir;

Is this the same as the --work parameter to the console? From the notes it says this is just the TestAssembly location, sort of the same as the BinDirectory. Or, does this setting do something else in the engine?

Looking more, we ALWAYS set this parameter to the engine, the runsettings just make it possible to change it. So, there must be another setting to generate the output. I know we discussed this earlier, and you mentioned some way to trigger this in the engine.

(I have never tried this option)

[CharliePoole]

@OsirisTerje Yes, that's exactly right. Our WorkDirectory is the place where output files are intended to be saved for the run, whether created by NUnit or by the user, who can access the work directory using TestContext. It's different from TestDirectory, which is the directory containing the test assembly. For a run with multiple assemblies, there could be multiple TestDirectories, but only one WorkDirectory.

The engine has a facility to create the XML output file, but the runner has to call it. It isn't created automatically. You could do something like this...

var resultService = _engine.Services.GetService<IResultService>();
// Following null argument should work for nunit3 format. Empty array is OK as well.
// If you decide to handle other formats in the runsettings, it needs more work.
var resultWriter = resultService.GetResultWriter("nunit3", null); 
resultWriter.WriteResultFile(yourResult, yourOutputPath);

[OsirisTerje]

Thanks @CharliePoole ! Just what I need! I'll add this in, then at least we have the possibility for an output file implemented.

[LirazShay]

@OsirisTerje
Thanks a lot for taking care of this!!!
@CharliePoole
Many thanks for your help!!

[OsirisTerje]

@LirazShay @PriyankaDhamdhere This should now work. There is a pre-release package at https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter/3.12.0-dev-01009 . Would appreciate if you could test it out. I am OOF from Wednesday evening, so if this is going out, it has to be tomorrow. To use it, add the runsettings file, with another NUnit property named TestOutput. That should point to a relative or absolute folder path where you want the output. I'll document the details later. In any case, there is then one result xml file for each test assembly.

[OsirisTerje]

Version 3.12 release, see https://github.com/nunit/docs/wiki/Adapter-Release-Notes,
Download here: https://www.nuget.org/packages/NUnit3TestAdapter/3.12.0

[LirazShay]

@OsirisTerje
Great work!! thanks!

[LirazShay]

@OsirisTerje
Sorry, I couldn't get this work,
What am I doing wrong?

demo.runsettings.txt

[OsirisTerje]

@LirazShay You seems to have some settings included from the template which is bogus. Can you try with only TestOutput ?

[LirazShay]

Sorry,
It's still not working, when I pass

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
   <NUnit>
	<TestOutput>TestResults.xml</TestOutput>
 </NUnit>
</RunSettings>

The test results is not saved into bin folder
What am I doing wrong? Am I missing something?
I'm using VS2017 Enterprise, NUnit 3 Test Adapter v 3.12, test project with nunit 3.11.0

[OsirisTerje]

@LirazShay First, my apologies and thanks for using this - you "found" a bug, documentation and code is not in sync :-(

The correct settings is TestOutputXml, not TestOutput

(I believe this was a late change, which then of course went sour)

Then , the parameter to the setting is a folder name, it seems you try to set a file name there.
Second, as described here, you must either have an absolute folder location, or, if you use a relative location, it has to be relative to the work directory, or whatever the .net runtime points to. Just specifying a relative folder name there, will normally give an unauthorized exception, because it tries to put it where the .net runtime is running from, like:

Since this is hard to change from the code, because this runs before any test code is executed, I think we need to change this behavious inside the adapter, so that it handles relative paths better, and then relative to the folder where the test is running.

But, if you specify an absolute location you should be clear for now. I'll raise a separate issue to fix the relative path to behave better.

[OsirisTerje]

@LirazShay I have just added a PR to fix the relative path issue (#596 ). I think that is what you need. If you want to try earlier, I have enclosed a pre-release debug version here
NUnit3TestAdapter.3.13.0-a08-dbg.zip

Just unzip and you have the nuget package. Place it somewhere local.

[LirazShay]

@OsirisTerje
Thanks a lot!
This runsettings indeed works perfect:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
   <NUnit>
     <TestOutputXml>C:\TestResults</TestOutputXml>
 </NUnit>
</RunSettings>

It saved the results XML in the specified folder with the name [TestProjectName].xml

About the new version to fix the relative path,
I installed it from nuget console via this cmd
Install-Package "C:\Users\liraz\Downloads\NUnit3TestAdapter.3.13.0-a08-dbg\NUnit3TestAdapter.3.13.0-a08-dbg.nupkg"
It logged: "Successfully installed 'NUnit3TestAdapter 3.13.0-a08-dbg' to DemoTestProjectWithOutput
"

But when I put this row in the settings:

     <TestOutputXml>TestResults</TestOutputXml>

It didn't save the xml in the bin folder of the test project.
What am I doing wrong?

[OsirisTerje]

@LirazShay Are you sure there are not a folder below the bin named TestResults ?

Also, if you look in the Output/Test window, it should state there which folder it will store to. You might have to set <Verbosity>5</Verbosity> into the NUnit runsettings to see the message.

[LirazShay]

I think I have a problem with installing the pre-release debug version
In the Visual Studio Extensions and Updates screen, I see that I'm still using version 3.12.0.0

[OsirisTerje]

@LirazShay It is not a vsix, it is a nuget package so it is added to the solution directly. I have uploaded the vsix version with this comment. Please be aware you should uninstall it afterwards, since it will have the same version number as the upcoming release version - there is unfortunately no such thing as "pre-release" on vsix packages, only on nuget packages. You should try to move your solutions to the nuget adapter, it will also ease the use of them in CI environments.
NUnit3TestAdapter-3.13.0-a08-dbg (2).zip

[LirazShay]

Works well with the vsix (saved relatively to test folder), thanks!
If this is matter to you, I I've got an exception:

It seems to be in
D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs
It happens when I run the tests with the vsix installed

[OsirisTerje]

@LirazShay Yeah, I sent you a debug vsix, was a bit too quick there. I'll drop you a non-debug version later today, but then we know it works.
Thanks for confirming it!

[OsirisTerje]

@LirazShay
Non.debug versions of the vsix and nuget package
NUnit3TestAdapter-3.13.0 (3).zip

Note: You must uninstall the former vsix first, then install this one, since they have the same version number.

[LirazShay]

@OsirisTerje
Problem solved, no exceptions, great work!

[iWheeler5]

@OsirisTerje
Thanks for this update, has greatly helped in my NUnit developments. When will this change be pushed to nuget? Since currently our Automated build pipeline is failing due to looking for the Nuget package on my local machine.

Excellent work :)

[OsirisTerje]

@iWheeler5 Thanks! Good to hear it helps! I am planning on getting 3.13 out this week. I have one more issue I want to get in there, and then it should go.

Just FYI, since that issue and PR is now merged, you can grab it off the pre-release feed.
This should be the latest prerelease: https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter/3.13.0-dev-01042

[naeemakram]

Does this work from Visual Studio 2022 test explorer or do I need to run the NUnit console from the command line?

[OsirisTerje]

@naeemakram It works from VS.

[Suban5]

I am using Nunit 3 Test Adapter v3.17.0. My test.runsettings file is as below

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
	<NUnit>
		<TestOutputXml>C:\TestResults</TestOutputXml>
	</NUnit>
</RunSettings>

while running test case from Microsoft Visual Studio Professional 2019 (v16.11.16) Test explorer, it is generating xml file in C:\TestResults folder like following:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<command-line><![CDATA["C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\Extensions\TestPlatform\Extensions\..//testhost.net472.x86.exe"  --port 63710 --endpoint 127.0.0.1:063710 --role client --parentprocessid 29320 --telemetryoptedin true]]></command-line>

but while running same test case from nunit3-console using following command i am getting different result:
nunit3-console.exe --labels=Before "--out=path to output.txt" "--result= path to output.xml" "path to project.dll" --where "cat = test1"
The xml generated from above command consist of specflow steps.
what am i doing wrong here?
How can i get the same result that is generated from running command using nunit3-console using the .runsettings file?

[OsirisTerje]

@Suban5 Looks like there are no tests there. Enable the dump execution file, and see what you got there. If there is nothing, your tests don't run. Also, try to update to 4.3.0 of the adapter. When I tested here now I got:

[Suban5]

Can you please tell me how to enable the dump execution file?

[OsirisTerje]

Go here: https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html , and https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#dumpxmltestdiscovery-and-dumpxmltestresults specific

Then add a .runsettings file and include the property(ies) there like

<RunSettings>
   <NUnit>
       <DumpXmlTestDiscovery >true</DumpXmlTestDiscovery >
       <DumpXmlTestResults>true</DumpXmlTestResults>
   </NUnit>
</RunSettings>

[Suban5]

I have added DumpXmlTestDiscovery and DumpXmlTestResults in test.runsetings file. It is still generating the same XML file as I have commented above. I can assure you that my test is running as expected.
Following is the part of the output in the visual studio test output

========== Starting test run ==========
NUnit Adapter 3.17.0.0: Test execution started
Running selected tests in C:\Projects\IntegrationTest\bin\Debug\IntegrationTest.dll
  Test Output folder checked/created : C:\TestResults12 
   NUnit3TestExecutor discovered 1 of 1 NUnit test cases
Given I have an empty raw and in and out folder
-> done: StepDefinition.GivenIHaveAnEmptyRawAndInAndOutFolder() (0.0s)
Given I have an empty IN and TW and Opt1 and OUT folder
-> done: StepDefinition.GivenIHaveAnEmptyINAndTWAndOptAndOUTFolder() (0.0s)
Given I have truncated all tables
-> done: StepDefinition.GivenIHaveTruncatedAllTables() (1.3s)
Given I added dummy data in POLS
-> done: StepDefinition.GivenIAddedDummyDataInPOLS() (0.0s)
Given I have input files in directory "\\QATestFeature\\Other\\AddressStructureCode\\Files\\1"
-> done: StepDefinition.GivenIHaveInputFilesInDirectory("\\QATestFeature\\...") (0.0s)
When I execute module: "P001"
-> done: StepDefinition.WhenIExecuteModule("P001") (27.5s)
Then I execute module: "P002"
-> done: StepDefinition.WhenIExecuteModule("P002") (0.0s)

   Test results written to C:\TestResults12\VH.wcPrism.POLSGAIC.IntegrationTest.xml
NUnit Adapter 3.17.0.0: Test execution complete
========== Test run finished: 1 Tests (1 Passed, 0 Failed, 0 Skipped) run in 34.3 sec ==========

I am currently using nunit3TestAdapter from the visual studio marketplace extension instead of using it in Project and it is not showing any option to upgrade from 3.17.0.0 to the latest version.

Can you please tell me why I am not getting the expected test output XML file? Is there any other way to get the test output XML file?
Can you please share the test project that you use to test with me?

[OsirisTerje]

1. The VSIX (from the marketplace) is deprecated. It means it will not be updated beyond 3.17, nor will we do any more fixes on it. Please use the nuget package.
2. You asked for dump settings, those files are in a subfolder named Dump in your binary folder.
3. Install version 4.3.0 and then see what you get. Please uninstall the vsix completely before you try.
4. About a test project, do as following:

In an empty folder:

Run dotnet new nunit

Add your runsettings file to the same folder (assuming it is named .runsettings

Then run

dotnet test  --logger "console;verbosity=normal" -s .runsettings

or use it directly

dotnet test  --logger "console;verbosity=normal" -- NUnit.TestOutputXml=C:\TestResults

[Suban5]

I am getting the following error:

========== Starting test run ==========
Error initializing RunSettings. Default settings will be used
System.ArgumentNullException: Value cannot be null. (Parameter 'path1')
   at System.IO.Path.Combine(String path1, String path2)
   at NUnit.VisualStudio.TestAdapter.AdapterSettings.SetTestOutputFolder() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\AdapterSettings.cs:line 434
   at NUnit.VisualStudio.TestAdapter.AdapterSettings.Load(String settingsXml) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\AdapterSettings.cs:line 345
   at NUnit.VisualStudio.TestAdapter.AdapterSettings.Load(IDiscoveryContext context) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\AdapterSettings.cs:line 311
   at NUnit.VisualStudio.TestAdapter.NUnitTestAdapter.Initialize(IDiscoveryContext context, IMessageLogger messageLogger) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitTestAdapter.cs:line 135
NUnit Adapter 4.3.0.0: Test execution started
Running all tests in C:\D\Test\bin\Debug\net5.0\Test.dll
Exception System.ArgumentException,    Exception thrown executing tests in C:\D\Test\bin\Debug\net5.0\Test.dll
Path cannot be the empty string or all whitespace. (Parameter 'path')
   at System.IO.Directory.CreateDirectory(String path)
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.CreateTestOutputFolder() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 365
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, IGrouping`2 testCases, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 275
NUnit Adapter 4.3.0.0: Test execution complete
========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 114 ms ==========

The following are in my solution:

Following are the test in my UnitTest1.cs

And Following is the content in test1.runsettings file

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
	<NUnit>
		<TestOutputXml>C:\TestResults</TestOutputXml>
	</NUnit>
</RunSettings>

What am I doing wrong? Am I missing something?

[OsirisTerje]

You have to set the OutputXmlFolderMode property to AsSpecified.

In the next version we will make this easier, so that if an absolute path is specified, then this mode is selected automatically., so sorry about this.

[Suban5]

Setting OutputXmlFolderMode property to AsSpecified throws the following error.

But setting OutputXmlFolderMode property to RelativeToResultDirectory fixed my issue.
Thanks you 👍

========== Starting test run ==========
NUnit Adapter 4.3.0.0: Test execution started
Running selected tests in C:\D\Test\bin\Debug\net5.0\Test.dll
Exception System.ArgumentException,    Exception thrown executing tests in C:\D\Test\bin\Debug\net5.0\Test.dll
Path cannot be the empty string or all whitespace. (Parameter 'path')
   at System.IO.Directory.CreateDirectory(String path)
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.CreateTestOutputFolder() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 365
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, IGrouping`2 testCases, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 275
NUnit Adapter 4.3.0.0: Test execution complete
========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 98 ms ==========

Note: I don't have any D drive in my system