Support for .Net Core projects

[CharliePoole]

@galvesribeiro commented on Sat Jan 14 2017

Visual Studio is throwing this:

------ Discover test started ------
NUnit Adapter 3.6.0.0: Test discovery starting
Error: Unable to get runner for this assembly. Check installation, including any extensions.
FileNotFoundException: Could not load file or assembly 'nunit.framework' or one of its dependencies. The system cannot find the file specified.
Dependent Assembly nunit.framework of C:\Users\gutem\documents\visual studio 2017\Projects\ClassLibrary1\ClassLibrary1\bin\Debug\netstandard1.6\ClassLibrary1.dll not found. Can be ignored if not a NUnit project.
NUnit Adapter 3.6.0.0: Test discovery complete
========== Discover test finished: 0 found (0:00:00.655) ==========

This is the sample .csproj:

<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="15.0">
  <PropertyGroup>
    <TargetFramework>netstandard1.6</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="**\*.cs" />
    <EmbeddedResource Include="**\*.resx" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20170113-02" />
    <PackageReference Include="NETStandard.Library" Version="1.6.1" />
    <PackageReference Include="NUnit" Version="3.6.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.6.0" />
  </ItemGroup>
</Project>

With this sample test:

[TestFixture]
    public class Class1
    {
        string teststring = "";

        [SetUp]
        public void SetUp()
        {
            teststring = "AAA";
        }

        [Test]
        public void TestDemo()
        {
            Assert.True(teststring.Equals("AAA"));
        }
    }

Do we need to do anything else?

Thank you.

---

@jnm2 commented on Sat Jan 14 2017

@CharliePoole I wonder if this is #296?

@galvesribeiro Can you see if the NUnit console runner runs the tests properly?

---

@galvesribeiro commented on Sat Jan 14 2017

@jnm2 what should I do to use the console runner on .net standard projects?

---

@JustinRChou commented on Sat Jan 14 2017

Seems like it.
NUnit Adapter 3.6.1.0 has the same issue.

---

@jnm2 commented on Sat Jan 14 2017

@galvesribeiro Download the zip from https://github.com/nunit/nunit-console/releases and run nunit3-console "path\to\test\assembly.dll" at the command line.

---

@jnm2 commented on Sat Jan 14 2017

@JustinRChou can we make these new-style .csproj's part of the VS adapter tests, both .NET Standard and .NET Framework? A single project can multitarget and produce both binaries.

[rprouse]

@galvesribeiro this is a known issue. The NUnit Engine does not currently support .NET Core projects and the dotnet-test-nunit adapter doesn't work with the new .NET Core CSPROJ format. Microsoft decided to drop support for the various dotnet-test-* style adapters in favor of the old style visual studio adapters and we are still playing catchup.

[galvesribeiro]

@rprouse that is strange... xUnit still working just fine since the beginning of .Net Core with DNX. I though when you released 3.6 which is supposed to support .Net Standard 1.6 projects it should work. My mistake...

[rprouse]

@galvesribeiro xUnit has the advantage that the .NET core team is using xUnit, so they were updated first 😄

We released the .NET Standard version of the framework as a first step towards runner support. As a holdover until we are ready, you can make your tests self-executable using NUnitLite.

[galvesribeiro]

@rprouse right. The only problem with that is that I can't use VS to run/debug the tests... Ok, I understood...

Thanks, please close the issue.

[jnm2]

@rprouse Are you planning to support .NET Standard 1.4 or 2.0? The reason I ask is that I have .NET Standard DLLs that I want to test on the .NET Framework, not .NET Core, runtime.

[rprouse]

@jnm2 we will probably replace the PCL project with .NET Standard 1.0 or 1.3 and we may add 2.0 support if needed. Remember, you can always use a lower version of .NET Standard NUnit framework in a higher level project.

[CharliePoole]

Lost the internet for a while or I would have commented sooner...

The 3.6 release of the adapter is not the new one we are working on. It came out a few months ago, before we had any support for .NET Standard. I'm working on finishing the 3.7 release, but that isn't going to solve the problem either.

Runners (like the console, gui or VS adapter) depend on the nunit engine to load and run tests. As soon as we have an engine that supports .NET standard, we'll be able to make the runners support it. Since we release the console together with the engine, that support will be immediate. For the adapter, it won't happen until (a) there is an engine with support and (b) the adapter is updated to use that engine.

We won't close this because this is support we want to provide. But I'm marking it as blocked until there is engine support for it.

[jnm2]

@CharliePoole when you have a project that targets netcoreapp10;net45;net46 it's obvious what to do- run each of the three output binaries against its target TFM. (The reason you might want to test against both 4.5 and 4.6 is quirking.)

However, if one of a project's targets is netstandard14, what runtime is that output binary tested against? .NET Core and .NET Framework are both common candidates and there are other runtimes besides. It would be awesome if I could compile to a single .NET Standard binary and do all my testing off that same binary for every platform (.NET Core, .NET Framework, Xamarin, UWP, whatever).

Will there be a way to specify to NUnit which runtimes to test against, or will it test against all of them that it can?

[CharliePoole]

@jnm2 It's less obvious how to do it. 😄

We are a project that needs to test against multiple runtimes and we do it by replicating test projects. It would be cool to be able to do it in an easier way, but first we have to be able to run the net standard tests at all. IOW, I'm trying to figure out how to get into orbit and you're ready to land on Mars. 😄

NUnit - up to now - makes no decisions on this. It just runs against what you tell it to run against. I don't think that automatically running against multiple runtimes can be an engine feature. It has to be in the runner. The reason I say that is that runners are likely to get pretty confused if they tell the engine to run an assembly and it runs it three times. The runner has to be ready to receive what the engine produces. OTOH, the engine could introduce a setting that allows passing multiple platforms as we now pass in a single platform.

Currently, if you have a .NET 2.0 assembly (to use old technology as an example) you can tell the engine to run it under .NET 4.0. If you don't tell it, it will use 2.0 if available, otherwise the lowest compatible runtime available. Hypothetically, we could pass in an argument something like "net-2.0+net-4.0+net-4.5" and expect it to be run three times. Something similar could be done for portable or net standard tests - it's just that nobody ever thought of it before. Feel free to introduce it as an issue so we don't forget it and also to keep this issue clean.

[galvesribeiro]

I'm not sure if run multiple times against multiple frameworks is something that worth... The principle of netstandard is to support the same API surface between multiple TFMs so, I believe if you target your library to netstandard even if your main facus are net45 framework would be the best solution. Multitargeting on netstandard tooling is supported but is not encouraged since it goes against its purpose.

[galvesribeiro]

But like @CharliePoole said, if you REALLY want to test it that way, make your library to multitarget and create multiple test projects and if they are all have the same set of test code, put it in a shared test project which is referenced on each test project for that specific platform.

[jnm2]

@galvesribeiro The way I really want to test is to produce a single netstandard output and have it run against various versions of .NET Core and .NET Framework. I don't want to multitarget if I don't have to. Then again, there will be scenarios where I need to target netstandard1.4;net40 and I won't have an option.

[jnm2]

@CharliePoole

Feel free to introduce it as an issue so we don't forget it and also to keep this issue clean.

I'm happy to. Not sure where to put the issue since it affects more than one runner.

Let's just take a common case, where I'm forced to target netstandard14;net40. Which runtimes will be used to test the two binaries? What would be ideal for me is for the net40 output binary to be tested against net40 and for the netstandard14 output binary to be tested against the earliest version of .NET Core (1.0) and the earliest version of .NET Framework (4.6.1) that support it. However, someone else may want to also test the netstandard binary against UWP 10 or Xamarin vNext or the latest .NET Core.

[CharliePoole]

@jnm2 I think it has to be done in the engine and console first. It can be added elsewhere after that's done. Each runner would get a separate issue because we are trying to treat each runner as a separate project but there's no point in creating (for example) an adapter issue until there's a feature in the engine that allows the adapter to do this.

Currently the engine runs tests once, under the specified framework or an automatically selected framework. Since the engine doesn't work with multi-targeted assemblies, it can always automatically select a single framework. I think the initial option needed to power all of this is the ability to tell the engine to run tests under multiple frameworks. Everything comes from that.

[jnm2]

I understood you to be saying it would not be done in the engine:

I don't think that automatically running against multiple runtimes can be an engine feature. It has to be in the runner. The reason I say that is that runners are likely to get pretty confused if they tell the engine to run an assembly and it runs it three times.

But since I quietly disagreed with that earlier, I'll just stay out of the way now 😆

[CharliePoole]

If we want the tests to run three times in one Run call, then the engine has to do that. What I wrote earlier was that the engine can't do that automatically because it doesn't know whether the runner calling it can handle the triple output. The engine isn't intended to put demands on runners, but to accept commands from them.

OTOH, in order for a runner to command the engine to do it, we have to implement the capability in the engine. Does that make more sense?

[CharliePoole]

BTW,,, I missed your earlier disagreement... still can't find it.

[jnm2]

That makes perfect sense.

BTW,,, I missed your earlier disagreement... still can't find it.

quietly internally.
Everything looks good. I'll try not to confuse anything from this point forward. 🙄

[rprouse]

Now that VSTest has been open sourced, I have been reading through the documentation and wanted to add some notes to this issue in preparation for working on it. This issue is blocked until we have .NET Core support in the engine, but how we implement that support in the engine may be driven by how we will consume it in this adapter.

For example, AFAIK, this adapter runs tests in-process and does not use the agents. The new dotnet test command line will also use this adapter to run tests at the command line. That will be the documented way to run tests from the command line, so do we need to also add support to the console runner?

I've been looking at how MSTest and xUnit package their adapters now to support .NET Core. It is very similar to other NuGet packages with platform specific versions of the *.TestAdapter.dll in platform directories. Both place their adapters in a Build subdirectory, not a Tools directory. Another interesting thing is the props file which points to platform independent versions of referenced assemblies.

Here is the layout for the MSTest adapter,

Notice that the Microsoft.VisualStudio.TestPlatform.TestFramework.dll is in the _common directory and only platform specific assemblies are in the platform directories. The test framework is .NET standard.

The props files in each platform directory then tell it which assemblies to use for each platform target.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <Content Include="$(MSBuildThisFileDirectory)..\_common\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll">
      <Link>Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>False</Visible>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)..\_common\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.dll">
      <Link>Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.dll</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>False</Visible>
    </Content>
	<Content Include="$(MSBuildThisFileDirectory)..\uap10.0\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll">
      <Link>Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>False</Visible>
    </Content>
  </ItemGroup>
</Project>

I am not sure what the targets files are for, fallback resource files for translations?

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <EnableMSTestV2CopyResources Condition="$(EnableMSTestV2CopyResources) == ''">true</EnableMSTestV2CopyResources>
  </PropertyGroup>

  <Target Name="GetMSTestV2CultureHierarchy">
    <!-- Only traversing 5 levels in the culture hierarchy. This is the maximum lenght for all cultures and should be sufficient to get to a culture name that maps to a resource folder we package. 
    The root culture name for all cultures is invariant whose name is ''(empty) and the parent for invariant culture is invariant itself.(https://msdn.microsoft.com/en-us/library/system.globalization.cultureinfo.parent(v=vs.110).aspx.) 
    So the below code should not break build in any case. -->
    <ItemGroup>
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Name)" />
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Name)"  Condition="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Name) != ''"/>
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Name)"  Condition="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Name) != ''"/>
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Parent.Name)"  Condition="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Parent.Name) != ''"/>
      <CurrentUICultureHierarchy Include="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Parent.Parent.Name)"  Condition="$([System.Globalization.CultureInfo]::CurrentUICulture.Parent.Parent.Parent.Parent.Name) != ''"/>
    </ItemGroup>
  </Target>

  <!-- Copy resources over to $(TargetDir) if this is a localized build. -->
  <Target Name="CopyMSTestV2Resources" BeforeTargets="PrepareForBuild" Condition="$(EnableMSTestV2CopyResources) == 'true'" DependsOnTargets="GetMSTestV2CultureHierarchy">
    <ItemGroup>
      <MSTestV2ResourceFiles Include="$(MSBuildThisFileDirectory)..\_common\%(CurrentUICultureHierarchy.Identity)\*resources.dll">
        <CultureString>%(CurrentUICultureHierarchy.Identity)</CultureString>
      </MSTestV2ResourceFiles>

      <Content Include="@(MSTestV2ResourceFiles)" Condition="@(MSTestV2ResourceFiles) != ''">
        <Link>%(MSTestV2ResourceFiles.CultureString)\%(Filename)%(Extension)</Link>
        <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
        <Visible>False</Visible>
      </Content>
    </ItemGroup>
  </Target>

  <!-- This is required because an empty resource folder is left even though the files within are cleaned up. -->
  <Target Name="CleanupMSTestV2ResourceFolders" AfterTargets="AfterClean" Condition="$(EnableMSTestV2CopyResources) == 'true'" DependsOnTargets="GetMSTestV2CultureHierarchy">
    <ItemGroup>
       <ResourceDirectories Include="$(TargetDir)%(CurrentUICultureHierarchy.Identity)" />
    </ItemGroup>
    <!-- RemoveDir does not throw if the folder does not exist. Continue on error - In any case do not fail build if this task fails(Warn and move on).-->
    <RemoveDir Directories="@(ResourceDirectories)" ContinueOnError="true"/>
  </Target>
</Project>

[rprouse]

Here is the layout of the xUnit adapter,

[galvesribeiro]

@rprouse the new SDK removes a lot of boilerplate code from the .csproj because it automatically import/infer some props/targets from the .Net SDK. Those props are targeting the correct reference for each one of the underlying multitargeted platforms.

I would suggest take that approach for the Engine and any dependency library. The test adapter can be a netstandard library. Don't need to multitarget I guess.

Regarding the console... There are 2 ways to do it AFAIK. (1) Use the test adapter SDK as xUnit guys did and (2) create a Tool which I don't think is well documented yet on .Net SDK. The tool can be used while building the project or as a standalone exe. But I would not be concerned about it. Everyone targeting netstandard (even people on net4xx) will eventually fall in dotnet CLI, so they will end up using dotnet test from now on. If people stay behind not using dotnet CLI, they can just use the old test mode/nugets/runner IMHO.

[duncanawoods]

Whats the current status? Are .Net Core users having to migrate away from NUnit?

[rprouse]

@duncanawoods we have working builds, see #313. As soon as that PR is fully reviewed and merged, I will create an alpha release to NuGet, hopefully this week.

[duncanawoods]

@rprouse great thanks - need a hand with anything?

[rprouse]

@duncanawoods once the alpha is released, I will need help testing it, so using in your projects and reporting issues 😄

[bernardbr]

@rprouse you can count on me!
Is this version will be at nuget.org?

[rprouse]

@bernardbr yes, I will only be releasing the alphas to nuget.org. I will not release early alphas of the VSIX as VSIX does not support .NET Core (a limitation of the VSIX format, not NUnit).

[bernardbr]

Thanks @rprouse!
I'll wait!

[rprouse]

Update for those that are watching this issue, @OsirisTerje has done a fairly extensive code review. There are a few decisions that we need to make, but we are very close to being ready to do an alpha release of this.

[galvesribeiro]

@rprouse Great work! Please notify us here (or gitter) whenever you release something. There are much people depending on it. Thanks!

[Shockolate]

@galvesribeiro Looks like 3.8.0-alpha1 has been pushed to nuget.org

[rprouse]

The alpha has been pushed and the release notes should get most people started. https://github.com/nunit/nunit3-vs-adapter/releases/tag/3.8-alpha1

I plan on writing a more thorough explanation this afternoon or evening and will post a link when that is done.

[bernardbr]

@rprouse
The debug functionality still doesn't work without the beta version of OmniSharp.
Need I do something different to make it work?

Thanks!

[rprouse]

@bernardbr I haven't had a chance to figure out what is wrong with Visual Studio Code without the OmniSharp fixes. Hopefully they will ship it soon so I can forget about the fix on our end 😉

[bernardbr]

Don't worry @rprouse ! Because with the last beta version of omnisharp the debug works fine! 😄
Thanks!

[rprouse]

For those following this issue who wants to start working with the new test adapter, I have written a blog post covering it's usage. Once everything settles down, I will migrate the core info into the docs, but for now, instructions are at http://www.alteridem.net/2017/05/04/test-net-core-nunit-vs2017/

[yaakov-h]

Fairly straightforward, working well so far. Thanks @rprouse et al.

[canton7]

Once everything settles down, I will migrate the core info into the docs, but for now, instructions are at http://www.alteridem.net/2016/10/03/nunit-unit-tests/

@rprouse, did you mean http://www.alteridem.net/2017/05/04/test-net-core-nunit-vs2017/ ?

[ghost]

Hi @rprouse, this new version is perfectly working when you run the tests both from dotnet test command or from Visual Studio! Thanks a lot for that!!

But it seems that debugging a selected test is not working (not sure debugging functionality was already foreseen to work in this alpha release).

Thanks a lot for all your heavy work on this last days!

[bunnu]

I already created an issue for the fatal error: #320

[daveabramson]

When I create NUnit tests in a .NET Core Library, 3.8.0 alpha1 is working for me in Visual Studio Test Explorer. But the tests do not get picked up in the Resharper Unit Test Explorer. When I create NUnit tests in a .NET Framework Library the tests are seen by both. XUnit tests are seen by both. My example code is attached.

ToyWithExamples.zip

[CharliePoole]

IIRC xUnit publish there own interfacing assembly for Resharper. We don't, but rely on Resharper to figure it out.

[daveabramson]

Thanks @CharliePoole , I will go bug them :)

[CharliePoole]

The other option is for us to do as xUnit has done. I'm not sure we have enough people to add that to our list of projects, but maybe somebody will read this and volunteer. 😄