Question: running tests on .NET 5

[dsyme]

I'm trying to run FSharp.Compiler.Service.Tests on .NET 5, I thought all I'd need is this

    netcoreapp3.1;net5.0

However this fails with the usual kind of weirdness we all expect from MSBuild, with rubbish about .NET Framework

Severity	Code	Description	Project	File	Line	Suppression State
Error	MSB3644	The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks	FSharp.Compiler.Service.Tests	E:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets	1186	

I think this is because our global.json is pinning us to .NET SDK 3.1.302 and this toolchain doesn't know about .NET 5.

  "sdk": {
    "version": "3.1.302",
    "rollForward": "minor"
  },

I'm concerned that we've got no in-repo coverage of running our tests. Also some tests like those for GetProjectOptionsFromScript and anything to do with F# Interactive are fundamentally sensitive to the .NET SDK we are connecting to to find default reference for F# scripts.

This is related to #10575, where I was trying to add net5.0 testing for FSHarp.Compiler.Service.Tests.fsproj

[cartermp]

I'm concerned that we've got no in-repo coverage of running our tests.

Could you elaborate on what that means?

The SDK is sadly pinned until internal build servers update. It always takes them a long time, sadly.

[dsyme]

Could you elaborate on what that means?

I'm concerned that we've got no in-repo coverage of running our tests on .NET 5.0 SDK

[dsyme]

For example, FSharp.Formatting fsdocs is an FCS-based .NET Command line tool that processes scripts. It currently is tied to netcoreapp3.1 and requires an install of the .NET Core App 3.1 Runtime.

Adding a roll-forward to this tool in fsprojects/FSharp.Formatting#621 exposes a failure when running fsdocs on .NET 5.0 SDK/Runtime because of an incompleteness in the list of "system assemblies" in FCS, hopefully fixed by #10575 (need to fully verify).

The underlying problem is that we are not running our FCS or FSI tests with resepect to the .NET 5.0 SDK. I assume FSI tests are done at SDK integration so it's likely fairly solid. But the FCS tests have to be done here.

[cartermp]

Okay, seems reasonable. This will likely have to wait for #10179 (and thus internal VMs). We'll likely just change the test target to net5.0 instead of netcoreapp3.1 and other places where we explicitly target .NET Core

[baronfel]

It is feasible to add a 'simple' test run that doesn't use the MS infraastructure and uses public SDKs in a parallel way? I'm talking something like doing a simple dotnet test on the FCS service tests and getting a bellwether for the state of things. Because it seems like being tied to MS testing/CI infrastructure, while beneficial in many ways, is currently a negative for FCS-the-library.

[dsyme]

Yes, I was wondering that. If FCS were outside this repo we could test on .NET 5.0 no problem today. Or we could add a GitHub action to do it.

That would however add another CI gate.

[cartermp]

Yeah, it's feasible to add another run (like the dotnet build smoke test) that doesn't use internal stuff. @brettfo would be able to say otherwise, though

[brettfo]

We can't have a run that uses .NET 5, even unofficially, because .NET 5 requires MSBuild 16.8 which only comes with VS 16.8 which hasn't yet been deployed to all internal VMS.

I just received, however, an internal email that the private VMS are being updated tomorrow morning, Pacific time, so as soon as I've confirmed that internal builds pass, I'll be able to merge #10179.

[cartermp]

Now that we build the repo with the .NET 5 SDK, we can also do testing against .NET 5.