Inconsistent coverlet execution results

[oblomingo]

Hey,
I've got different results with the same command line and code version, but using different machines.
The xunit project has all required libraries:

<TargetFramework>net7.0-windows</TargetFramework>
...
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.1" />
<PackageReference Include="coverlet.collector" Version="6.0.0">
  <PrivateAssets>all</PrivateAssets>
  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>

The command line:

dotnet test "test\bin\Debug\App.Test.dll" --collect:"XPlat Code Coverage" --test-adapter-path:. --logger:"junit;LogFilePath=.\test-results\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"

1). The first machine works as expected and generates TestResults\8658657e-d078-4428-8e0d-458dfae204f2\coverage.cobertura.xml
2). The second executes the same command with warnings:

PS C:\GitLab-Runner\...\app> dotnet test "test\bin\Debug\App.Test.dll" --collect:"XPlat Code Coverage" --test-adapter-path:. --logger:"junit;LogFilePath=.\test-results\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"
Microsoft (R) Test Execution Command Line Tool Version 17.8.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Data collection : Unable to find a datacollector with friendly name 'XPlat Code Coverage'.
Data collection : Could not find data collector 'XPlat Code Coverage'

and doesn't generate a file.
Additional information:
First machine:

PS C:\> dotnet --info
.NET SDK:
 Version:   8.0.100-rc.1.23463.5
 Commit:    e7f4de8816

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22621
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.100-rc.1.23463.5\

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.0-rc.1.23419.4
  Architecture: x64
  Commit:       92959931a3
  RID:          win-x64

.NET SDKs installed:
  3.1.426 [C:\Program Files\dotnet\sdk]
  6.0.317 [C:\Program Files\dotnet\sdk]
  7.0.401 [C:\Program Files\dotnet\sdk]
  7.0.402 [C:\Program Files\dotnet\sdk]
  8.0.100-rc.1.23463.5 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0-rc.1.23421.29 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-rc.1.23419.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.23 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.0-rc.1.23420.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Second machine:

PS C:\> dotnet --info
.NET SDK:
 Version:   8.0.100-rc.2.23502.2
 Commit:    0abacfc2b6

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.20348
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.100-rc.2.23502.2\

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.0-rc.2.23479.6
  Architecture: x64
  Commit:       0b25e38ad3

.NET SDKs installed:
  3.1.424 [C:\Program Files\dotnet\sdk]
  7.0.202 [C:\Program Files\dotnet\sdk]
  8.0.100-rc.2.23502.2 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0-rc.2.23480.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-rc.2.23479.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.0-rc.2.23479.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Why the second machine is getting the error? How to fix it?

Off topic question: It seems Gitlab can't use big files to show coverage visualization. Is it possible somehow to split coverage files to smaller parts? Does coverlet has this feature?

[daveMueller]

Are you sure that on this second machine all references can be resolved/restored?

Data collection : Unable to find a datacollector with friendly name 'XPlat Code Coverage'.
Data collection : Could not find data collector 'XPlat Code Coverage'

This looks like the assemblies of coverlet aren't loaded by vstest when the test run starts. Can you find the package anywhere on this second machine?

[oblomingo]

The project uses xunit, therefore the command line has test, not vstest.
Despite the fact that dotnet test has implicit restore, I've executed dotnet restore Solution.sln and I have found coverlet.collector\6.0.0 folder in C:\Users\user_name\.nuget\packages\ folder. I guess the reference can be resolved.

[Bertk]

Hi,

could you please append --diag:".\artifacts\log\test.diag.log;tracelevel=verbose to the dotnet test command and provide the log file.

[daveMueller]

The project uses xunit, therefore the command line has test, not vstest. Despite the fact that dotnet test has implicit restore, I've executed dotnet restore Solution.sln and I have found coverlet.collector\6.0.0 folder in C:\Users\user_name\.nuget\packages\ folder. I guess the reference can be resolved.

Yes you are right. The test platform should be able to load our custom collector in this case. @MarcoRossignoli any idea what the issue could be? I never faced an issue where the collector couldn't be loaded.
Like @Bertk already requested, the logs of the test run could have some valuable information for us.

[MarcoRossignoli]

PS C:\GitLab-Runner...\app> dotnet test "test\bin\Debug\App.Test.dll" --collect:"XPlat Code Coverage" --test-adapter-path:. --logger:"junit;LogFilePath=.\test-results{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"

When you run directly the test dll and not the sln or csproj you have to be sure that coverlet collector is in the bin close to the App.Test.dll or you have to add the directory where coverlet adapter resides using the --test-adapter-path

so or you do

dotnet publish 
dotnet test "test\bin\Debug\App.Test.dll" --collect:"XPlat Code Coverage"

and the coverlet adapter will be in the bin thanks to https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.collector/build/netstandard2.0/coverlet.collector.targets#L12
or

 dotnet test "test\bin\Debug\App.Test.dll" --collect:"XPlat Code Coverage" --test-adapter-path:"coverlet collector path" --logger:"junit;LogFilePath=.\test-results\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"

by default dotnet test *.csproj/sln works because we inject the adapter path using msbuild https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.collector/build/netstandard2.0/coverlet.collector.targets#L20 and we point to the nuget restore folder

[github-actions]

This issue is stale because it has been open for 3 months with no activity.