Code coverage not working for Globalization* test projects

[ViktorHofer]

• System.Globalization.CalendarsWithConfigSwitch.Tests.csproj
• Invariant.Tests.csproj

Issues:

• Using old msbuild property: UsingCoverageDedicatedRuntime which is now called UseCoverageDedicatedRuntime.
• This property isn't set at evaluation time as it is defined in CodeCoverage.targets which is only available when targets are imported, which happens at the end of a project file.

I'm going to submit a fix for this but the bigger issue is that we currently store all properties in the targets file instead of props files. We'll want to refactor that.

[ViktorHofer]

cc @pjanotti. After fixing this with a temporary workaround and now placing the right runtime config files into the test directory, the coverage is still not generated correctly for System.Globalization* ones. Can you please take a look?

[ViktorHofer]

@pjanotti please let me know if you have an idea why coverage isn't measured for these projects.

[pjanotti]

@ViktorHofer if it is using coverlet it typically it is a matter of not instrumenting the right assemblies. The runtime config file should not affect runs with OpenCover.

[ViktorHofer]

The runtime config file should not affect runs with OpenCover.

I initially thought this has to do with runtime config files. Sorry for the confusion, but that's happening just with the default settings in master.

C:\git\corefx\src\System.Globalization\tests\Invariant>dotnet msbuild /t:RebuildAndTest /p:Coverage=true
Microsoft (R) Build Engine version 15.9.13-preview+g331625c07d for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Invariant.Tests -> C:\git\corefx\bin\AnyOS.AnyCPU.Debug\Invariant.Tests\netcoreapp\Invariant.Tests.dll
  Replacing Portable PDB with Windows Pdb in place!
  Using C:\git\corefx\bin\testhost\netcoreapp-Windows_NT-Debug-x64\ as the test runtime folder.
  Executing in C:\git\corefx\bin\tests\Invariant.Tests\netcoreapp-Windows_NT-Debug-x64\
  ----- start 22:11:31.22 ===============  To repro directly: =====================================================
  pushd C:\git\corefx\bin\tests\Invariant.Tests\netcoreapp-Windows_NT-Debug-x64\
  call C:\git\corefx\packages/OpenCover\4.6.519\tools\OpenCover.Console.exe -oldStyle -filter:+[System.Globalization]* -excludebyfile:*\Common\src\System\SR.* -nodefaultfilters -excludebyattribute:*.ExcludeFromCodeCoverage* -skipautoprops -hideskipped:All -threshold:1 -returntargetcode -register:user -target:C:\git\corefx\bin\testhost\netcoreapp-Windows_NT-Debug-x64\\dotnet.exe -output:C:\git\corefx\bin/tests/coverage\Invariant.Tests.coverage.xml -targetargs:xunit.console.dll Invariant.Tests.dll  -xml testResults.xml -notrait category=nonnetcoreapptests -notrait category=nonwindowstests  -notrait category=OuterLoop -notrait category=failing  -notrait Benchmark=true
  popd
  ===========================================================================================================
  Executing: C:\git\corefx\bin\testhost\netcoreapp-Windows_NT-Debug-x64\dotnet.exe
  xUnit.net Console Runner v99.99.99-dev (64-bit .NET Core 4.6.26918.01)
    Discovering: Invariant.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  Invariant.Tests (found 14 test cases)
    Starting:    Invariant.Tests (parallel test collections = on, max threads = 4)
    Finished:    Invariant.Tests
  === TEST EXECUTION SUMMARY ===
     Invariant.Tests  Total: 289, Errors: 0, Failed: 0, Skipped: 0, Time: 0.532s
  Committing...
  No results, this could be for a number of reasons. The most common reasons are:
      1) missing PDBs for the assemblies that match the filter please review the
      output file and refer to the Usage guide (Usage.rtf) about filters.
      2) the profiler may not be registered correctly, please refer to the Usage
      guide and the -register switch.
  ----- end 22:11:36.13 ----- exit code 0 ----------------------------------------------------------
  Loading report 'C:\git\corefx\bin\tests\coverage\Invariant.Tests.coverage.xml'
   Preprocessing report
   Initiating parser for OpenCover
    Current Assembly: System.Globalization
  Initializing report builders for report types: Html, Badges
  Analyzing 0 classes
   Creating summary
  Report generation took 0.7 seconds

[pjanotti]

@ViktorHofer at least for System.Globalization what is happening is that the actual code is in System.Private.CoreLib so to get coverage for it you need to add /p:CodeCoverageAssemblies="System.Private.CoreLib"

On the CI runs we add CoreLib for all tests but actually I would like to be more explicit about it, i.e.: each test project that the actual code resides in System.Private.CoreLib should be explicit about it.

[pjanotti]

side note: we can't run coverlet against CoreLib (already tracked in a separate issue)

[ViktorHofer]

By introducing a msbuild variable for that or somehow auto detecting it? What's the plan?

[pjanotti]

My initial idea is for test projects to explicitly include a property to indicate that they are testing code in CoreLib. I don't want to automate that since most test code eventually touches CoreLib.

[ViktorHofer]

Sounds good. Please open a PR for the fix :)

[pjanotti]

Eventually :) - perhaps it happens sooner since it may be part of a workaround for the problems with the official coverage run, but I'm not sure yet.

[pjanotti]

Last coverage run removing CoreLib indicates that we should do this rather soon: one test crashed (known issue with OpenCover) but it completed the pass.

[pjanotti]

Forgot to close this one.... now with TestRuntime property this is not an issue anymore