[BuildCheck] E2E testing plan

[f-alizada]

Context

The functionality provided by the BuildCheck module is increasing, hence as part of the #10316 we would like to stabilize the behaviour by running the tests (manual) and fixing bugs if appeared during the effort.

Goal

• Validate that documentation of BuildCheck is aligned with actual implementation.
• Validate that MSBuild with enabled BuildCheck runs successfully on Dotnet Repositories ( MSBuild including )

Part 1

Steps:

1. Prepare repro project and validate/confirm that the documentation and actual implementation (main branch) are aligned
2. In case of finding a bug: Check if it is not reported already if not then please report it :) with the label "Area: BuildCheck"
3. In case something is not clear in Documentation that is also an issue, please report it.

Categories/Scope/How to s

• Inbox (built-in) analyzers that run during the build execution.

  • How to:
    • BC0101 - shared-output-path ( Person to contact: jankrivanek)
    • BC0102 - double-writes ( Person to contact: jankrivanek)
    • BC0103 - used-environment-variable ( Person to contact: YuliiaKovalova)
  • Scope: Reproduce the BuildCheck errors on different OS: windows, unix, mac.

• Custom Analyzers

  • Person to contact: YuliiaKovalova
  • How to:
    • https://aka.ms/BuildCheck#custom-analyzers-authoring
  • Scope: Prepare simple build check and integrate it into the solution.

• Configuration

  • Person to contact: f-alizada
  • How to:
    • https://aka.ms/BuildCheck#configuration
  • Scope:
    • Enablment of the rules is configurble and its severity
    • Specification of .editorconfig will be observed

• Tracing

  • Person to contact: maridematte
  • How to: -v:d and times should be at the end of logs
  • Scope:
    • Opt-in reporting of time spent via specific analyzers and infra overall.

• Replay mode

  • Person to contact: surayya-MS
  • How to: https://aka.ms/BuildCheck#binlog-replay-mode
  • Scope: https://aka.ms/BuildCheck#binlog-replay-mode

Part 2

This part contains only of running the MSBuild on existing project with BuildCheck enabled.
Steps:

1. Choose repository in dotnet
2. Run the MSBuild with BuildCheck enabled on selected repository