Automate BinSkim runs over official builds

[tkapin]

We are required to run BinSkim over the build artifacts of our official builds. This is one of the requirements to complete compliance (ask @marcpopMSFT for details).

The original instructions are available at AzDO Task 998265 - Run SDL code analysis tools and automatically file bugs for identified security issues.

The instructions in this issue were provided by @mmitche and @garath. Please double-check and comment if some parts are incorrect or not clear. Also /cc @GrabYourPitchforks for awareness.

Current state & known facts

• BinSkim runs over binaries rather than over source code
• We are not on OneBranch (no support for OSS) so that we don't get BinSkim enabled by default
• BinSkim is not enabled in our product repos
• Similar source-code checks are run
• The SDL validation in the staging and nightly pipelines doesn’t download the required input binaries
• PoC run on the arcade repo: arcade-official-ci/20230609.5

Automating the process

• We need to alter the SDL validation stages to pull also binaries and unpack them. This happens in Required Validation and can probably be reused.
• Update the config of the SDL runs (e.g., runtime/sdl-tsa-vars.config at main · dotnet/runtime · GitHub) in each repo to run binskim as well
• SDL runs in nightly builds for all product repos are run as Validate-DotNet pipeline
  • The stage yaml that runs this is also the same as the staging pipeline, so if this can be enabled in the nightly validation, it should work for staging as well
  • Rename "Source Code Validation" to be more descriptive ('SDL and Loc Validation', probably?)

Milestones

• Enable BinSkim for the product repos #2661
• Enable binskim in the staging pipeline as part of the SDL runs "Source Code Validation" stage. #2662
• Enable binskim in the nightly pipeline (Validate-DotNet) as part of the SDL runs.

Caveats, to be found yet

• Source code validation runs before signing today. That might need to change so we can pull signed binaries.
• It's not clear how baselining should work. Guardian files bugs, but we need to find out if we need to integrate additional baselining mechanism.

Due date

Should be automated by RC1

[mmitche]

Added additional info.

[mmitche]

Because the nightly validation pipeline runs off the same logic as the staging pipeline, it can be used for testing and dev iteration.

[andriipatsula]

Repo: https://github.com/dotnet/windowsdesktop
Without BinSkim enabled: https://dev.azure.com/dnceng/internal/_build/results?buildId=2212577&view=results
With BinSkim enabled: https://dev.azure.com/dnceng/internal/_build/results?buildId=2212587&view=results

[andriipatsula]

Link https://dev.azure.com/dnceng/internal/_git/dotnet-release/pullrequest/32508

The nightly validation pipeline doesn't have any runs for the razor repo because in the repos-to-validate.txt file that is used to schedule validation runs the name is razor-tooling. Changing it to razor which is the correct name of the repo

[andriipatsula]

• We enabled BinSkim scanning for the product repos as part of SDL scanning in CI pipelines, the Stage-DotNet and Validate-DotNet pipelines. This allows us to detect various security issues and maintain the quality of our product. The list of repositories is described in Enable BinSkim for the product repos · Issue #2661.
• The overall effort took more than expected as we detected and addressed several problems:
  • We identified an issue with the artifacts produced by the dotnet/runtime repository. The presence of incorrect PDBs was causing BinSkim to crash. We reported and communicated the issue to both the Binskim support team and the dotnet/runtime team, and we also developed a workaround on our side.
  • We identified and resolved a bug in an arcade script used to extract NuGet package. This enabled us to ensure that we are scanning all binaries produced by a repository. This issue is described in arcade/extract-artifact-packages.ps1 script does not preserve the full file tree · Issue #2769
  • We identified and resolved the issue of duplicated TSA reports by creating a tool within the arcade Microsoft.DotNet.VersionTools.Cli which trims the version from NuGet packages. Additionally, we developed an infrastructure that allows for the proper version to be executed in CI pipelines.
  • We collaborated with the ASPNet team to address and eliminate the duplication of TSA work items for the dotnet/aspnetcore repository. This ensures that the product owner can resolve and/or suppress the issues. This issue is outlined in Binskim results for ASPNetCore contain duplicated issues · Issue #2817