[Feature]: dotnet list package --vulnerable could have an non-zero exit code when a vulnerable package is found

[reality77]

NuGet Product(s) Involved

dotnet.exe

The Elevator Pitch

dotnet list package --vulnerable is a great feature we can use in CI workflows.

It would be great to quickly know if a project is vulnerable during the CI processes without having to parse the output of the command.

A quick way to achieve this would be to add an option like --exit-code (just an example) and then the command would return a non-zero exit code if there is at least one vulnerable package.

Additional Context and Details

No response

[DonkeyKongJr]

I would love to see this happen. It would be a great addition to make the CI process smoother.

[trampster]

Just found this issue in our CI, dotnet list package --vulnerable --include-transitive was finding issues but was not failing the build because the exit code was still zero.

This is actually a really important fix, I would imagine others are using vulnerable code without realising it because of this issue.

[trampster]

This is what we ended up doing in our gitlab CI:

        - dotnet restore
        - dotnet list package --vulnerable --include-transitive | tee vulnerable.txt
        - sh -c "! grep 'has the following vulnerable packages' vulnerable.txt"

[bordecal]

This would be helpful for the --deprecated and --outdated options as well.

[erdembayar]

Considering I recently refactored this code path, I would be best person to work on this, so assigned to myself. Looking at Dec or next Jan-Feb sprint timeframe. I already proposed this in my dotnet list package --format json spec.

[tomkerkhove]

Awesome, thank you!

[erdembayar]

@NuGet/nuget-client @JonDouglas @agr @joelverhagen
Quick question? Is there any preceding artwork or doc regarding returning different exit codes depending on output state of dotnet command?
Recently I already refactored it to make it return 1 if there was problem with generating the report.
Currently my intention is let customers get insight without actually having to parse the actual json output on both CI pipeline and local cli. They can do the parsing json output if they get signal of there is something you're interested is wrong.
I'm thinking about return 0b0010 if there is security vulnerability, 0xb0100 if there is there is deprecated package, 0x0b1000 if there is outdated package. This way we can combine different results into 1 byte number. So, customer can check if exit code has interested issue by doing bit masking operation. For example, 0x1110 means it has all 3 issues.

@baronfel mentioned it could be difficult to do bit mask/flag checking in powershell. Probably I need to check how difficult it would be with bash, ms-dos too.
Full offline comment from him:

A flags-style exit code isn't unheard of at all, though there's nothing in the rest of the CLI that does this. I think that's a clever solution given the kinds of data you want to represent.

It's kind of uncommon to do this (its more common to have single, specific error codes), but bash makes this kind of check reasonably easy. Powershell is much more difficult, though. I think powershell users might just end up checking for the specific error code permutations that mean 'outdated' for example

[JonDouglas]

#11549 includes an experience that contains exit codes for auditing.

https://github.com/NuGet/Home/blob/0310d373a1c20240863ccbb0574aae481d45d873/proposed/2021/DotNetAudit.md#dotnet-audit-exit-codes

[JonDouglas]

Some prior art might be https://github.com/dotnet/templating/wiki/Exit-Codes for example

[ricardoboss]

Any progress on this one? Is this still planned to be implemented? If so, when can we expect it?

[linkdotnet]

Slightly related: dotnet list package --deprecated could also return a non-zero exit code. Maybe with a specific toggle: dotnet list package --deprecated --treat-deprecated-as-error.

[nulltoken]

related dotnet/sdk#11510

[GraniLuk]

I would really like to see this feature implemented!

[ay-azara]

Well, as nice as it would be for this to be built-in, I'm no C# developer so the most I can contribute is this Powershell code to read the JSON output, convert it to objects and toss an error.

<#
.SYNOPSIS
    Convert results of `dotnet list package --vulnerable`
    into objects and error at a given severity threshold.

.EXAMPLE
    . ./Resolve-DotnetVulnerability.ps1
    dotnet list $env:SOLUTION package --vulnerable --format json |
        Resolve-DotnetVulnerability -ErrorAction 'Stop'
#>
function Resolve-DotnetVulnerability {
    [CmdletBinding()]
    param(
        [Parameter(Mandatory,ValueFromPipeline,HelpMessage = 'Output from dotnet package list --vulnerable --format json')]
        [string] $Json,

        [Parameter(HelpMessage = 'Severity level which should trigger an exception')]
        [ValidateSet('LOW', 'MODERATE', 'HIGH', 'CRITICAL')]
        [string] $SeverityThreshold = 'HIGH'
    )

    # Had issues piping on Linux without aggregating input
    process {
        [string] $JsonAgg += $Json
    }

    end {

        $VulnerabilityReport = ConvertFrom-Json $JsonAgg
        $ThresholdCount = 0

        class SeverityThresholdExceeded : Exception {
            SeverityThresholdExceeded($Message) : base($Message) {}
        }

        enum VulnSeverity { LOW; MODERATE; HIGH; CRITICAL }

        foreach ($Project in $VulnerabilityReport.Projects) {
        foreach ($Framework in $Project.Frameworks) {
        foreach ($Package in $Framework.TopLevelPackages) {
            $HighestSeverity = $Null

            # Find the highest severity vulnerability for a given package
            foreach ($Vulnerability in $Package.Vulnerabilities) {
                if (-not $HighestSeverity -or [VulnSeverity] $HighestSeverity -lt [VulnSeverity] $Vulnerability.Severity) {
                    $HighestSeverity = [VulnSeverity] $Vulnerability.Severity
                }
            }

            # Track packages that meet or exceed severity threshold
            if ($HighestSeverity -ge $SeverityThreshold) { $ThresholdCount++ }

            $Properties = @(
                @{ Name='Project';          Expression={ $Project.path | Split-Path -Leaf | ForEach-Object { $_.Substring(0, $_.LastIndexOf('.')) } }}
                @{ Name='ProjectPath';      Expression={ $Project.path }}
                @{ Name='Framework';        Expression={ $Framework.Framework }}
                @{ Name='Package';          Expression={ $Package.id }}
                @{ Name='ResolvedVersion';  Expression={ $Package.resolvedVersion }}
                @{ Name='RequestedVersion'; Expression={ $Package.requestedVersion }}
                @{ Name='Vulnerabilities';  Expression={ $Package.Vulnerabilities }}
                @{ Name='HighestSeverity';  Expression={ $HighestSeverity }}
            )
            $Package | Select-Object -Property $Properties
        }}}

        if ($ThresholdCount -gt 0) {
            $Exception = [SeverityThresholdExceeded]::new("$ThresholdCount package(s) contain a vulnerability of severity $SeverityThreshold or greater.")
            Write-Error -Exception $Exception
        }
    }
}

[Joedmin]

Hello, are there any updates without any workarounds?

[ay-azara]

Hello, are there any updates without any workarounds?

Work on this feature appears to be de-prioritized based on this thread. Resources are going towards a new (ish? er? as old?) thing called NugetAudit.

While it is disappointing that we have to switch tooling I will admit putting the following file in the project root has given me the desired result of failing the build by turning vuln warnings generated by Nuget Audit into errors.

Directory.Builds.props

<Project>
  <PropertyGroup>
    <!--
    Treat vulnerability warnings of given severity as errors.
    https://learn.microsoft.com/en-us/nuget/concepts/auditing-packages#warning-codes
    -->
    <WarningsAsErrors>NU1902,NU1903,NU1904,NU1905</WarningsAsErrors>
  </PropertyGroup>
</Project>

[Joedmin]

@ay-azara hmm, I see. Thank you very much for the thread and your workaround. I guess I will stick to it untill the new tooling is released

[nkolev92]

This feature request was created before NuGetAudit was available as a feature.

NuGetAudit was added to the .NET 8.0.100 SDK, and can report packages with known vulnerabilities at restore time for both direct and transitive packages (configurable).
Starting with .NET 9.0.100 SDK, known vulnerabilities are reported for both direct and transitive packages by default.

[ay-azara]

This feature request was created before NuGetAudit was available as a feature.

NuGetAudit was added to the .NET 8.0.100 SDK, and can report packages with known vulnerabilities at restore time for both direct and transitive packages (configurable). Starting with .NET 9.0.100 SDK, known vulnerabilities are reported for both direct and transitive packages by default.

Does NugetAudit support warnings for deprecated/outdated packages? And if not, is there a plan to add that functionality?

[nkolev92]

Please follow #12244 for auditing deprecations at restore time.