3.2.4: System.Exception: Invalid dotnet list output. Run dotnet restore then build again.

[natan-abolafya]

Describe the bug
Upgrading from 3.2.3 to 3.2.4 breaks a couple of our builds with the following message:
..................csproj : error : Encountered a fatal exception while checking for Dependencies in .......csproj. Exception: System.Exception: Invalid dotnet list output. Run dotnet restore then build again.

.NET Version: 8.0.202
OS:

• Android build on Ubuntu 22:04 docker image
• macOS build on macOS 13.5.1

[digitalcoyote]

I've seen that a few times since early in 3.x. typically rerunning the build success, but I'm not sure yet why the dotnet list is failing when we already run a dotnet restore in our CI beforehand.

If this is easily reproducable I'll try to get some logging together to see what it's getting from dotnet list that's problematic.

[natan-abolafya]

I tried rebuilding first, the result was the same. Note that our linux builds were fine. Can't test Windows due to #95 :). I'll be happy to collect more logs if you tell me how.

[digitalcoyote]

I'll put some logging together tonight. We have only run into this on Windows so that's at least some helpful info.

There's a unit test I use to make sure past issues with dotnet list parsing don't reocur. If it's not intermittent, you could try running one with the output from dotnet list --include-transitive

[natan-abolafya]

On first attempt, it failed with this error:

No assets file was found for .......csproj. Please run restore before running this command.

Then I added a dotnet restore command first. Now the build passes. Should that be required? I thought dotnet build would take care of it. Nevermind, the branch didn't have 3.2.4. Will update this when that's in place.
Meanwhile, this is the output:

Project 'Client' has the following package references
   [net8.0-macos14.2]: 
   Top-level Package                    Requested               Resolved             
   > INIFileParserDotNetCore            2.5.2                   2.5.2                
   > Microsoft.NET.ILLink.Tasks   (A)   [8.0.3, )               8.0.3                
   > Mono.Unix                          7.1.0-final.1.21458.1   7.1.0-final.1.21458.1
   > NuGetDefense                       3.2.3                   3.2.3                
   > Pkcs11Interop                      5.1.2                   5.1.2                

   Transitive Package                                Resolved
   > MessagePack                                     2.5.108 
   > MessagePack.Annotations                         2.5.108 
   > Microsoft.Bcl.AsyncInterfaces                   7.0.0   
   > Microsoft.NET.StringTools                       17.4.0  
   > Microsoft.NETCore.Platforms                     5.0.0   
   > Microsoft.VisualStudio.Threading                17.7.35 
   > Microsoft.VisualStudio.Threading.Analyzers      17.7.35 
   > Microsoft.VisualStudio.Validation               17.6.11 
   > Microsoft.Win32.Registry                        5.0.0   
   > Nerdbank.Streams                                2.10.69 
   > Newtonsoft.Json                                 13.0.1  
   > StreamJsonRpc                                   2.17.11 
   > System.Collections.Immutable                    7.0.0   
   > System.Diagnostics.DiagnosticSource             7.0.2   
   > System.IO.Pipelines                             7.0.0   
   > System.Memory                                   4.5.5   
   > System.Runtime.CompilerServices.Unsafe          6.0.0   
   > System.Security.AccessControl                   5.0.0   
   > System.Security.Principal.Windows               5.0.0   
   > System.Text.Encodings.Web                       8.0.0   
   > System.Text.Json                                8.0.3   
   > System.Threading.Tasks.Dataflow                 7.0.0   
   > System.Threading.Tasks.Extensions               4.5.4    

(A) : Auto-referenced package.

[natan-abolafya]

yeah, failed after I bumped to 3.2.4. dotnet list output looks the same.

[digitalcoyote]

Version 3.2.4.1-issue180 should log out the output of dotnet list that NuGetDefense is trying to parse. If it's not more than 3 lines it's assumed that it failed or had no packages returned.

Give that a try and share that part of the build log if you can. If you can't, details about the first 3-4 lines may be enough to figure out what's happening.

[natan-abolafya]

this is all I could find:

   ...../AndroidClient.csproj : error : Encountered a fatal exception while checking for Dependencies in ...../AndroidClient.csproj. Exception: System.Exception: Invalid dotnet list output. Run `dotnet restore` then build again.
         at NuGetDefense.Core.NuGetFile.ParseListPackages(String dotnetListOutput)
         at NuGetDefense.Core.NuGetFile.dotnetListPackages(String projectFile, String targetFramework, Dictionary`2& projectectReferencePackages)
        at NuGetDefense.Core.NuGetFile.LoadPackages(String targetFramework, Boolean checkTransitiveDependencies, Boolean checkReferencedProjects)
         at NuGetDefense.Scanner.ScanVulnerabilities(ScanOptions options)

[digitalcoyote]

That makes me think it didn't get any output from the command. Interesting... I'll dig more into this after work today. I may need to rework how it's running dotnet list and getting the output.

[digitalcoyote]

If you're ok with ignoring transitive dependencies I can put a build up that ignores that error until I have a better idea of why it's not reading the output of the command properly.

[natan-abolafya]

We could stay at 3.2.3 for now I think. So there is no rush. But thanks.

[digitalcoyote]

I've had a few unexpected issues with trying to parse dotnet list. I'm going to try parsing the project.assets.json file instead and deprecate this method. With any luck that will be a much easier to debug as well.

[digitalcoyote]

I'll have a release soon that I believe will fix this. If not I'll reopen it.

[digitalcoyote]

Haven't had time to test it in our CI yet. Hoping to get to that tomorrow.

[digitalcoyote]

2nd prerelease out and a stable version with an improvement for the dotnet list parsing that should improve reliabiltiy a little. I apparently forgot about F# and VB.Net for a moment when I coded that.

[natan-abolafya]

that seems to have solved the problem. Thank you!