dotnet publish fails with 'Metadata generation failed' sometimes

[chuchuva]

Repro steps

Provide the steps required to reproduce the problem:

dotnet publish C:\temp\OurFunctionApp.csproj -c Release -o C:\temp\output

Expected behavior

It should build without an error.

Actual behavior

OurFunctionApp -> C:\temp\bin\Release\netcoreapp2.1\bin\OurFunctionApp.dll
C:\Users\build-admin\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(20,5): error : Metadata generation failed. [C:\temp\OurFunctionApp.csproj]

Output of dotnet --version: 2.1.503

This error occurs sporadically on our Jenkins server. I can't reproduce it on my computer. Also, I cannot reproduce it if run this command on the Jenkins server from command line prompt.

Can you help?

[ishepherd]

@paulbatum or colleagues - How should we investigate this error?

[ishepherd]

Also asked in Stack Overflow: https://stackoverflow.com/questions/54660616/dotnet-publish-fails-with-metadata-generation-failed-sometimes

[paulbatum]

Maybe if you amend your command to include verbose output, there will be a clue of what the error is? See the -v option documented here.

[ishepherd]

Hi @paulbatum - It came back today and I captured logs with -v diag
I get 50 MB of junk but it doesn't look like any of it is coming from the Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.dll.

I notice that most of the extensions.json files do not get created - That's an expected output of this task, I think?

---

Anonymized snippet - Let me know if you want the whole thing privately

04:53:18.465   1:7>Target "_GenerateFunctionsExtensionsMetadataPostBuild: (TargetId:501)" in file "C:\Users\build-admin\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets" from project "<FULL-CSPROJ-PATH>" (target "_PublishBuildAlternative" depends on it):
                   Using "GenerateFunctionsExtensionsMetadata" task from assembly "C:\Users\build-admin\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\..\tools\netstandard2.0\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.dll".
                   Task "GenerateFunctionsExtensionsMetadata" (TaskId:285)
                     Task Parameter:SourcePath=<PROJECT-ROOT-DIR>\bin\Release\netcoreapp2.1\bin (TaskId:285)
                     Task Parameter:OutputPath=<PROJECT-ROOT-DIR>\bin\Release\netcoreapp2.1\bin (TaskId:285)
04:53:18.506   1:7>C:\Users\build-admin\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(20,5): error : Metadata generation failed. [<FULL-CSPROJ-PATH>]
                   Done executing task "GenerateFunctionsExtensionsMetadata" -- FAILED. (TaskId:285)
04:53:18.506   1:7>Done building target "_GenerateFunctionsExtensionsMetadataPostBuild" in project "<CSPROJ-NAME>" -- FAILED.: (TargetId:501)
04:53:18.506   1:7>Done Building Project "<FULL-CSPROJ-PATH>" (Publish target(s)) -- FAILED.

[jonathanantoine]

Same for me I have 3 agents on the same Virtual Machine and sometimes one of them produces this kind of error.

[brettsam]

@ishepherd -- could you try again with our latest sdk package? https://www.nuget.org/packages/Microsoft.NET.Sdk.Functions/1.0.26. Do you get the same thing?

[paulbatum]

@jonathanantoine please let us know if you still see this with version 1.0.26 that Brett mentioned above

[ishepherd]

@ishepherd -- could you try again with our latest sdk package? https://www.nuget.org/packages/Microsoft.NET.Sdk.Functions/1.0.26. Do you get the same thing?

Hi @brettsam @paulbatum, we upgraded a week ago and fine all week - but just now the problem reoccurred and all our builds are broken again.

The snippet I posted (#4055 (comment)) is the same, except:

• microsoft.azure.webjobs.script.extensionsmetadatagenerator version is now 1.0.2 instead of 1.0.1
• (TaskIds/TargetIds are different)

[jonathanantoine]

Hello,

We switched to VS hosted agent and it's working fine now. I don't know if the latest package you mention wouod have fix the issue.

[waveparticlepixel]

We're having the same issue on a VM with several Build Agents installed. Event with the latest version of the SDK.

[HansDahle]

We're also experiencing this error intermittently on a server running multiple agents. Consireded if the isse was both agents using the same nuget cache, causing some lock issue etc, but even disabling all but one agent still resultet in issues.

Workaround for us:
After switching to MSBuild task instead of dotnet build, the problem seems to have gone away.

[brettsam]

Thanks for the reports -- we're trying to figure out where this is coming from but no one has been able to capture it for us yet.

For those reporting it: Is it reproducible once you see it? Can you hop on the agent and run your build command manually and get the same error?

If so, I've got a few things that could help us narrow it down. What's ultimately happening is that our build task calls into a console application to generate the extensions.json file. It's failing, but with no output (or at least none that we've captured). If this is reproducible, you should be able to hop on and run the same command from the command prompt and see if that fails. You'll need three things:

• The path to the console app. This lives in your nuget cache and should be something like: C:\Users\{user}\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.0.2\tools\netstandard2.0\generator\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.Console.dll
• The SourcePath. You can find that in the build logs. For example in @ishepherd's example above you can see the SourcePath of the failed task.
• The OutputPath. This is also in the logs, but you need to add \extensions.json to the path you see there.

Once you have those, you can open a command prompt and run: dotnet [path-to-console-dll] [source-path] [output-path-with-extensions.json].

I'm also trying to beef up our logging with #4314 to see if that helps us diagnose this.

[brettsam]

Note that I've just pushed a new version of Microsoft.NET.Sdk.Functions -- 1.0.27 -- that contains some better logging in an effort to catch the underlying error. If possible, it'd be great if anyone experiencing this could move to this version so we can get some better output (hopefully!).

[ksstott]

Note that I've just pushed a new version of Microsoft.NET.Sdk.Functions -- 1.0.27 -- that contains some better logging in an effort to catch the underlying error. If possible, it'd be great if anyone experiencing this could move to this version so we can get some better output (hopefully!).

I've updated my project to version 1.0.27 and I get the following error:
C:\Users\user\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.0\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(33,5): error : Metadata generation failed. Exit code: '-1073741571' Output: '' Error: 'Process is terminating due to StackOverflowException.' [<csprojpath>]

Once you have those, you can open a command prompt and run: dotnet [path-to-console-dll] [source-path] [output-path-with-extensions.json].

I also tried this, and assuming I passed the correct parameters I am getting no output whatsoever

[brettsam]

@ksstott thank you! Does this happen every time you build, or is it sporadic?

[ksstott]

@ksstott thank you! Does this happen every time you build, or is it sporadic?

It is happening all the time for me running locally (running on windows) but not happening when we are building in a docker container using the microsoft/dotnet:2.2-sdk docker image

[pgussow]

For me this is the error it is giving:
Error processing D:\projects\FunctionApp\bin\Debug\netstandard2.0\bin\Microsoft.CSharp.dll for extension metadata: FileLoadException Could not load file or assembly 'Microsoft.CSharp, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

My scenario is a bit different though: I had a (working) solution. Then I had to migrate to a different repo, so I copied the projects into a new solution. And all of a sudden the extensions.json does not get generated anymore. On file level (within the solution) the files are equal. So I have no idea where this is coming from. The only difference I see is that they both use a different global Nuget cache folder...

[brettsam]

@ksstott -- I'm going to try to think through if there's any diagnostics we can try to gather that would help us narrow this down while you've got a busted setup.

@pgussow -- do you see any error messages in your build that starts with "Metadata generation failed"?

[pgussow]

Unfortunately no. It was not generating any extensions.json file anymore. So based on other Google-results I added the package Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator explicitly to my FunctionApp project.

I'm expecting 2 extensions there:

• Service Bus Trigger
• AutofacOnFunctions (lib I'm using for DI).

Maybe this gives a bit of a clue. Based on the ouytput might it be the sequencing of the targets? As the metadata is called before the function.json is generated?

build_output.txt

[brettsam]

@pgussow -- I've forked a new issue here for us to dig into what you're seeing: #4383

[pksorensen]

@brettsam

I copied over the new dll and could move past a problem:

here is the output (assuming thats fine, no functions stuff should be in the view dll):
1>C:\Users\pksor.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.0\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(20,5): warning : Could not evaluate 'IO-Board.Portal.Views.dll' for extension metadata. If this assembly contains a Functions extension, ensure that all dependent assemblies exist in 'C:\dev\io-board\io-board\functions\IO-Board.Portal.FunctionHost\bin\Debug\netcoreapp2.2\bin'. If this assembly does not contain any Functions extensions, this message can be ignored. Exception message: Could not load file or assembly 'Microsoft.AspNetCore.Mvc.RazorPages, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

[pksorensen]

In my case, I was working on razor pages and had to include these two packages to my function project also (when referencing an aspnet core project from the function project).

    <PackageReference Include="Microsoft.AspNetCore.Mvc.RazorPages" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Razor.Runtime" Version="2.2.0" />

[thecontrarycat]

@brettsam That updated console.dll you linked made the difference for me. For reference, I was using the Jil serialiser in an assembly referenced by my functions project, and it in turn references Sigil which is a netstandard1.5 assembly that does all sorts of fun Reflection.Emit things.

The message I get now is this:

Could not evaluate 'Sigil.dll' for extension metadata. 
If this assembly contains a Functions extension, ensure that all dependent assemblies exist in 
    '<snipped>\bin\Debug\netcoreapp2.1\bin'. 
If this assembly does not contain any Functions extensions, this message can be ignored. 
Exception message: 
    Could not load file or assembly 'System.Runtime.InteropServices.PInvoke, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. 
The system cannot find the file specified.

[brettsam]

Thanks for the confirmation @thecontrarycat! Did you see that message as a build warning? I've changed it to be a Message (rather than Warning) so that you shouldn't get constant warnings about an assembly that likely won't have any Functions extensiosn in it.

[brettsam]

https://www.nuget.org/packages/Microsoft.NET.Sdk.Functions/1.0.28 has been released, which contains this fix. Please give it a try and let me know if you continue to have issues with your builds.

Huge thanks to everyone that helped us debug this. The latest release has more diagnostic logging so it may be easier to detect these kinds of things in the future.

[thierry-prost]

The 1.0.28 update has broken my build with a new 'Metadata generation failed' (when running both locally and in VSTS). I simply reverted from 1.0.* to 1.0.27 because the older package still works fine.

However, I thought the bug report might be relevent here. The fix might be easy as you'll notice the "binbin" extension and I suspect it has to do with the "_FunctionsExtensionsTasksDir", but this is where my expertise ends.

Thanks for the great SDK

[...].nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.1\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(33,5): error : Metadata generation failed. Exit code: '-532462766' Error: 'Error generating extension metadata: System.IO.DirectoryNotFoundException: The path '[...]\bin\Debug\netcoreapp2.1\binbin' does not exist. Unable to generate Azure Functions extensions metadata file.   at ExtensionsMetadataGenerator.ExtensionsMetadataGenerator.Generate(String sourcePath, String outputPath, ConsoleLogger logger) in C:\azure-webjobs-sdk-script\tools\ExtensionsMetadataGenerator\src\ExtensionsMetadataGenerator.Console\ExtensionsMetadataGenerator.cs:line 23   at ExtensionsMetadataGenerator.Console.Program.Main(String[] args) in C:\azure-webjobs-sdk-script\tools\ExtensionsMetadataGenerator\src\ExtensionsMetadataGenerator.Console\Program.cs:line 32Unhandled Exception: System.IO.DirectoryNotFoundException: The path '[...]\bin\Debug\netcoreapp2.1\binbin' does not exist. Unable to generate Azure Functions extensions metadata file.   at ExtensionsMetadataGenerator.ExtensionsMetadataGenerator.Generate(String sourcePath, String outputPath, ConsoleLogger logger) in C:\azure-webjobs-sdk-script\tools\ExtensionsMetadataGenerator\src\ExtensionsMetadataGenerator.Console\ExtensionsMetadataGenerator.cs:line 23   at ExtensionsMetadataGenerator.Console.Program.Main(String[] args) in C:\azure-webjobs-sdk-script\tools\ExtensionsMetadataGenerator\src\ExtensionsMetadataGenerator.Console\Program.cs:line 37```

[brettsam]

Thanks for the report @QuantechTP! Is there any chance that you could kick off a build with "/bl" as your command line and email me the msbuild.binlog file that it generates? My microsoft email is in my github profile.

[pksorensen]

Is there a solution to the following problem: https://twitter.com/pksorensen/status/1131080645696983040

How do i get to build when one of my other dependencies need json >=12.0.2?

3>DotNETDevOps.FrontDoor.RouterFunction -> C:\dev\DotNETDevOps.FrontDoor\functions\DotNETDevOps.FrontDoor.RouterFunction\bin\Debug\netcoreapp2.2\bin\DotNETDevOps.FrontDoor.RouterFunction.dll
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error : System.IO.FileLoadException: Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error : File name: 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' ---> System.IO.FileLoadException: Could not load file or assembly 'Newtonsoft.Json, Version=12.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'.
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Runtime.Loader.AssemblyLoadContext.LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, String ilPath, String niPath, ObjectHandleOnStack retAssembly)
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Reflection.Assembly.LoadFrom(String assemblyFile)
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args)
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.AppDomain.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name)
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Reflection.RuntimeAssembly.GetExportedTypes(RuntimeAssembly assembly, ObjectHandleOnStack retTypes)
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at System.Reflection.RuntimeAssembly.GetExportedTypes()
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at MakeFunctionJson.FunctionJsonConverter.TryGenerateFunctionJsons()
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :    at MakeFunctionJson.FunctionJsonConverter.TryRun()
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error : 
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error : 
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error : Error generating functions metadata
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error :
3>C:\Users\pksor\.nuget\packages\microsoft.net.sdk.functions\1.0.28\build\netstandard1.0\Microsoft.NET.Sdk.Functions.Build.targets(41,5): error : Metadata generation failed.
3>Done building project "DotNETDevOps.FrontDoor.RouterFunction.csproj" -- FAILED.

[mioconnor]

@QuantechTP @brettsam Did you manage to get to the bottom of the /binbin path issue?

[brettsam]

I can't remember -- there's been a bunch of fixes here :-)

If you're seeing a problem with Microsoft.Net.Sdk.functions 1.0.29, can you open a new issue with the description and I can take it from there? Feel free to mention me (and link it in a comment here) and I can take a look. If you're able to run the build with "/bl" and email me (email is in my github profile) the msbuild.binlog file that it generates, that has been a huge help in investigating these.

[drlawler]

I was able to get past the "binbin" issue by editing the
Directory.Build.targets file in the function project.

I changed the line
<_FunctionsExtensionsDir Condition="$(_IsFunctionsSdkBuild) == 'true'">$(_FunctionsExtensionsDir)bin</_FunctionsExtensionsDir>

to remove the "bin"

<_FunctionsExtensionsDir Condition="$(_IsFunctionsSdkBuild) == 'true'">$(_FunctionsExtensionsDir)</_FunctionsExtensionsDir>

it looks like the $(TargetDir) variable already includes the "bin", so having it here also doubles it up.

This was for an existing project. When I created a new project, the entire file is not there.

So then I tried deleting the Directory.Build.targets file, and that also seemed to work.

Hope that helps somebody ;)

[brettsam]

We already perform that tranformation here:

azure-functions-host/tools/ExtensionsMetadataGenerator/src/ExtensionsMetadataGenerator/Targets/Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets

Line 23 in f7f9d4f

<_FunctionsExtensionsDir Condition="$(_IsFunctionsSdkBuild) == 'true'">$(_FunctionsExtensionsDir)bin</_FunctionsExtensionsDir>

Your Directory.Build.targets file may have been there to fix some older issue and it's no longer needed (and in fact, seems like it's breaking things). Thanks for coming back to let us know!

[ThatBlokeCalledJay]

I was able to get past the "binbin" issue by editing the
Directory.Build.targets file in the function project.

I changed the line
<_FunctionsExtensionsDir Condition="$(_IsFunctionsSdkBuild) == 'true'">$(_FunctionsExtensionsDir)bin</_FunctionsExtensionsDir>

to remove the "bin"

<_FunctionsExtensionsDir Condition="$(_IsFunctionsSdkBuild) == 'true'">$(_FunctionsExtensionsDir)</_FunctionsExtensionsDir>

it looks like the $(TargetDir) variable already includes the "bin", so having it here also doubles it up.

This was for an existing project. When I created a new project, the entire file is not there.

So then I tried deleting the Directory.Build.targets file, and that also seemed to work.

Hope that helps somebody ;)

This worked for me (1.0.29)

[25308985]

Upgrade your Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator to 1.1.2 will solve the issue

[pcocko]

I've upgraded Microsoft.NET.Sdk.Functions to last version (1.0.29) and it solved my problem. It also includes Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator 1.1.2 version.

[zuckerthoben]

I have updated to 1.0.29 but ran in this issue again today.
I also have the issue that sometimes the zip / web deploy options are missing when I click on "publish" in the right-click menu of the Azure Function in VS 2019 (16.3.3)

[didaskein]

In order to avoid the same issue, i have created a Windows accounts (Admin) by DevOps Agent.

The goal of that is to have for each account it's own folder for nugets. (No more // issues on the same folder)
C:\Users\Agent1.nuget
C:\Users\Agent2.nuget
...

Since i have change all Windows service with the new windows account i have no more errors in my builds ;) (need to restart the Agent Windows Service)

[didaskein]

In order to avoid the same issue, i have created a Windows accounts (Admin) by DevOps Agent.

The goal of that is to have for each account it's own folder for nugets. (No more // issues on the same folder)
C:\Users\Agent1.nuget
C:\Users\Agent2.nuget
...

Since i have change all Windows service with the new windows account i have no more errors in my builds ;) (need to restart the Agent Windows Service)

[gilmishal]

tried upgrading my functions to v3 with Microsoft.NET.Sdk.Functions 1.30.0-beta2 and I get the "binbin" problem, it uses Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator 1.1.2.

[gilmishal]

I was able to get past the "binbin" issue by editing the
Directory.Build.targets file in the function project.

I changed the line
<_FunctionsExtensionsDir Condition="$(_IsFunctionsSdkBuild) == 'true'">$(_FunctionsExtensionsDir)bin</_FunctionsExtensionsDir>

to remove the "bin"

<_FunctionsExtensionsDir Condition="$(_IsFunctionsSdkBuild) == 'true'">$(_FunctionsExtensionsDir)</_FunctionsExtensionsDir>

it looks like the $(TargetDir) variable already includes the "bin", so having it here also doubles it up.

This was for an existing project. When I created a new project, the entire file is not there.

So then I tried deleting the Directory.Build.targets file, and that also seemed to work.

Hope that helps somebody ;)

This worked for me (1.30.0-beta2)

[brettsam]

I think the usage of Directory.Build.targets was for an old bug. Some people still have that file floating around, but it shouldn't be needed anymore and ends up applying "bin" twice.