[BUG] BinaryData.ToString() throws nullreference exception after upgrading Microsoft.Extensions.Azure/Azure.Core

[mlankamp]

Library name and version

Azure.Core 1.44.0

Describe the bug

The dependency System.Memory.Data is upgraded in version 1.44.0 to 6.0.0, this version has very annoying bug. When BinaryData is empty, the .ToString() method throws. This is fixed in a later version of the System.Memory.Data library.

The previous version (1.42.0) used System.Memory.Data version 1.0.2, this version doesn't have the problem.

Expected behavior

BinaryData.ToString() should not throw

Actual behavior

It does :-(

Reproduction Steps

Add manual package dependency to Microsoft.Extensions.Azure 1.7.6
public async Task UpdateBuildingFileStatusActivity(
[BlobInput("path/to/emptyfile.txt", Connection = "MadasterStorage")] BlobClient blob)
{
var content = await blob.DownloadContentAsync();
var text = content.Value.Content.ToString();
}

Environment

No response

[mlankamp]

Change: #46134 (this mentioned the bug)

[jsquire]

@m-redding: Would you please coordinate with the Storage team to make sure a release is going out this week with your fixes?

[m-redding]

Hello @mlankamp the best way to fix this is to add a direct reference in your application on System.Memory.Data 8.0.0 or protect against the bug in the 6.0.0 package with a try/catch. Unfortunately, this is a bug in the System.Memory.Data package so it's not something that can be fixed in the Azure.Storage.Blobs package.

[jsquire]

After internal discussion, it's clear there was an error in my initial analysis. My understanding that the exception was internal to the Storage libraries was incorrect and, as Maddy noted, a new release won't resolve the issue. Apologies for the confusion.

To add some additional context, there's a number of our dependencies that are in the process of being upgraded in order to ensure that we remain on supported versions and avoid vulnerabilities. Unfortunately, we also have to ensure compatibility with existing platforms such as Azure Functions and Azure PowerShell which add constraints to the versions that we're able to adopt.

In this case, our version of System.Memory.Data was flagged as needing to be upgraded. Unfortunately, due to the dependency that it has on System.Text.Json, we're required to stick to the 6.x line for the immediate future. Because .NET has chosen not to patch the ToString bug in the 6.x line and we're unable to move to the 8.x, there is friction between our priorities of ensuring that we are using safe dependencies and avoiding platform bugs. To mitigate, we've updated calls toSystem.Memory.Data internal to the Azure SDK packages to ensure there's no behavioral change.

Unfortunately, we cannot do the same for application calls to ToString and regret the inconvenience caused by the behavioral change to your code. As Maddy recommended, the best path forward is to take a direct dependency on the newest System.Memory.Data which contains the fix. This will hoist the version used by the Azure SDK packages as well. If you are unable to do so, for some reason (such as the application being an in-proc Azure Function), then guarding ToString use with a try/catch or a length check would be the best path forward.

[github-actions]

Hi @mlankamp. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[mlankamp]

@jsquire Thank you for the update. Maybe the release notes for https://github.com/Azure/azure-sdk-for-net/releases/tag/Microsoft.Extensions.Azure_1.7.6 should be updated, so that people know that, Azure.Core has been updated and with the side effect mentioned.

[bvmeer]

@jsquire Side effect with this (minor) upgrade Microsoft.Extensions.Azure from 1.7.5 --> 1.7.6 is/was that our azure functions running .net 8 inprocess won't handle servicebus triggers any longer. We didn't see any "toString()" but just the system.memory.data binary could not be found.
Due we see more issues with missing assembly's it cost some time to realize this small nuget update caused the issue. So if someone is also struggling with this could help to rollback to 1.7.5

The error which shows up:

Exception while executing function: ServicebusHandlerName Could not load file or assembly 'System.Memory.Data, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)
Inner exception System.IO.FileNotFoundException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
at XX+d__3.MoveNext (XX.Functions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at XX.Functions.XX.Run (XX.Functions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at lambda_method81 (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)
at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker2+<InvokeAsync>d__2.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:20) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:53) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:523) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306)

Note. HTTP trigger within the same azure function project just works with no errors.

[m-redding]

@bvmeer I've seen this kind of error before with .NET framework (#39798) but haven't encountered it in functions before. The solution with .NET framework is to make sure that binding redirects are enabled, but I will take a look at this and see what the right solution would be for functions.

[m-redding]

Also @bvmeer would you be mind copying & pasting what you wrote into a separate issue? Just in case anyone else running into this then it'll be easier to have it in one spot

[Kiruyuto]

It looks like on top of what @bvmeer mentioned this upgrade seems to completely break .NET 6 in-process model to the point where host refuses to start. From what Ive seen this issue seems to occur not only on local machine but also in Azure Function Apps.

Copy-pasted ver

Azure Functions Core Tools
Core Tools Version:       4.0.6280 Commit hash: N/A +421f0144b42047aa289ce691dc6db4fc8b6143e6 (64-bit)
Function Runtime Version: 4.834.3.22875

[2024-10-13T12:09:47.288Z] Found C:\Users\XDD\Desktop\Temp\Company.FunctionApp1\Company.FunctionApp1\Company.FunctionApp1.csproj. Using for user secrets file configuration.
[2024-10-13T12:09:48.546Z] A host error has occurred during startup operation '4d9a5020-76ed-4b92-bf08-6618f0cbfd38'.
[2024-10-13T12:09:48.547Z] System.Private.CoreLib: Exception has been thrown by the target of an invocation. Microsoft.Azure.WebJobs.Extensions.Storage.Queues: Could not load file or assembly 'System.Memory.Data, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
Value cannot be null. (Parameter 'provider')
Press any key to continue....[2024-10-13T12:09:49.700Z] A host error has occurred during startup operation 'f2002cfc-2a84-4fac-8a13-40778b32baf6'.
[2024-10-13T12:09:49.701Z] System.Private.CoreLib: Exception has been thrown by the target of an invocation. Microsoft.Azure.WebJobs.Extensions.Storage.Queues: Could not load file or assembly 'System.Memory.Data, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

In my case upgrading any of the following packages seems to break. The worst part is that this is considered a patch upgrade, so it was auto-merged by Renovate app to master.

Package	Change
Azure.Storage.Blobs (source)	12.22.1 -> 12.22.2
Azure.Storage.Queues (source)	12.20.0 -> 12.20.1
Microsoft.Azure.WebJobs.Extensions.Storage (source)	5.3.2 -> 5.3.3

Here's an example repository I used for testing: https://github.com/Kiruyuto/.net6-inprocess-break.
All three packages in it are updated to their newest versions. Downgrading or rather not upgrading them appears to resolve the issue.

[github-actions]

Hi @mlankamp, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.

[mlankamp]

/unresolve

[m-redding]

@mlankamp @Kiruyuto @bvmeer see: Azure/azure-functions-host#10575

[github-actions]

Hi @mlankamp. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[github-actions]

Hi @mlankamp, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.

[cataggar]

/unresolve

Because .NET has chosen not to patch the ToString bug in the 6.x line and we're unable to move to the 8.x, there is friction between our priorities of ensuring that we are using safe dependencies and avoiding platform bugs. To mitigate, we've updated calls to System.Memory.Data internal to the Azure SDK packages to ensure there's no behavioral change.

@jsquire, thanks for the explanation. Azure.Core 1.40.0 is pulled in with Azure.Identity 1.12.0. For us, the System.Memory.Data 6.0.0 is broken, since with it any empty response fails. Sure, we can attempt upgrading it everywhere it gets pulled in like this internal pull request, but it is not a great solution. If we miss it, empty responses are going to fail.

With .NET 6 out of support, is there a path forward for referencing a newer System.Memory.Data version from Azure.Core?

[jsquire]

@cataggar: To clarify, the .NET runtime 6.0 is out of support, but the 6.x line of packages are still actively maintained and supported. We're currently working to bump the 6.x dependencies to 8.x, but doing so requires internal coordination to ensure we retain compatibility with the Azure Functions host and supported versions of PowerShell.

From context, I'm not sure if you're inquiring about an Azure Functions application (which is the context of this issue) or this is an unrelated question. If you are asking about a stand-alone scenario, then you can take a direct reference to System.Memory.Data in your application and that version will take precedence over any of your dependencies, hoisting them up.

If you're asking about an Azure Functions application, then you do not have that option and will have to wait for the host dependencies to be confirmed upgraded and fully deployed so that we can safely bump the Azure SDK dependencies.

[github-actions]

Hi @mlankamp. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[cataggar]

I'll create a new issue. My appologies for reopening this one.