DistributedLock.Azure - Locks not being released as of 7/1/2024 #226
Labels
Comments
|
When you say the using never exits, do you mean that DisposeAsync() is being called but it never returns? Is this something you can reproduce reliably? What is the full repro? did you try playing around with the lock options? Just curious if changing any of them makes a difference. |
|
The repo is private so I can't share much more than what I have. Yes, we can reproduce reliably. For example, here's a test we ran:
By other locking do you mean the other providers? We have not. This is code that's in maintenance mode for at least 6 months, so we're just trying to restore normal operation. Are there any options in the library (for blob) to specify how long to retain the lock before giving up and releasing it anyway? |
|
Update: I was able replicate the issue in a isolated environment: New Storage Account w/ the code below running locally. Here's what I discovered:
The Payload class: namespace WB.Test.Locking
{
public class Payload
{
public string Id { get; set; }
public string Name { get; set; }
public string SubId { get; set; }
}
}The Azure Function: using Azure.Storage.Blobs;
using Medallion.Threading.Azure;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
using System.ComponentModel;
namespace WB.Test.Locking
{
public class Function1
{
private readonly ILogger<Function1> logger;
public Function1(ILogger<Function1> logger)
{
this.logger = logger;
}
[Function("Function1")]
public async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequest req, [Microsoft.Azure.Functions.Worker.Http.FromBody] Payload payload)
{
logger.LogInformation("C# HTTP trigger function processed a request.");
//var container = new BlobContainerClient("...", "locking-test");
var container = new BlobContainerClient("...", "locking-test");
await ExecuteWithDistributedLock(payload.Id, container, async () => {
await DoWork(payload, container);
});
logger.LogInformation($"Lock for {payload.Id} released");
return new OkObjectResult("Welcome to Azure Functions!");
}
private async Task DoWork(Payload payload, BlobContainerClient container)
{
// wait 10 seconds
logger.LogInformation($"Waiting 10 seconds...");
await Task.Delay(10000);
logger.LogInformation($"Wait complete...");
try
{
await ExecuteWithDistributedLock(payload.SubId, container, async () =>
{
logger.LogInformation($"Waiting 10 seconds...");
await Task.Delay(10000);
logger.LogInformation($"Wait complete...");
});
}
catch (Exception ex)
{
logger.LogError("sub lock failed");
}
}
private async Task ExecuteWithDistributedLock(string id, BlobContainerClient container, Func<Task> function)
{
var @lock = new AzureBlobLeaseDistributedLock(container, id);
logger.LogInformation($"Trying to acquire lock for {id}...");
await using (var handle = await @lock.TryAcquireAsync(new TimeSpan(0, 0, 11)))
{
if (handle != null)
{
logger.LogInformation($"Lock acquired for {id}...");
await function.Invoke();
logger.LogInformation($"Wait complete for {id}...");
}
else
{
throw new Exception($"Failed to get lock for {id}");
}
}
}
}
}Here's the function output: The testing process and behavior:
Output in text form: |
|
This issue is also reproduced for me. Changing "Soft Delete Options" didn't help. But I did another workaround: I manually created empty file with lock resource name. Then this file is not deleted after lease release as it was before. Now the file is permanent and only lease status is changing. |
|
Oh, that's a nice work around! Thanks for sharing! |
|
@wnbittle based on your findings do you have a theory of why the library is hanging? Are we making some assumption that’s invalid u see certain configuration scenarios? |
I tried to deep dive but stack trace doesn't reveal the steps. As this error is happening when trying to delete blob it is better to check all such places. We also know that blob which existed before doesn't cause any problems. Probably |
|
My suggestion was wrong. I was debugging and saw that it was the Microsoft client which raised this exception when trying to acquire a lease.
...
|
|
@algreat2 interesting. What I don’t understand is why it hangs without releasing the lock. The code you’re showing looks like an Acquire failure. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We've been using this library for a few years now with success. We recently ran into an issue with it on all of our function apps around 7/1/2024 timeframe.
The errors we're getting are:
The behavior we're seeing (we created a new storage container to test) is as follows:
When we look at the storage account the file is gone (so it successfully cleaned up). Here's the code we're using (sorry for the screenshot).
The crazy thing is that I can't find any information on the error above (SoftDeletedBlobNotFound) either.
What am I missing?
The text was updated successfully, but these errors were encountered: