Host stopped and disposed twice when WebApplicationFactory<> is used with the minimal api

[ghost]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Description
When using WebApplicationFactory<> together with the minimal api WebApplication, the underlying Host is stopped and disposed twice from separate threads.

This can lead to integration tests failing e.g. when there are custom IHostedServices registered. Dpending on the race condition, a particular instance of IHostedService can be in order: stopped, then disposed, then attempted to stop again from a different thread; or attempted to stop twice from different threads at the same time.

What happens under the hood

With minimal api and the WebApplicationFactory<Program>, the whole Program content is executed. It goes up to Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync() and waits for the IHostApplicationLifetime.ApplicationStopping event.

The test runs and at the end the WebApplicationFactory is being disposed. It calls (with some wrapper classes skipped here for brevity) Microsoft.Extensions.Hosting.Internal.Host.StopAsync()

The Host.StopAsync() calls ApplicationLifetime.StopApplication(), and thus raises the beforementioned IHostApplicationLifetime.ApplicationStopping event, thus unblocking the "main" thread executing HostingAbstractionsHostExtensions.WaitForShutdownAsync(), which then... calls Host.StopAsync() again.

Expected Behavior

Host and any registered IHostedServices should be stopped and disposed once.

Steps To Reproduce

https://github.com/mateusz-duchnowski-trainline/disposing-host-twice-issue

run dotnet test

example output, where A is an IHostedService:

[xUnit.net 00:00:01.19] DisposingHostTwiceIssue.Tests: A started
[xUnit.net 00:00:01.28] DisposingHostTwiceIssue.Tests: A stopping
[xUnit.net 00:00:01.28] DisposingHostTwiceIssue.Tests: A stopping
[xUnit.net 00:00:01.50] DisposingHostTwiceIssue.Tests: A stopped
[xUnit.net 00:00:01.50] DisposingHostTwiceIssue.Tests: A stopped

Exceptions (if any)

No response

.NET Version

6.0.200-preview.22055.15

Anything else?

No response

[martincostello]

This sounds a bit like #37631.

[ghost]

Hi @martincostello,
Looks similar. But the main issue here (which is not mentioned in #37631) is that it comes from two threads and that it affects both methods: StopAsync and DisposeAsync. So one thread can be executing StopAsync and the other one can start executing StopAsync too, or even call Dispose in the meantime.

It would be strange having to implement IHostedServices with that in mind, just for the integration test purposes.

[ghost]

Thanks for contacting us.
We're moving this issue to the .NET 7 Planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[dmitriyse]

Confirms, the problem exists. I am facing with the problem in the same scenario.

1. .Net 6 minimal api (.Net 6 Web App template)
2. builder.Services.AddHostedService(); // used to register the IHostedService
3. WebApplicationFactory in xUnit test
4. IHostedService.StopAsync <-- called twice

[federicojoselucia]

I'm experiencing the same issue.

[ghost]

Thanks for contacting us.
We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[salarzobir]

+1 any progress or a workaround for this?

[mitchdenny]

Does anyone have a minimal repro of this - the repro linked in the opening post doesn't appear to be there anymore.

[salarzobir]

Hey @mitchdenny, I just created one, https://github.com/salarzobir/DisposingHostTwiceIssue

[EraYaN]

This also happens without using minimal API, while using an old school Startup class.

[AntonSmolkov]

Same here. Dotnet 7, Xunit, WebApplicationFactory

[salarzobir]

Same problem still exists in .NET 8
I wonder if there is a workaround using the new IHostedLifecycleService ???

[mr-dokara]

Same problem in .NET 8 with Xunit, WebApplicationFactory. I had to do workaround through modifying StartAsync and StopAsync methods of the problem service

[AqlaSolutions]

Same problem

[halter73]

I changed the labels because this issue has seen a bit activity over the years, and I just got pinged about it. I think the proper fix belongs in dotnet/runtime's Host.cs, but we could also reconsider #37631.

[salarzobir]

Is this already fixed in .NET 9?

[nickogl]

The workaround is to override WebApplicationFactory.DisposeAsync() and use IHostApplicationLifetime.StopApplication() instead of Host.StopAsync(), then wait until shutdown of the hosted services and disposal of the service collection is complete.

Here's a rudimentary implementation of this workaround:

using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace Sample.Test;

public sealed class WebApplicationFactoryWrapper<TEntryPoint> : WebApplicationFactory<TEntryPoint> where TEntryPoint : class
{
	private readonly static TimeSpan ShutdownTimeout = TimeSpan.FromSeconds(5);

	public override async ValueTask DisposeAsync()
	{
		await StopApplication().ConfigureAwait(false);
		await WaitForDisposal().ConfigureAwait(false);

		foreach (var factory in Factories)
		{
			await factory.DisposeAsync().ConfigureAwait(false);
		}
	}

	private async Task StopApplication(CancellationToken forcefulStoppingToken = default)
	{
		try
		{
			var tcs = new TaskCompletionSource();
			var lifetime = Services.GetRequiredService<IHostApplicationLifetime>();
			lifetime.ApplicationStopped.Register(() => tcs.TrySetResult());
			lifetime.StopApplication();
			await tcs.Task.WaitAsync(ShutdownTimeout, forcefulStoppingToken).ConfigureAwait(false);
		}
		catch (Exception)
		{
		}
	}

	private async Task WaitForDisposal(CancellationToken cancellationToken = default)
	{
		try
		{
			while (!cancellationToken.IsCancellationRequested)
			{
				// IHostApplicationLifetime.ApplicationStopped is triggered before the host (and its service collection)
				// is disposed, so additionally wait until the service collection is disposed for a clean shutdown.
				_ = Services.GetRequiredService<IHostApplicationLifetime>();
				await Task.Delay(1, cancellationToken).ConfigureAwait(false);
			}
		}
		catch (Exception)
		{
		}
	}
}

It's hacky, but it works and suffices for testing. The proper fix I think would be to use IHostApplicationLifetime.StopApplication() in WebApplicationFactory.DisposeAsync().

Feel free to check out https://github.com/nickogl/AspNetCore.IntegrationTesting which includes this workaround and offers some neat functionality on top of WebApplicationFactory. I use this approach for orchestrating complex integration tests at work.