503 error with InProcess hosting, completely fine with OutOfProcess hosting

[cvocvo]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

We have an ASP.NET website we upgrade from .NET framework 4.8 to .NET 6 with ASP.NET Core and EF Core. We've been struggling with a really odd issue for a few weeks where our upgraded site seemingly works and doesn't work.

Conditions where it works:

• If we deploy in debug mode -- it runs completely fine (this is with InProcess hosting)
• If we deploy in release mode with OutOfProcess hosting
• If we run locally on our dev PCs in debug mode

Conditions where it doesn't work (we get a 503 error)

• If we deploy in release mode with InProcess hosting
• If we run locally on our dev PCs in release mode

What happens when it doesn't work:

• If you stop the site and application pool and then start it up, we can get the site to load for ~10 seconds or so and then it starts throwing 503 errors on all types of requests -- everything from static content we want to load (images/css/js) to API calls.
• After this, in the browser we get a 503 error on every attempted page load.
• Application pool seems to continue to run, IIS site seems to continue to stay running
• No errors in event viewer
• Nothing notable in the logs when we use stdoutLogEnabled="true"
• We tried getting extra logs with Nlog to no avail
• No error details in failed request trace logs

We have tried a lot of things to try and solve this and I think we'd have it probably sorted out if we could get a more descriptive error than the 503 error we're seeing. Any ideas / help would be very much appreciated!!

Some similar posts we've looked at:

• https://weblog.west-wind.com/posts/2020/Jan/14/ASPNET-Core-IIS-InProcess-Hosting-Issues-in-NET-Core-31
• Microsoft.Data.SqlClient.SNI.x64.dll crashes when using multiple AppDomains in IIS Express and IIS Server SqlClient#1418
• https://github.com/dotnet/SqlClient/blob/main/release-notes/5.0/5.0.1.md
• Occasional crash in IIS after compile #44546
• SPA running under IIS go to 503 but w3wp dont stop #44108 (comment)
  • This seems like the exact same problem we are having.
• ANCM (In-proc) can cause IIS worker process to become unhealthy #44211

Maybe there's some extra module(s) being loaded?

Expected Behavior

We expect the site to launch and run in release mode with InProcess hosting.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

6.0.407

Anything else?

• Entity Framework Core: EF Core 8.0.0-preview.2.23113.4

  • We are using this because we need support for SQL queries for unmapped types -- more details here: Support raw SQL queries without defining an entity type for the result efcore#10753

• ASP.NET Core version: 6.0.13

  • We also tried targeting .NET 7 and ASP.NET Core 7 but that didn't help.

• The IDE (VS / VS Code/ VS4Mac) you're running on, and its version:

  • Visual Studio 2022 version 17.5.3

• Server: Windows Server 2016 (fully patched)

  • If for some reason upgrading this would solve the problem we could do that.

• Include the output of dotnet --info:
  From a dev PC:

dotnet --info
.NET SDK:
 Version:   7.0.202
 Commit:    6c74320bc3

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19045
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\7.0.202\

Host:
  Version:      7.0.4
  Architecture: x64
  Commit:       0a396acafe

.NET SDKs installed:
  2.1.403 [C:\Program Files\dotnet\sdk]
  2.1.526 [C:\Program Files\dotnet\sdk]
  2.1.617 [C:\Program Files\dotnet\sdk]
  2.1.700 [C:\Program Files\dotnet\sdk]
  2.1.818 [C:\Program Files\dotnet\sdk]
  3.1.426 [C:\Program Files\dotnet\sdk]
  5.0.104 [C:\Program Files\dotnet\sdk]
  5.0.214 [C:\Program Files\dotnet\sdk]
  5.0.303 [C:\Program Files\dotnet\sdk]
  5.0.408 [C:\Program Files\dotnet\sdk]
  6.0.310 [C:\Program Files\dotnet\sdk]
  7.0.202 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  arm64 [C:\Program Files\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\arm64\InstallLocation]
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

From the server:

dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.407
 Commit:    d8659ea151

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.14393
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.407\

Host:
  Version:      8.0.0-preview.2.23128.3
  Architecture: x64
  Commit:       30b879924a

.NET SDKs installed:
  6.0.407 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-preview.2.23128.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.0-preview.2.23128.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[Tratcher]

• Entity Framework Core: EF Core 8.0.0-preview.2.23113.4

Are you able to test without this?

If you create a new mostly empty application do you have the same issues?

• We tried getting extra logs with Nlog to no avail

Was that the AspNetCore logs with everything turned up to Debug levels?
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-7.0#configure-logging

Are you able to run it with the debugger and first-chance exceptions enabled?

When running locally in Release, can you run it with Kestrel or HttpSys instead of IIS?

[cvocvo]

@Tratcher Thank you for the quick reply; I will look into these items and circle back! We may have some debug logs still from when it was turned up; I will look. If not we can give that another go.
We may be able to test without EF Core 8 but will have to check; I do know we chose that path because it saved us days or weeks of rewrites.

Are you able to run it with the debugger and first-chance exceptions enabled?
When running locally in Release, can you run it with Kestrel or HttpSys instead of IIS?

Do you have a quick guide or docs for how to do both of those items?

[Tratcher]

https://dev.goshoom.net/2018/10/first-chance-exceptions/
https://dotnettutorials.net/lesson/kestrel-web-server-asp-net-core/

We may be able to test without EF Core 8 but will have to check; I do know we chose that path because it saved us days or weeks of rewrites.

Even if you need to disable some functionality temporarily, its good to eliminate oddities from the repro.

[ghost]

Hi @cvocvo. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[cvocvo]

Hi @cvocvo. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

We're working to gather more data; please keep this open.

[cvocvo]

@Tratcher
Sorry for the delay -- we were sorting out some other things and wanted to get that stable before diving into this more deeply.

• We downgraded from EF Core 8.0.0-preview.4.23210.7 to the latest EF Core 7.x version:
  • Inprocess hosting worked fine locally with iisexpress
  • Inprocess hosting worked fine on our staging server with IIS
  • But we still needed EF Core 8 to support SQL queries for unmapped types
• We upgraded from EF Core 8.0.0-preview.4.23210.7 to EF Core 8.2.23113.4 (latest) and everything works with inprocess hosting on staging and locally.
  • It looks like something that was causing our problem was fixed between the two EF Core 8 versions.

Thus at the moment, we're pretty confident we can close this and move on. Thank you for your help!