DbContext from pool contains many InternalEntityEntry instances after clearing change tracker

[mu88]

Ask a question

I've implemented a job scheduler that processes many records from our PostgreSQL database every few minutes. Within one of our Grafana dashboards, I noticed that the Gen2 size slowly increases over time and finally reaches a plateau:

So I was wondering what causes Gen2 to become that large and created a memory dump. To my surprise, there are thousands of InternalEntityEntry objects which retain more than a million objects from being GCed:

To be honest, this comes to my surprise as I'd have expected that context.ChangeTracker.Clear() discards those elements after saving them - but maybe I'm misunderstanding something? 🤔

So I'd like to know whether
a) that is normal/expected behavior and if yes,
b) why is it that there are so many InternalEntityEntry objects ending up in Gen2?

Include your code

public class InteractionPoint
{
    public int Key { get; set; }
    public ProcessState State { get; set; } = ProcessState.Received;
}

public enum ProcessState
{
    Received = 0,
    Incomplete = 1,
    Processed = 2,
    TombstoneReceived = 3,
    TombstoneProcessed = 4
}

public class DeliveryPlanDbContext : DbContext
{
    public DeliveryPlanDbContext(DbContextOptions options)
        : base(options)
    {
    }

    public DbSet<InteractionPoint> InteractionPoints { get; set; }
}

public async Task Foo(IDbContextFactory<MyDbContext> factory, ILogger logger)
{
    var context = factory.CreateDbContext();
    
    var strategy = context.Database.CreateExecutionStrategy();
    await strategy.ExecuteAsync(async () => 
    {
        using IDbContextTransaction transaction = await context.Database.BeginTransactionAsync(ct);
        
        var entities = await dbContext.Set<InteractionPoint>()
            .Where(entity => entity.State == ProcessState.Received)
            .OrderBy(entity => entity.Key)
            .Take(1000)
            .AsTracking()
            .ToListAsync(ct);
        await HandleEntitiesAsync(entities, ct);
        
        await context.SaveChangesAsync(ct);
        context.ChangeTracker.Clear();
        await transaction.CommitAsync();
    });
}

public async Task HandleEntitiesAsync(List<InteractionPoint> entities, CancellationToken ct)
{
    foreach (var entity in entities)
    {
        // do some processing
        entity.State = ProcessState.Processed;
    }
}

...and the configuration:

void Options(IServiceProvider provider, DbContextOptionsBuilder builder)
{
    builder
        .UseQueryTrackingBehavior(QueryTrackingBehavior.NoTracking)
        .UseNpgsql(
            connectionString,
            sqlOptions => sqlOptions.CommandTimeout(40));
}

services.AddDbContextPool<DeliveryPlanDbContext>(Options);
services.AddPooledDbContextFactory<DeliveryPlanDbContext>(Options);

For a full repro case, see this repo: https://github.com/mu88/Repro_EFCore_MemoryUsage

Include stack traces

There are no errors or exceptions.

Include verbose output

Using project 'C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\MyCompany.MyApp.Service.csproj'.
Using startup project 'C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\MyCompany.MyApp.Service.csproj'.
Writing 'C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\obj\MyCompany.MyApp.Service.csproj.EntityFrameworkCore.targets'...
dotnet msbuild /target:GetEFProjectMetadata /property:EFProjectMetadataFile=C:\Users\MyUser\AppData\Local\Temp\tmp5chq3h.tmp /verbosity:quiet /nologo C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\MyCompany.MyApp.Service.csproj
Writing 'C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\obj\MyCompany.MyApp.Service.csproj.EntityFrameworkCore.targets'...
dotnet msbuild /target:GetEFProjectMetadata /property:EFProjectMetadataFile=C:\Users\MyUser\AppData\Local\Temp\tmpj3tm2x.tmp /verbosity:quiet /nologo C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\MyCompany.MyApp.Service.csproj
Build started...
dotnet build C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\MyCompany.MyApp.Service.csproj /verbosity:quiet /nologo /p:PublishAot=false

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:05.15
Build succeeded.
dotnet exec --depsfile C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\bin\Debug\net7.0\MyCompany.MyApp.Service.deps.json --additionalprobingpath C:\Users\MyUser\.nuget\packages --additionalprobingpath "C:\Program Files (x86)\Microsoft Visual Studio\Shared\NuGetPackages" --additionalprobingpath "C:\Program Files\dotnet\sdk\NuGetFallbackFolder" --runtimeconfig C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\bin\Debug\net7.0\MyCompany.MyApp.Service.runtimeconfig.json C:\Users\MyUser\.dotnet\tools\.store\dotnet-ef\8.0.1\dotnet-ef\8.0.1\tools\net8.0\any\tools\netcoreapp2.0\any\ef.dll dbcontext list --assembly C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\bin\Debug\net7.0\MyCompany.MyApp.Service.dll --project C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\MyCompany.MyApp.Service.csproj --startup-assembly C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\bin\Debug\net7.0\MyCompany.MyApp.Service.dll --startup-project C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\MyCompany.MyApp.Service.csproj --project-dir C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\ --root-namespace MyCompany.MyApp.Service --language C# --framework net7.0 --nullable --working-dir C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service --verbose
Using assembly 'MyCompany.MyApp.Service'.
Using startup assembly 'MyCompany.MyApp.Service'.
Using application base 'C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\bin\Debug\net7.0'.
Using working directory 'C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service'.
Using root namespace 'MyCompany.MyApp.Service'.
Using project directory 'C:\work\Bitbucket\MyOrg\my-app\src\MyCompany.MyApp.Service\'.
Remaining arguments: .
Finding DbContext classes...
Finding IDesignTimeDbContextFactory implementations...
Finding application service provider in assembly 'MyCompany.MyApp.Service'...
Finding Microsoft.Extensions.Hosting service provider...
Using environment 'Development'.
Using application service provider from Microsoft.Extensions.Hosting.
Found DbContext 'DeliveryPlanDbContext'.
Finding DbContext classes in the project...
MyCompany.MyApp.Core.Persistence.DeliveryPlanDbContext

Include provider and version information

EF Core version: 7.0
Database provider: Npgsql.EntityFrameworkCore.PostgreSQL
Target framework: .NET 7.0
Operating system: Alpine Linux
IDE: not relevant as it happens on production

[ajcvickers]

This issue is lacking enough information for us to be able to fully understand what is happening. Please attach a small, runnable project or post a small, runnable code listing that reproduces what you are seeing so that we can investigate.

[mu88]

What is missing for you in the mentioned code, @ajcvickers ?

EDIT: I've added both the model and the config code

[mu88]

@ajcvickers I've created this repro app: https://github.com/mu88/Repro_EFCore_MemoryUsage

After setting up the PostgreSQL database, InteractionPoints can be seeded via HTTP POST /query?count=2000000. The background job will then pick up the newly created data and process them. After that, the processing can be stopped via HTTP POST /disable. When creating a memory dump now, the aforementioned InternalEntityEntry objects show up.

[ajcvickers]

@mu88 I am not able to access that repo. Did you make it public?

[mu88]

🤦🏻‍♂️ sry about that, it's public now

[ajcvickers]

@mu88 Your code never disposes the DbContext instance.

[mu88]

It does in our production code, it was just missing in the repo - I've updated the repro code

[ajcvickers]

Duplicate of #32652. @mu88 Can you test with EF Core 9 preview 2 and report back if you are still seeing the issue?

[mu88]

I'm sorry @ajcvickers but I cannot use a preview version on our machines 😕

Can I somehow mitigate the issue?

[ajcvickers]

@mu88 The best workaround would be to stop using context pooling.

[mu88]

Thx for your precious time and help, @ajcvickers ! Without context pooling, the memory footprint is much smaller:

[mu88]

Okay @ajcvickers , here's a screenshot from another memory dump without pooling:

As you can see, there are still thousands of InternalEntityEntry objects. But rather than being referenced from the pool, they are now held by the retry strategy 🤔

And please don't answer the best workaround would be to stop using retries 🤣

[mu88]

Is there any news about this, @AndriySvyryd / @ajcvickers ?

[AndriySvyryd]

If you take a memory dump after all contexts have been disposed it won't show any abnormalities.

Closing, as the original issue is a duplicate of #32652

[mu88]

@AndriySvyryd , I'm actually taking the snapshots after the DbContext has been disposed (see here) - so why does the issue occur then?

[AndriySvyryd]

@AndriySvyryd , I'm actually taking the snapshots after the DbContext has been disposed (see here) - so why does the issue occur then?

I disabled context pooling in your sample and I don't see InternalEntityEntry objects in the snapshot after the contexts have been disposed and GC'd