HybridCache L1 cache getting out of sync when using multiple instances

[Rick-Anderson]

Description

moved from dotnet/AspNetCore.Docs#34083, created by @dukesteen

Description

The current documentation for HybridCache does not adequately warn developers about the risk of L1 cache getting out of sync when invalidating cache entries from another instance. Consider the following example:

Instance 1
 - memorycache with tenant id 1 inside
 - redis cache nothing inside
Instance 2
 - memorycache with nothing inside
 - redis cache also nothing inside

Now instance 2 does an update on the tenant and invalidates both the redis and local cache, which gives you the following situation

Instance 1
 - memorycache with tenant id 1 inside
 - redis cache also has tenant id 1 inside
Instance 2
 - memorycache with tenant id 1 inside
 - redis cache also has tenant id 1 inside

Instance 1's layer 1 cache is now out of sync, and can respond to requests with the wrong data. This can be catastrophic if instance 2 removed access for a user while instance 1 is still serving requests for this user as if they were active.

Related issue: dotnet/aspnetcore#55308

Page URL

https://learn.microsoft.com/en-us/aspnet/core/performance/caching/hybrid?view=aspnetcore-9.0

Reproduction Steps

@dukesteen please provide public GitHub link with the simplest possible app that reproduces the problem.

Expected behavior

@dukesteen please provide

Actual behavior

@dukesteen please provide

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response