Memory Leak in new ASPNET 2.2 routing? #6102

bchavez · 2018-12-22T19:19:00Z

Describe the bug

I was editing some Razor Pages (the Project\Pages kind) then refreshing the changes in Chrome (Version 71.0.3578.98). My dev loop is tight, so I make a small edit then refresh the page, and the change is visible in Chrome.

Eventually, I noticed my edits were not making their way to the browser. On a few occasions, I noticed waiting a longer time delay before refreshing the browser sometimes helped the situation. Eventually, though, edits would no longer make their way to the browser.

Originally, I was using InProcess but decided to switch to OutOfProcess, rebooted PC, and the same problem occurred. Somewhere between 15 minutes to 1 hour of constantly editing Razor Pages in Visual Studio, I begin to notice my edits not showing up in the browser.

With out of OutOfProcess enabled, I noticed, a massive 9.7 GB memory usage in dotnet.exe

(later, trying to debug dotnet.exe, we're up at 27GB)

dotnet.exe:

As dotnet.exe consumes memory, there appears to be a few threads that pop off coreclr_shutdown. Not much to discern, but worth noting.

Next, I attached a VS debugger to dotnet.exe, broke into the debugger and landed in Program.Main(). None of my user code was running. Continue & Pausing the debugger multiple times did show one thread standing out. Specifically, Thread 3952 playing around in the new ASPNET Core 22 routing code? Microsoft.AspNetCore.Routing.DecisionTree.

Specifically, Array.Copy, DecisionTree.Resize and Capacity.set seem to be popular destinations with Thread 3952. Thread 3952 is the only thread that appears to be active during this memory leak.

To Reproduce

Steps to reproduce the behavior:

Using this version of ASP.NET Core '2.2'
Edit somewhat complex Razor Pages for about 15 minutes to 1 hour in a tight dev loop.

Expected behavior

I should be able to make edits in Razor Pages and see my changes in the browser without having to restart IISExpress or kill dotnet.exe manually or experience a memory leak during the editing experience.

Additional context

Operating System

Windows 7

IDE

Visual Studio 15.9.4

C:\>dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.101
 Commit:    236713b0b7

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.101\

Host (useful for support):
  Version: 2.2.0
  Commit:  1249f08fed

.NET Core SDKs installed:
  1.0.0 [C:\Program Files\dotnet\sdk]
  1.0.1 [C:\Program Files\dotnet\sdk]
  1.0.2 [C:\Program Files\dotnet\sdk]
  1.0.3 [C:\Program Files\dotnet\sdk]
  1.0.4 [C:\Program Files\dotnet\sdk]
  1.1.0 [C:\Program Files\dotnet\sdk]
  2.0.0 [C:\Program Files\dotnet\sdk]
  2.0.3 [C:\Program Files\dotnet\sdk]
  2.1.4 [C:\Program Files\dotnet\sdk]
  2.1.101 [C:\Program Files\dotnet\sdk]
  2.1.103 [C:\Program Files\dotnet\sdk]
  2.1.104 [C:\Program Files\dotnet\sdk]
  2.1.200 [C:\Program Files\dotnet\sdk]
  2.1.201 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.300 [C:\Program Files\dotnet\sdk]
  2.1.302 [C:\Program Files\dotnet\sdk]
  2.1.401 [C:\Program Files\dotnet\sdk]
  2.1.402 [C:\Program Files\dotnet\sdk]
  2.1.403 [C:\Program Files\dotnet\sdk]
  2.1.500 [C:\Program Files\dotnet\sdk]
  2.1.502 [C:\Program Files\dotnet\sdk]
  2.2.100 [C:\Program Files\dotnet\sdk]
  2.2.101 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Also, here is Process Monitor monitoring file access during the memory leak event:

I will continue to do more testing to see if I can narrow down the issue. Thanks for your help in advance.

Thanks,
Brian Chavez

🍫 🍪 🍭 Ronald Jenkees - Stay Crunchy

The text was updated successfully, but these errors were encountered:

rynowak · 2018-12-22T20:00:47Z

As a workaround - could you try setting EnableEndpointRouting = false and let us know if that solves the problem? We'll investigate this but that will help us narrow it down if it's related to new routing features or not.

services.AddMvc(options =>
{
    options.EnableEndpointRouting = false;
});

bchavez · 2018-12-22T20:09:15Z

Yes sir, thank you for the help! Will do, I'll report back what I find.

davidfowl · 2018-12-22T21:50:56Z

If you have a dump I’d gladly take a look (david.fowler at microsoft)

bchavez · 2018-12-23T07:45:53Z

I did more testing and found a reliable repro on my machine:

Straight up: dotnet new webapp; then edit Index.cshtml in the following way:

Press enter several times at the end of the Index.cshtml file. Maybe add a few characters like fff and aa.
Save the file.
Refresh the browser. (Notice memory increase when browser reloads.)
Delete a few characters you added.
Save the file.
Refresh the browser.
Repeat; go to 1.

Executing the repro above in a fairly quick way you can ramp up the memory usage to 3 GB in almost no time flat. In my case, the new app almost consumes 1 GB / minute. (I have an AutoHotkey shortcut that refreshes the last browser tab used).

Here is the project I used to create the following diag chart: fff.zip

Check it out:

`dotnet new webapp -o fff`

I tried taking a memory snapshot but got dialoged by Visual Studio...

So, if we load the 3 GB memory dump in WinDbg, the tail end of !DumpHeap -stat shows us the following:

0:000> .load C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0\sos.dll
0:000> !DumpHeap -stat
Statistics:
              MT    Count    TotalSize Class Name
......
000007fecfd94040       12         1784 System.Char[][]
000007fecfd88c80       16         1793 System.Boolean[]
000007fe75e05038        6         1824 System.Collections.Immutable.RefAsValueType`1[[System.Collections.Immutable.SecurePooledObject`1[[System.Collections.Generic.Stack`1[[System.Collections.Immutable.RefAsValueType`1[[System.Collections.Immutable.SortedInt32KeyNode`1[[System.Collections.Immutable.ImmutableHashSet`1+HashBucket[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]], System.Collections.Immutable]], System.Collections.Immutable]], System.Collections]], System.Collections.Immutable]][]
000007fe750fc2f8       38         1824 Microsoft.Extensions.Logging.LoggerMessage+<>c__DisplayClass4_0
000007fecfdc6a90       59         1888 System.Reflection.Emit.LineNumberInfo
000007fecf39f1e8       24         1920 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]]
000007fe7523ecf0       10         1920 Microsoft.Extensions.Caching.Memory.CacheEntry
000007fe751e78a8       24         1920 System.ComponentModel.ReflectTypeDescriptionProvider+ReflectedTypeData
000007fe74d9c9f8        6         1920 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.String, System.Private.CoreLib],[Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher+ChangeTokenInfo, Microsoft.Extensions.FileProviders.Physical]][]
000007fe7516bc28        2         1968 System.Xml.XmlWellFormedWriter+State[]
000007fe74e46e38       41         1968 Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine+<>c__DisplayClass1_0
000007fe7500c078       51         2040 Microsoft.Extensions.Logging.Logger
000007fe74fd0b98       23         2048 System.Reflection.RuntimeMethodInfo[][]
000007fe74c33ea0       18         2064 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fe758fcce0       65         2080 System.Collections.Immutable.ImmutableArray`1+Builder[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]]
000007fe758fbe10       65         2080 Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder`1[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]]
000007fe751c25c8       11         2112 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelBindingMessageProvider
000007fe756ee0f0        1         2160 System.Collections.Generic.Dictionary`2+Entry[[Microsoft.AspNetCore.Razor.Language.Legacy.CSharpTokenType, Microsoft.AspNetCore.Razor.Language],[System.String, System.Private.CoreLib]][]
000007fe756c3378        1         2160 System.Collections.Generic.Dictionary`2+Entry[[System.Type, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]][]
000007fe7545cfc0        1         2160 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Razor.Language.Legacy.CSharpKeyword, Microsoft.AspNetCore.Razor.Language]][]
000007fe75447890        1         2160 System.Collections.Generic.Dictionary`2+Entry[[System.Int32, System.Private.CoreLib],[System.String, System.Private.CoreLib]][]
000007fe75012f60        1         2160 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.Extensions.Logging.Logger, Microsoft.Extensions.Logging]][]
000007fe74c3ba88       27         2160 System.Collections.Generic.Dictionary`2[[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fecfda43c8       46         2208 System.Text.StringBuilder
000007fe75a5e4e0       40         2240 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken+SyntaxTokenWithValueAndTrivia`1[[System.Int32, System.Private.CoreLib]]
000007fe754c9880       15         2280 System.IO.Pipelines.BufferSegment[]
000007fe74fc8f28       74         2368 Microsoft.Extensions.DependencyInjection.ServiceLookup.TransientCallSite
000007fecfdc7068       76         2432 System.Reflection.Emit.GenericFieldInfo
000007fe750fc398       38         2432 System.Action`2[[Microsoft.Extensions.Logging.ILogger, Microsoft.Extensions.Logging.Abstractions],[System.Exception, System.Private.CoreLib]]
000007fe75012b20       51         2448 Microsoft.Extensions.Logging.EventSource.EventSourceLogger
000007fe74fe2718       51         2448 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.String, System.Private.CoreLib],[Microsoft.Extensions.Logging.Console.ConsoleLogger, Microsoft.Extensions.Logging.Console]]
000007fe751e5aa0       13         2496 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelMetadata
000007fe74e42b40        2         2512 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection],[System.Object, System.Private.CoreLib]], System.Private.CoreLib]][]
000007fe74c383a8        6         2512 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Resources.ResourceLocator, System.Private.CoreLib]][]
000007fe75469a48       63         2520 Microsoft.AspNetCore.Razor.Language.RazorDiagnosticDescriptor
000007fe75e15588        3         2632 System.Runtime.CompilerServices.ConditionalWeakTable`2+Entry[[Microsoft.CodeAnalysis.SyntaxNode, Microsoft.CodeAnalysis],[System.Collections.Generic.Dictionary`2[[Microsoft.CodeAnalysis.SyntaxTrivia, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.SyntaxNode, Microsoft.CodeAnalysis]], System.Private.CoreLib]][]
000007fecfdbdd38       24         2688 System.Diagnostics.Tracing.EventSource+OverideEventProvider
000007fe75222798        4         2736 Microsoft.CodeAnalysis.MetadataReference[]
000007fecfdaa1a8       50         2800 System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[[System.Reflection.RuntimeMethodInfo, System.Private.CoreLib]]
000007fe759734f0       70         2800 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEMethodSymbol+SignatureData
000007fe754c8958       10         2800 System.IO.Pipelines.PipeCompletionCallback[]
000007fecfdc6f10       59         2832 System.Reflection.Emit.DynamicMethod+RTDynamicMethod
000007fecfdc6a30       59         2832 System.Reflection.Emit.ScopeTree
000007fecfdc68a0       59         2832 System.Reflection.Emit.SignatureHelper
000007fecfda45d0      118         2832 System.RuntimeTypeHandle
000007fecf3a0270       75         3000 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]
000007fe75a69f88       95         3040 Roslyn.Utilities.TextKeyedCache`1+SharedEntryValue[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]]
000007fe755df028       31         3096 Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ViewBufferValue[][]
000007fe750e0ec0        5         3120 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection
000007fe7544cb30       49         3136 System.Action`2[[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fe75530e88       66         3168 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]]
000007fe754d40b0        5         3240 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseHeaders
000007fecfd84f20       12         3264 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Int32, System.Private.CoreLib],[System.IntPtr, System.Private.CoreLib]][]
000007fe74fe1b08       51         3264 Microsoft.Extensions.Logging.Console.ConsoleLogger
000007fe750fabd8       71         3408 Microsoft.Extensions.Logging.LoggerMessage+<>c__DisplayClass5_0`1[[System.String, System.Private.CoreLib]]
000007fe754a2e58       62         3472 System.Buffers.MemoryPoolBlock
000007fe75a704d0       73         3504 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.AttributeArgumentSyntax
000007fe754c71d0       10         3520 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs
000007fecfdb0ec8       63         3528 System.Threading.CancellationTokenSource+CallbackPartition
000007fe75751848       56         3584 System.Func`2[[Roslyn.Utilities.ObjectReader, Microsoft.CodeAnalysis],[Roslyn.Utilities.IObjectWritable, Microsoft.CodeAnalysis]]
000007fecfda84c8      151         3624 System.RuntimeMethodHandle
000007fecfda94c8       77         3696 System.Reflection.RuntimeAssembly
000007fe754d2e38        5         3720 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestHeaders
000007fe75530f30       14         3808 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]][]
000007fecfd95ee8       45         3888 System.Reflection.RuntimePropertyInfo[]
000007fe758b67e8       21         3944 Microsoft.CodeAnalysis.UnifiedAssembly`1[[Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe74e42a78       84         4032 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection],[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
000007fe758c51a8      130         4160 System.Collections.Immutable.ImmutableArray`1+Builder[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]]
000007fe758c1308      130         4160 Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder`1[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]]
000007fecfd89520       15         4200 System.Threading.IThreadPoolWorkItem[]
000007fecfdc6db8       59         4248 System.Reflection.Emit.DynamicResolver
000007fe75a685b8      138         4416 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.LiteralExpressionSyntax
000007fe74fdb490       70         4480 System.Func`1[[System.String, System.Private.CoreLib]]
000007fe758c7910      142         4544 Microsoft.CodeAnalysis.Grouping`2[[System.String, System.Private.CoreLib],[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]]
000007fe750fac80       71         4544 System.Action`3[[Microsoft.Extensions.Logging.ILogger, Microsoft.Extensions.Logging.Abstractions],[System.String, System.Private.CoreLib],[System.Exception, System.Private.CoreLib]]
000007fe74e4bfb0      114         4560 Microsoft.Extensions.DependencyInjection.ServiceLookup.IEnumerableCallSite
000007fe75ef2488       11         4664 System.ArraySegment`1[[System.Byte, System.Private.CoreLib]][]
000007fe758c83e0      118         4720 System.Collections.Generic.List`1[[System.Linq.IGrouping`2[[System.String, System.Private.CoreLib],[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]], System.Linq]]
000007fe759631b8        1         4752 System.Collections.Generic.Dictionary`2+Entry[[System.Reflection.Metadata.MethodDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEMethodSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe74c3c8e8        2         4848 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fecfdac860       76         4864 System.Reflection.RuntimeModule
000007fe758960c0      204         4896 Microsoft.CodeAnalysis.MetadataLocation
000007fe758ce730      156         4992 System.Collections.Immutable.ImmutableArray`1+Builder[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe758ca3f0      156         4992 Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe754c9010       15         5040 System.IO.Pipelines.Pipe
000007fe754a56a0       12         5088 System.Collections.Concurrent.ConcurrentQueue`1+Segment[[Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.IOQueue+Work, Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets]]
000007fecfdb9bb8       59         5192 System.RuntimeMethodInfoStub
000007fe75a5c3b0       96         5376 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken+SyntaxTokenWithValueAndTrivia`1[[System.String, System.Private.CoreLib]]
000007fe75883288      112         5376 Microsoft.CodeAnalysis.AttributeDescription
000007fe74e2e320       84         5376 System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection],[System.Object, System.Private.CoreLib]]
000007fecfdc75c8      226         5424 System.Reflection.Emit.OpCode
000007fecfdb0338       85         5440 System.Threading.CancellationTokenSource
000007fe7594a508       68         5440 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]]
000007fecfdc6e80       59         5664 System.Reflection.Emit.DynamicMethod
000007fe75af1e80        1         5760 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.WellKnownType, Microsoft.CodeAnalysis]][]
000007fecfdb20f8       52         5824 System.Threading.OverlappedData
000007fecfd89778       41         6232 System.Threading.CancellationTokenSource+CallbackPartition[]
000007fe7589b690      204         6528 Microsoft.CodeAnalysis.MetadataLocation[]
000007fe758956b8      205         6552 Microsoft.CodeAnalysis.CSharp.Symbols.ModuleSymbol[]
000007fe74e47b28      168         6720 Microsoft.Extensions.DependencyInjection.ServiceLookup.SingletonCallSite
000007fe758c72c8      119         6728 System.Linq.IGrouping`2[[System.String, System.Private.CoreLib],[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]][]
000007fe75978168       98         6736 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.TypedConstant, Microsoft.CodeAnalysis]][]
000007fe74cd7ad8        8         6864 System.Collections.Hashtable+bucket[]
000007fe75ae3c90        1         6920 System.Collections.Generic.HashSet`1+Slot[[Microsoft.CodeAnalysis.CSharp.Symbols.NamespaceSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe75a518a0      304         7296 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken
000007fecfd94d88      213         7544 System.Reflection.RuntimeConstructorInfo[]
000007fe756c8048      319         7656 System.Collections.Generic.Dictionary`2+KeyCollection[[System.String, System.Private.CoreLib],[System.ValueTuple`2[[System.Int32, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], System.Private.CoreLib]]
000007fe756fb7c8      322         7728 System.Reflection.Internal.NativeHeapMemoryBlock+DisposableData
000007fe74d909d8       26         7952 Microsoft.Extensions.DependencyInjection.ServiceDescriptor[]
000007fe7589a668      204         8160 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Reflection.Metadata.TypeReferenceHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75898fb0      204         8160 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Reflection.Metadata.TypeReferenceHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75898368      204         8160 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75896cb0      204         8160 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75895490      204         8160 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[Microsoft.CodeAnalysis.MetadataTypeName+Key, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75893b60      204         8160 System.Collections.Concurrent.ConcurrentDictionary`2[[Microsoft.CodeAnalysis.MetadataTypeName+Key, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75778c88      204         8160 Microsoft.CodeAnalysis.ModuleReferences`1[[Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe754c97e0       85         8160 System.IO.Pipelines.BufferSegment
000007fe75a57b98        1         8216 Microsoft.CodeAnalysis.CachingIdentityFactory`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.CSharp.SyntaxKind, Microsoft.CodeAnalysis.CSharp]][]
000007fecfd94248      151         8496 System.Byte[][]
000007fe7593cee0      175         8520 Microsoft.CodeAnalysis.TypedConstant[]
000007fe75da59c8        1         8544 Microsoft.CodeAnalysis.RuntimeMembers.MemberDescriptor[]
000007fe75011ef0       51         8568 Microsoft.Extensions.Logging.LoggerInformation[]
000007fecfdc6cd0       59         8968 System.Reflection.Emit.DynamicILGenerator
000007fe74fc6eb0      194         9312 Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite
000007fe754a58d0       12         9504 System.Collections.Concurrent.ConcurrentQueue`1+Segment+Slot[[Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.IOQueue+Work, Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets]][]
000007fe74e4bce0      235         9640 Microsoft.Extensions.DependencyInjection.ServiceLookup.IServiceCallSite[]
000007fe75727fb8      322        10304 Roslyn.Utilities.WeakList`1[[Microsoft.CodeAnalysis.IAssemblySymbol, Microsoft.CodeAnalysis]]
000007fe757272a8      322        10304 Microsoft.CodeAnalysis.ModuleMetadata[]
000007fe75722850      322        10304 System.Reflection.Internal.NativeHeapMemoryBlock
000007fe75855b58      323        10328 Microsoft.CodeAnalysis.PEModule[]
000007fe75854430      323        10336 Microsoft.CodeAnalysis.AssemblyMetadata+Data
000007fe758644d8        1        10368 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Generic.List`1[[Microsoft.CodeAnalysis.CommonReferenceManager`2+ReferencedAssemblyIdentity[[Microsoft.CodeAnalysis.CSharp.CSharpCompilation, Microsoft.CodeAnalysis.CSharp],[Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol, Microsoft.CodeAnalysis.CSharp]], Microsoft.CodeAnalysis]], System.Private.CoreLib]][]
000007fe750f98c8      264        10560 Microsoft.Extensions.Logging.Internal.LogValuesFormatter
000007fe74e41e00        2        10800 System.Collections.Generic.Dictionary`2+Entry[[System.Type, System.Private.CoreLib],[Microsoft.Extensions.DependencyInjection.ServiceLookup.IServiceCallSite, Microsoft.Extensions.DependencyInjection]][]
000007fecf3a03a8      282        11280 System.Collections.Generic.List`1[[System.String, System.Private.CoreLib]]
000007fe75973b80      120        11520 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEParameterSymbol
000007fecfdaa128      213        11928 System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[[System.Reflection.RuntimeConstructorInfo, System.Private.CoreLib]]
000007fe756fdec8      322        12880 System.Reflection.PortableExecutable.CoffHeader
000007fe756fb178      322        12880 Microsoft.CodeAnalysis.ModuleMetadata
000007fe7588f810      332        13280 System.Collections.Generic.List`1[[System.Collections.Immutable.ImmutableArray`1[[System.Byte, System.Private.CoreLib]], System.Collections.Immutable]]
000007fe75935c80      170        13600 System.Collections.Generic.Dictionary`2[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[System.Collections.Immutable.ImmutableArray`1[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]], System.Collections.Immutable]]
000007fe75889548      205        13640 Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol[]
000007fecfd94e00       71        13848 System.Reflection.RuntimeMethodInfo[]
000007fe74cd0438      261        14616 Microsoft.Extensions.DependencyInjection.ServiceDescriptor
000007fe7589b488      204        14688 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEGlobalNamespaceSymbol
000007fe757279d0      638        15312 System.WeakReference`1[[Microsoft.CodeAnalysis.IAssemblySymbol, Microsoft.CodeAnalysis]]
000007fe75722b80      644        15456 Microsoft.CodeAnalysis.MetadataId
000007fe7589e558       89        15552 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Collections.Immutable.ImmutableArray`1[[System.Byte, System.Private.CoreLib]], System.Collections.Immutable]], System.Private.CoreLib]][]
000007fe7590ec68      355        16016 Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol[]
000007fe74e417e8        2        16688 System.Collections.Generic.Dictionary`2+Entry[[System.Type, System.Private.CoreLib],[Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory+ServiceDescriptorCacheItem, Microsoft.Extensions.DependencyInjection]][]
000007fe75972648      494        17688 Microsoft.CodeAnalysis.CSharp.Symbols.CSharpAttributeData[]
000007fe755de758       19        17864 Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ViewBufferValue[]
000007fe75860038      322        18032 System.Reflection.Metadata.Ecma335.NamespaceCache
000007fe7589e1b8      333        18616 System.Collections.Immutable.ImmutableArray`1[[System.Byte, System.Private.CoreLib]][]
000007fe74c39bc0       21        18672 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]][]
000007fe7590fec0      803        19272 Microsoft.CodeAnalysis.CSharp.Symbols.TypeMap
000007fe75728980      322        20608 Microsoft.CodeAnalysis.MetadataImageReference
000007fe75726468      322        20608 Microsoft.CodeAnalysis.PEModule+PEHashProvider
000007fe756fb5e0      322        20608 System.Reflection.PortableExecutable.PEReader
000007fecfdb2df0      665        21280 System.LazyHelper
000007fe75855ac8      322        23184 Microsoft.CodeAnalysis.PEAssembly
000007fe757266e8      322        23184 Microsoft.CodeAnalysis.AssemblyMetadata
000007fe756fd328      322        23184 System.Reflection.PortableExecutable.PEHeaders
000007fe75966120       68        25344 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]][]
000007fe75890108      319        25520 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Collections.Immutable.ImmutableArray`1[[System.Byte, System.Private.CoreLib]], System.Collections.Immutable]], System.Private.CoreLib]]
000007fe756c7008      319        25520 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.ValueTuple`2[[System.Int32, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], System.Private.CoreLib]]
000007fe75915078      803        25696 Microsoft.CodeAnalysis.SmallDictionary`2[[Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol, Microsoft.CodeAnalysis.CSharp],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeWithModifiers, Microsoft.CodeAnalysis.CSharp]]
000007fe75724fa0      644        25760 System.Lazy`1[[Microsoft.CodeAnalysis.IdentifierCollection, Microsoft.CodeAnalysis]]
000007fe75895de8      204        27744 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEModuleSymbol
000007fe7588f1c0      204        27744 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEAssemblySymbol
000007fe75722368      322        28336 System.Reflection.PortableExecutable.CorHeader
000007fecfd86b70      477        28504 System.String[]
000007fecfd88258      913        28616 System.RuntimeType[]
000007fe75968730      239        28680 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEPropertySymbol
000007fe758de2f8      816        29184 Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol[]
000007fecfdad200      304        31616 System.Reflection.RuntimeConstructorInfo
000007fecfd96168     1068        31728 System.Reflection.ParameterInfo[]
000007fecfd834e8        8        33144 System.UInt32[]
000007fecfdaaa20      327        34008 System.Reflection.RuntimePropertyInfo
000007fecfd83550        6        35692 System.UInt16[]
000007fe757281f8      639        35752 System.WeakReference`1[[Microsoft.CodeAnalysis.IAssemblySymbol, Microsoft.CodeAnalysis]][]
000007fe75a5f028      921        36840 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken+SyntaxTokenWithTrivia
000007fecfda9fa8      245        37240 System.RuntimeType+RuntimeTypeCache
000007fe75724c48      322        38640 Microsoft.CodeAnalysis.PEModule
000007fe75724e80      644        41216 System.Func`1[[Microsoft.CodeAnalysis.IdentifierCollection, Microsoft.CodeAnalysis]]
000007fe758d9c20      697        44608 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.MetadataDecoder
000007fe756fbf50      322        46368 System.Reflection.PortableExecutable.SectionHeader[]
000007fe75a55848        1        49176 Roslyn.Utilities.TextKeyedCache`1+LocalEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe75a54ac8        1        49176 Roslyn.Utilities.TextKeyedCache`1+LocalEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxTrivia, Microsoft.CodeAnalysis.CSharp]][]
000007fe75895250      793        50752 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.CodeAnalysis.MetadataTypeName+Key, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75971bb8      697        55760 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEAttributeData
000007fe7589a428     1187        56976 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeReferenceHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe758d6068     2377        57048 System.Collections.Generic.Dictionary`2+ValueCollection[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]]
000007fe7589a4f0      204        60048 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeReferenceHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe758d10c8      983        61936 Microsoft.CodeAnalysis.CSharp.Symbol[]
000007fe758622f0      848        63376 Microsoft.CodeAnalysis.AssemblyIdentity[]
000007fecfdace08      696        66816 System.Reflection.RuntimeParameterInfo
000007fecfdacd88      844        67520 System.Signature
000007fe75895318      204        70176 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.CodeAnalysis.MetadataTypeName+Key, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe7594f0f0      677        70408 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEMethodSymbol
000007fe75916608     1129        72256 Microsoft.CodeAnalysis.SmallDictionary`2+AvlNode[[Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol, Microsoft.CodeAnalysis.CSharp],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeWithModifiers, Microsoft.CodeAnalysis.CSharp]]
000007fecfdac078     2425        77600 System.Version
000007fe758f8d80      883        77704 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+UncommonProperties
000007fe756fe9a8      322        79856 System.Reflection.PortableExecutable.PEHeader
000007fe758fe070     1705        81840 Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol+InterfaceInfo
000007fe7590efe0      937        89952 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PETypeParameterSymbol
000007fe758ca188      862        89952 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENestedNamespaceSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe75936840      148        90960 System.Collections.Generic.Dictionary`2+Entry[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[System.Collections.Immutable.ImmutableArray`1[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]], System.Collections.Immutable]][]
000007fecfda7e40     2317        92680 System.RuntimeType
000007fecfdaa468      893        92872 System.Reflection.RuntimeMethodInfo
000007fe7591d250      802        96240 Microsoft.CodeAnalysis.CSharp.Symbols.ConstructedNamedTypeSymbol
000007fe758c5728     2641        96508 System.Reflection.Metadata.TypeDefinitionHandle[]
000007fe758c95b8     1405       112400 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENestedNamespaceSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe758d8008     3052       115832 Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol[]
000007fe758c93b0     1319       116072 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENestedNamespaceSymbol
000007fecfda4420     5668       136032 System.Object
000007fe758981f0      204       178032 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe758cb980     2377       190160 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]]
000007fe7575ee08     2423       193840 Microsoft.CodeAnalysis.AssemblyIdentity
000007fe758cf5b0     1169       224448 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+PENamedTypeSymbolGeneric
000007fe756c8708       63       229968 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.ValueTuple`2[[System.Int32, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], System.Private.CoreLib]][]
000007fecfd861c8      951       236136 System.Object[]
000007fe75a5a258        4       272000 Microsoft.CodeAnalysis.ArrayElement`1[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fecfd88ce8      129       300858 System.Char[]
000007fe758d5488     1810       422832 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]][]
000007fe758ce998    12919       578824 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol[]
000007fe75898128    12308       590784 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe7585cc48      322       757344 System.Reflection.Metadata.MetadataReader
000007fe75a68a68        1      1048600 Microsoft.CodeAnalysis.Syntax.InternalSyntax.SyntaxNodeCache+Entry[]
000007fe75a54e60        1      1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe75a53d30        1      1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxTrivia, Microsoft.CodeAnalysis.CSharp]][]
000007fe7585e750        4      1146976 Roslyn.Utilities.StringTable+Entry[]
000007fecfd88d50     3882      1314856 System.Byte[]
000007fe758cf908    11139      1871352 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+PENamedTypeSymbolNonGeneric
000007fecfda3770    31429      2486442 System.String
000007fecfde0c88   142268      3414432 System.Collections.Generic.ComparisonComparer`1[[System.Int32, System.Private.CoreLib]]
000007fe75687238   142268      3414432 Microsoft.AspNetCore.Routing.Internal.LinkGenerationDecisionTree+OutboundMatchClassifier
000007fe75689f80   142270      3414480 Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValueEqualityComparer
000007fe75687430   142275      3414600 Microsoft.AspNetCore.Routing.RouteValueEqualityComparer
000007fe75686e70   142275      3414600 Microsoft.AspNetCore.Routing.Internal.LinkGenerationDecisionTree
000007fe75683688   142270      4552640 System.SZArrayHelper+SZGenericArrayEnumerator`1[[Microsoft.AspNetCore.Http.Endpoint, Microsoft.AspNetCore.Routing.Abstractions]]
000007fe75690128   142274      4552768 Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe756921e0   142270      5690784 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe75684e88   284554      6829296 Microsoft.AspNetCore.Routing.Template.TemplateSegment
000007fe756927a8   142268      7967008 System.Linq.EnumerableSorter`2[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
000007fe756907d0   142275      7967368 Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]][]
000007fecfda2170   142268      9105152 System.Comparison`1[[System.Int32, System.Private.CoreLib]]
000007fe756868e8   284555      9105752 Microsoft.AspNetCore.Routing.Template.TemplatePart[]
000007fecfdb0f38   284595      9107040 System.Threading.CancellationTokenRegistration
000007fe75693720   142270     10243440 System.Collections.Generic.HashSet`1+Slot[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe7568d1c8   142269     11381520 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe75692ce0   142275     11382000 System.Collections.Generic.Dictionary`2[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe75682ae0   142275     11382000 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Internal.OutboundMatchResult, Microsoft.AspNetCore.Routing]], System.Private.CoreLib]]
00000000004f8760   514256     12352146      Free
000007fe75685958   284536     13657728 System.Linq.Enumerable+SelectArrayIterator`2[[Microsoft.AspNetCore.Routing.Patterns.RoutePatternPathSegment, Microsoft.AspNetCore.Routing],[Microsoft.AspNetCore.Routing.Template.TemplateSegment, Microsoft.AspNetCore.Routing]]
000007fe7568fb70   142269     13657824 System.Collections.Generic.Dictionary`2+Entry[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing],[System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]][]
000007fe7568f380   142269     13657824 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe75693bd8   142274     13658304 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe756816e0   426831     13658592 Microsoft.AspNetCore.Routing.Tree.OutboundMatch
000007fe75686268   284554     15935024 Microsoft.AspNetCore.Routing.Template.TemplatePart
000007fe75685eb8   284555     15935048 Microsoft.AspNetCore.Routing.Template.TemplateSegment[]
000007fe756877f0   426810     17072400 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe756851f0   426831     17073240 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplateSegment, Microsoft.AspNetCore.Routing]]
000007fe75684d70   426831     17073240 Microsoft.AspNetCore.Routing.Template.RouteTemplate
000007fe7568a3a0   569072     18210304 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+TreeBuilderContext[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe75692c38   569096     18211072 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe756878a0   569076     22763040 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe756901d0   569097     22763880 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe7544c250   426806     23901136 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fe75681770   711369     28454760 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe756862f8   711385     28455400 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplatePart, Microsoft.AspNetCore.Routing]]
000007fe75687e90   569077     30730128 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]][]
000007fe74e80ce8   426806     30730928 System.Collections.Generic.HashSet`1+Slot[[System.String, System.Private.CoreLib]][]
000007fe756914e0   569070     31867920 System.Linq.OrderedEnumerable`2[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
000007fe756834a0   569095     31869288 Microsoft.AspNetCore.Routing.Tree.OutboundMatch[]
000007fe75688a18   426810     34144800 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fe7551c2a0   853678     34147120 Microsoft.AspNetCore.Routing.RouteValueDictionary
000007fe7568a448   569072     36420608 System.Collections.Generic.HashSet`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe74e80438   569075     36420800 System.Collections.Generic.HashSet`1[[System.String, System.Private.CoreLib]]
000007fe75691d98   569070     40973040 System.Linq.OrderedEnumerable`1+<GetEnumerator>d__3[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]]
000007fe75689ca8   426810     40973760 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]][]
000007fe75684b00   426831     40975776 Microsoft.AspNetCore.Routing.Tree.OutboundRouteEntry
000007fe7568d9a8   569070     45525600 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe7568e708   853605     47801880 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fecfd82aa8  1711821     59556316 System.Int32[]
000007fe74d12b28  1715460     68618400 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fecfd9a4b0   853692     75124192 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fe74d128d8  1715464    109789696 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fecfd68868  1715471    109790144 System.Action
000007fecfd95b20  1715491    109791424 System.Action`1[[System.Object, System.Private.CoreLib]]
000007fecfdb23c0  3168844    253507520 System.Threading.CancellationTokenSource+CallbackNode
Total 30766425 objects

And from my app that started this issue (1.7 GB dump):

0:000> .load C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0\sos.dll
0:000> !DumpHeap -stat
Statistics:
              MT    Count    TotalSize Class Name
......
000007fed0a28d50     4588      1042037 System.Byte[]
000007fe7682a1f0        1      1048600 Microsoft.CodeAnalysis.Syntax.InternalSyntax.SyntaxNodeCache+Entry[]
000007fe76816950        1      1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe76815820        1      1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxTrivia, Microsoft.CodeAnalysis.CSharp]][]
000007fe76299348    26681      1067240 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe762b1690    27000      1080000 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe76488c30        4      1146976 Roslyn.Utilities.StringTable+Entry[]
000007fe76296b40    21057      1179160 Microsoft.AspNetCore.Routing.Template.TemplateSegment[]
000007fe762923f8    29978      1199120 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe76155e88    23569      1319864 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fe76297770    39105      1467928 Microsoft.AspNetCore.Routing.Template.TemplatePart[]
000007fe762b29a0    26538      1486128 System.Linq.OrderedEnumerable`2[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
000007fe762b4be0     8873      1492488 System.Collections.Generic.HashSet`1+Slot[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe76299938    26682      1588656 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]][]
000007fe76296ef0    30080      1684480 Microsoft.AspNetCore.Routing.Template.TemplatePart
000007fe7629bef0    26550      1699200 System.Collections.Generic.HashSet`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe75ca2568    26554      1699456 System.Collections.Generic.HashSet`1[[System.String, System.Private.CoreLib]]
000007fe75ca2920    23588      1699552 System.Collections.Generic.HashSet`1+Slot[[System.String, System.Private.CoreLib]][]
000007fe76294128    29929      1771256 Microsoft.AspNetCore.Routing.Tree.OutboundMatch[]
000007fe7629a4c0    23752      1900160 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fe762b3258    26537      1910664 System.Linq.OrderedEnumerable`1+<GetEnumerator>d__3[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]]
000007fe76169bf0    48147      1925880 Microsoft.AspNetCore.Routing.RouteValueDictionary
000007fe7629f450    26545      2123600 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe762b10a8     8859      2129952 System.Collections.Generic.Dictionary`2+Entry[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing],[System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]][]
000007fe762b5098     8905      2146368 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe76296f80    54144      2165760 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplatePart, Microsoft.AspNetCore.Routing]]
000007fe7629b750    23752      2280192 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]][]
000007fe76295788    24064      2310144 Microsoft.AspNetCore.Routing.Tree.OutboundRouteEntry
000007fe762b0240    47141      2639896 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fed0a43770    33901      3117148 System.String
000007fed0a22aa8    84229      3798748 System.Int32[]
000007fed0a3a4b0    48168      4237744 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fe75612c10  2739466    109578640 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fe756129c0  2739473    175326272 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fed0a08868  2739477    175326528 System.Action
000007fed0a35b20  2739501    175328064 System.Action`1[[System.Object, System.Private.CoreLib]]
00000000005a8d90  1558991    175859240      Free
000007fed0a523c0  3381845    270547600 System.Threading.CancellationTokenSource+CallbackNode
Total 17048248 objects

FWIW, I found that EnableEndpointRouting = false helps the situation (in the case of the fff.zip project). Memory consumption is at a slower rate but still shares the same monotonically increasing characteristics. (approx 700MB / 10 minutes).

In both cases, memory is never reclaimed by the OS.

`EnableEndpointRouting = false`:

@davidfowl , I'll send you an email soon.

@rynowak , I'll continue to do more testing with EnableEndpointRouting = false. I just wanted to make a quick post with what I found so far.

Thanks,
Brian

💥 🔥 "Set it ablaze like a candle wick... Light it up, light it up..."

davidfowl · 2018-12-23T11:51:12Z

I have a hunch but I need to see the gc root of these:

000007fe75612c10  2739466    109578640 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fe756129c0  2739473    175326272 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fed0a08868  2739477    175326528 System.Action
000007fed0a35b20  2739501    175328064 System.Action`1[[System.Object, System.Private.CoreLib]]
000007fed0a523c0  3381845    270547600 System.Threading.CancellationTokenSource+CallbackNode

This looks like it's the leak. Something isn't cleaning up cancellation token registrations (I've been on a mission cleaning this up since 2.2 but we still have a ways to go).

davidfowl · 2018-12-23T12:29:21Z

Yep, got it:

davidfowl · 2018-12-23T12:40:20Z

It looks like these aren't rooted though. I think they'll eventually get GCed. Still looking.

bchavez · 2018-12-23T18:34:55Z

I don't know if this is two sides to the same coin, but in my webapp, (generally following the same repro steps), I can get ASPNET to consume memory without any file saves or browser refreshes.

In the GIF below:

The first ramp up is using the same repro steps posted.
However, I stopped and waited for ASPNET to catch up... memory usage leveled off (the plateau/flat part in the pic).
Then started the same repro steps again.. but about 10-20 iterations. Then stopped.
Except, this time, ASPNET continued eating memory without any typing, saving, or browser refresh steps.
Just sit back and relax. There's a persistent 10-30% CPU usage as ASPNET goes on by itself. This is what is shown in the 2nd ramp up.

Waited up to 7GB (not doing anything) and still no end in sight:

Paused the debugger, caught Thread 7 in Array.Copy:

Created dump. Loaded in WinDbg:

0:000> .load C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0\sos.dll
0:000> !threads -live
ThreadCount:      49
UnstartedThread:  0
BackgroundThread: 19
PendingThread:    0
DeadThread:       29
Hosted Runtime:   no
                                                                                                        Lock  
       ID OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
   0    1  6c4 00000000004fcbf0  202a028 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     MTA 
  15    2 32d0 0000000000561360    2b228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     MTA (Finalizer) 
  16    3 1274 0000000000480ac0  102a228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     MTA (Threadpool Worker) 
  17    7 3088 000000000d359270  10a9228 Preemptive  000000025D992928:000000025D9946F0 000000000045d670 4     MTA (Threadpool Worker) 
  18    8 2cb0 000000000d39de70  202b228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     MTA 
  19    9 291c 000000000d482010  202b228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     MTA 
  21   18 27e8 000000000d59ac70  10a9228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     MTA (Threadpool Worker) 
  22   34 1218 000000001277d050    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  23   35 2ccc 000000001277d6e0    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  24   36 2c0c 000000001277dd70    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  25   37 2c94 000000001277e400    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  26   38 2d54 000000001277ea90    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  27   39 322c 000000001277f120    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  28   40 23a8 00000000127917c0    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  29   41 2d90 0000000012791e50    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  30   42 2e30 00000000127924e0    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  31   43 2d6c 0000000012792b70    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  32   44 33b8 0000000012793200    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  33   45 2434 0000000012793890    21228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     Ukn 
  34   33 2810 000000000d5998c0  8029228 Preemptive  0000000000000000:0000000000000000 000000000045d670 0     MTA (Threadpool Completion Port) 
0:000> ~17s
ntdll!ZwWaitForSingleObject+0xa:
00000000`7742d9fa c3              ret
0:017> !clrstack
OS Thread Id: 0x3088 (17)
        Child SP               IP Call Site
000000000eacdca8 000000007742d9fa [HelperMethodFrame_1OBJ: 000000000eacdca8] System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32, Boolean)
000000000eacdf90 000007fed47de7f8 System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32) [E:\A\_work\287\s\src\mscorlib\src\System\Array.cs @ 231]
000000000eacdfd0 000007fed344236b System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].SetCapacity(Int32) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1192]
000000000eace030 000007fed34422e1 System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].IncreaseCapacity() [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1174]
000000000eace060 000007fed344250f System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].AddIfNotPresent(System.__Canon) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1247]
000000000eace0e0 000007fed34411db System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].Add(System.__Canon) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 469]
000000000eace110 000007fecfef6bc2 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1[[System.__Canon, System.Private.CoreLib]].GenerateNode(TreeBuilderContext<System.__Canon>, Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValueEqualityComparer, System.Collections.Generic.List`1<Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1<System.__Canon>>) [/_/shared/Microsoft.AspNetCore.Routing.DecisionTree.Sources/DecisionTreeBuilder.cs @ 161]
000000000eace240 000007fecfef6623 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1[[System.__Canon, System.Private.CoreLib]].GenerateTree(System.Collections.Generic.IReadOnlyList`1<System.__Canon>, Microsoft.AspNetCore.Routing.DecisionTree.IClassifier`1<System.__Canon>) [/_/shared/Microsoft.AspNetCore.Routing.DecisionTree.Sources/DecisionTreeBuilder.cs @ 91]
000000000eace2b0 000007fecff09932 Microsoft.AspNetCore.Routing.Internal.LinkGenerationDecisionTree..ctor(System.Collections.Generic.IReadOnlyList`1<Microsoft.AspNetCore.Routing.Tree.OutboundMatch>) [/_/src/Microsoft.AspNetCore.Routing/Internal/LinkGenerationDecisionTree.cs @ 27]
000000000eace2f0 000007fecfef5f06 Microsoft.AspNetCore.Routing.RouteValuesAddressScheme.BuildOutboundMatches() [/_/src/Microsoft.AspNetCore.Routing/RouteValuesAddressScheme.cs @ 95]
000000000eace340 000007fecfef5e53 Microsoft.AspNetCore.Routing.RouteValuesAddressScheme.HandleChange() [/_/src/Microsoft.AspNetCore.Routing/RouteValuesAddressScheme.cs @ 84]
000000000eace380 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace3c0 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace450 000007fecfeecc68 Microsoft.AspNetCore.Routing.CompositeEndpointDataSource.HandleChange() [/_/src/Microsoft.AspNetCore.Routing/CompositeEndpointDataSource.cs @ 112]
000000000eace4c0 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace500 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace590 000007fed2f6dc22 Microsoft.AspNetCore.Mvc.Internal.MvcEndpointDataSource.UpdateEndpoints() [/_/src/Microsoft.AspNetCore.Mvc.Core/Internal/MvcEndpointDataSource.cs @ 201]
000000000eace690 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace6d0 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace760 000007fed2f792c6 Microsoft.AspNetCore.Mvc.Infrastructure.DefaultActionDescriptorCollectionProvider.UpdateCollection() [/_/src/Microsoft.AspNetCore.Mvc.Core/Infrastructure/DefaultActionDescriptorCollectionProvider.cs @ 153]
000000000eace7d0 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace810 000007fed48977f5 System.Threading.CancellationTokenSource.InternalRegister(System.Action`1<System.Object>, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 546]
000000000eace8a0 000007fed486aa40 System.Threading.CancellationToken.Register(System.Action`1<System.Object>, System.Object, Boolean, Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationToken.cs @ 236]
000000000eace910 000007fed486a93e System.Threading.CancellationToken.Register(System.Action`1<System.Object>, System.Object) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationToken.cs @ 192]
000000000eace950 000007fed33e9636 Microsoft.Extensions.Primitives.CompositeChangeToken.RegisterChangeCallback(System.Action`1<System.Object>, System.Object) [/_/src/Primitives/src/CompositeChangeToken.cs @ 48]
000000000eace9a0 000007fed33ec5d9 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 47]
000000000eace9e0 000007fed4791999 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\287\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 167]
000000000eacea60 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eaceaf0 000007fed33e992a Microsoft.Extensions.Primitives.CompositeChangeToken.OnChange(System.Object) [/_/src/Primitives/src/CompositeChangeToken.cs @ 117]
000000000eaceb50 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eacebe0 000007fed2e0c370 Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher+<>c.<.cctor>b__43_0(System.Object) [/_/src/FileProviders/Physical/src/PhysicalFilesWatcher.cs @ 27]
000000000eacec10 000007fed4791999 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\287\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 167]
000000000eacec90 000007fed484453a System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\Tasks\Task.cs @ 2440]
000000000eaced30 000007fed487c0d2 System.Threading.ThreadPoolWorkQueue.Dispatch() [E:\A\_work\287\s\src\mscorlib\src\System\Threading\ThreadPool.cs @ 588]
000000000eacf1b0 000007fed51aa673 [DebuggerU2MCatchHandlerFrame: 000000000eacf1b0] 
0:017> !DumpHeap -stat
Statistics:
              MT    Count    TotalSize Class Name
.....
000007fed4b68ce8      123       263756 System.Char[]
000007fe766161f8      353       270608 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe7697fbe8        4       272000 Microsoft.CodeAnalysis.ArrayElement`1[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe76649530     4066       325280 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]]
000007fe7664d160     1720       330240 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+PENamedTypeSymbolGeneric
000007fed4b8a468     3366       350064 System.Reflection.RuntimeMethodInfo
000007fed4b87e40     9832       393280 System.RuntimeType
000007fe7641be48    13774       440768 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+TreeBuilderContext[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fed4b661c8     2065       482992 System.Object[]
000007fe764165e0    10453       501744 System.Linq.Enumerable+SelectArrayIterator`2[[Microsoft.AspNetCore.Routing.Patterns.RoutePatternPathSegment, Microsoft.AspNetCore.Routing],[Microsoft.AspNetCore.Routing.Template.TemplateSegment, Microsoft.AspNetCore.Routing]]
000007fe76419298    14440       577600 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe76415b10    24970       599280 Microsoft.AspNetCore.Routing.Template.TemplateSegment
000007fe76419348    15535       621400 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe76412368    19976       639232 Microsoft.AspNetCore.Routing.Tree.OutboundMatch
000007fe766632e8     2983       653232 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]][]
000007fe762d5e88    11889       665784 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fe764340f8    20874       667968 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe764329a0    13595       761320 System.Linq.OrderedEnumerable`2[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
000007fe76415e78    19976       799040 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplateSegment, Microsoft.AspNetCore.Routing]]
000007fe764159f8    19976       799040 Microsoft.AspNetCore.Routing.Template.RouteTemplate
000007fe76431690    20875       835000 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe76434be0     4864       841344 System.Collections.Generic.HashSet`1+Slot[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe765dac58      359       844368 System.Reflection.Metadata.MetadataReader
000007fe7664c548    19839       866632 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol[]
000007fe75e22920    12035       867736 System.Collections.Generic.HashSet`1+Slot[[System.String, System.Private.CoreLib]][]
000007fe7641bef0    13774       881536 System.Collections.Generic.HashSet`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe75e22568    13779       881856 System.Collections.Generic.HashSet`1[[System.String, System.Private.CoreLib]]
000007fe76616130    18637       894576 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe764123f8    22858       914320 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe76419938    15536       921664 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]][]
000007fe76433258    13573       977256 System.Linq.OrderedEnumerable`1+<GetEnumerator>d__3[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]]
000007fe76416b40    17480       978848 Microsoft.AspNetCore.Routing.Template.TemplateSegment[]
000007fe7698e060        1      1048600 Microsoft.CodeAnalysis.Syntax.InternalSyntax.SyntaxNodeCache+Entry[]
000007fe7697a7f0        1      1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe769796c0        1      1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxTrivia, Microsoft.CodeAnalysis.CSharp]][]
000007fe7641f450    13709      1096720 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe765dc760        4      1146976 Roslyn.Utilities.StringTable+Entry[]
000007fe7641a4c0    14440      1155200 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fe764310a8     4742      1183968 System.Collections.Generic.Dictionary`2+Entry[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing],[System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]][]
000007fe76417770    32462      1218560 Microsoft.AspNetCore.Routing.Template.TemplatePart[]
000007fed4b68d50     5018      1274282 System.Byte[]
000007fe76414128    22116      1301920 Microsoft.AspNetCore.Routing.Tree.OutboundMatch[]
000007fe76430240    23802      1332912 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fe7641b750    14440      1386240 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]][]
000007fe76416ef0    24970      1398320 Microsoft.AspNetCore.Routing.Template.TemplatePart
000007fe76435098     5454      1454928 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe762e9bf0    39995      1599800 Microsoft.AspNetCore.Routing.RouteValueDictionary
000007fe76416f80    44946      1797840 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplatePart, Microsoft.AspNetCore.Routing]]
000007fe76415788    19976      1917696 Microsoft.AspNetCore.Routing.Tree.OutboundRouteEntry
000007fed4b62aa8    53683      2569680 System.Int32[]
000007fe7664d4b8    16917      2842056 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+PENamedTypeSymbolNonGeneric
000007fed4b7a4b0    40035      3521128 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fed4b83770    39487      3634552 System.String
000000000054c350  5807424    291879606      Free
000007fe75792c10 15268935    610757400 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fe757929c0 15268942    977212288 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fed4b48868 15268958    977213312 System.Action
000007fed4b75b20 15268971    977214144 System.Action`1[[System.Object, System.Private.CoreLib]]
000007fed4b923c0 28440798   2275263840 System.Threading.CancellationTokenSource+CallbackNode
Total 96275880 objects
Fragmented blocks larger than 0.5 MB:
            Addr     Size      Followed by
00000003419c8240    0.6MB 0000000341a57b08 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol[]

@davidfowl , this memory dump is on the same server with filename dotnet.bitarmory.web.arraycopy.zip if you want to peek.

rynowak · 2018-12-23T23:23:29Z

This call stack implies that the file watcher is triggering routing to recompute the route table after a file change occurs. Looking at the previous provided list of threads/stacks shows a LOT of threads with similar stacks.

I don't think it's a leak because all of the memory is still live.

So one of a few obvious possibilities seems possible:

We have a broken behaviour when a file change occurs that will cause us to burn a thread by spinning over and over recomputing file changes. As you make more changes, more threads get stuck.
We subscribe to the same change token a bunch of times, causing us to to update the route table
N time when a file change occurs.
File watcher is going BANANAS and triggering like a million notifications for some reason leading to lots of updates that need to be processed.
You are legitimately triggering a bunch of file change notifications (somehow) which is causing the above.

I don't see anything obviously wrong in this call stack. DecisionTreeBuilder is thread-safe so there's no concurrency issue inside that.

@davidfowl - if you're looking into this also, there interesting bits are here - in order:
https://github.com/aspnet/AspNetCore/blob/release/2.2/src/Mvc/src/Microsoft.AspNetCore.Mvc.RazorPages/Internal/PageActionDescriptorChangeProvider.cs
https://github.com/aspnet/AspNetCore/blob/321327f84b2b22dcff2e9beb06a9a64236c5cced/src/Mvc/src/Microsoft.AspNetCore.Mvc.Core/Infrastructure/DefaultActionDescriptorCollectionProvider.cs#L144
https://github.com/aspnet/AspNetCore/blob/master/src/Mvc/src/Microsoft.AspNetCore.Mvc.Core/Routing/MvcEndpointDataSource.cs#L184
https://github.com/aspnet/AspNetCore/blob/master/src/Http/Routing/src/CompositeEndpointDataSource.cs#L139
https://github.com/aspnet/AspNetCore/blob/master/src/Http/Routing/src/RouteValuesAddressScheme.cs#L76

000000000eacdca8 000000007742d9fa [HelperMethodFrame_1OBJ: 000000000eacdca8] System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32, Boolean)
000000000eacdf90 000007fed47de7f8 System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32) [E:\A\_work\287\s\src\mscorlib\src\System\Array.cs @ 231]
000000000eacdfd0 000007fed344236b System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].SetCapacity(Int32) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1192]
000000000eace030 000007fed34422e1 System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].IncreaseCapacity() [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1174]
000000000eace060 000007fed344250f System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].AddIfNotPresent(System.__Canon) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1247]
000000000eace0e0 000007fed34411db System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].Add(System.__Canon) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 469]
000000000eace110 000007fecfef6bc2 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1[[System.__Canon, System.Private.CoreLib]].GenerateNode(TreeBuilderContext<System.__Canon>, Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValueEqualityComparer, System.Collections.Generic.List`1<Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1<System.__Canon>>) [/_/shared/Microsoft.AspNetCore.Routing.DecisionTree.Sources/DecisionTreeBuilder.cs @ 161]
000000000eace240 000007fecfef6623 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1[[System.__Canon, System.Private.CoreLib]].GenerateTree(System.Collections.Generic.IReadOnlyList`1<System.__Canon>, Microsoft.AspNetCore.Routing.DecisionTree.IClassifier`1<System.__Canon>) [/_/shared/Microsoft.AspNetCore.Routing.DecisionTree.Sources/DecisionTreeBuilder.cs @ 91]
000000000eace2b0 000007fecff09932 Microsoft.AspNetCore.Routing.Internal.LinkGenerationDecisionTree..ctor(System.Collections.Generic.IReadOnlyList`1<Microsoft.AspNetCore.Routing.Tree.OutboundMatch>) [/_/src/Microsoft.AspNetCore.Routing/Internal/LinkGenerationDecisionTree.cs @ 27]
000000000eace2f0 000007fecfef5f06 Microsoft.AspNetCore.Routing.RouteValuesAddressScheme.BuildOutboundMatches() [/_/src/Microsoft.AspNetCore.Routing/RouteValuesAddressScheme.cs @ 95]
000000000eace340 000007fecfef5e53 Microsoft.AspNetCore.Routing.RouteValuesAddressScheme.HandleChange() [/_/src/Microsoft.AspNetCore.Routing/RouteValuesAddressScheme.cs @ 84]
000000000eace380 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace3c0 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace450 000007fecfeecc68 Microsoft.AspNetCore.Routing.CompositeEndpointDataSource.HandleChange() [/_/src/Microsoft.AspNetCore.Routing/CompositeEndpointDataSource.cs @ 112]
000000000eace4c0 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace500 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace590 000007fed2f6dc22 Microsoft.AspNetCore.Mvc.Internal.MvcEndpointDataSource.UpdateEndpoints() [/_/src/Microsoft.AspNetCore.Mvc.Core/Internal/MvcEndpointDataSource.cs @ 201]
000000000eace690 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace6d0 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace760 000007fed2f792c6 Microsoft.AspNetCore.Mvc.Infrastructure.DefaultActionDescriptorCollectionProvider.UpdateCollection() [/_/src/Microsoft.AspNetCore.Mvc.Core/Infrastructure/DefaultActionDescriptorCollectionProvider.cs @ 153]
000000000eace7d0 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace810 000007fed48977f5 System.Threading.CancellationTokenSource.InternalRegister(System.Action`1<System.Object>, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 546]
000000000eace8a0 000007fed486aa40 System.Threading.CancellationToken.Register(System.Action`1<System.Object>, System.Object, Boolean, Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationToken.cs @ 236]
000000000eace910 000007fed486a93e System.Threading.CancellationToken.Register(System.Action`1<System.Object>, System.Object) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationToken.cs @ 192]
000000000eace950 000007fed33e9636 Microsoft.Extensions.Primitives.CompositeChangeToken.RegisterChangeCallback(System.Action`1<System.Object>, System.Object) [/_/src/Primitives/src/CompositeChangeToken.cs @ 48]
000000000eace9a0 000007fed33ec5d9 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 47]
000000000eace9e0 000007fed4791999 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\287\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 167]
000000000eacea60 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eaceaf0 000007fed33e992a Microsoft.Extensions.Primitives.CompositeChangeToken.OnChange(System.Object) [/_/src/Primitives/src/CompositeChangeToken.cs @ 117]
000000000eaceb50 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eacebe0 000007fed2e0c370 Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher+<>c.<.cctor>b__43_0(System.Object) [/_/src/FileProviders/Physical/src/PhysicalFilesWatcher.cs @ 27]
000000000eacec10 000007fed4791999 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\287\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 167]
000000000eacec90 000007fed484453a System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\Tasks\Task.cs @ 2440]
000000000eaced30 000007fed487c0d2 System.Threading.ThreadPoolWorkQueue.Dispatch() [E:\A\_work\287\s\src\mscorlib\src\System\Threading\ThreadPool.cs @ 588]
000000000eacf1b0 000007fed51aa673 [DebuggerU2MCatchHandlerFrame: 000000000eacf1b0]

bchavez · 2018-12-24T00:31:05Z

I think you might be on to something @rynowak.

Do you know where I could enable more tracing for the File Watcher? Perhaps we could get a trace of the file changes that are triggering recompilation.
Are there any significant changes to the File (System?) Watcher behavior between Windows 7 and Windows 10 that might have been overlooked?

Earlier I tried to set VSIISEXELAUNCHER_TRACEFOLDER for VSIISExeLauncher.exe but Visual Studio wasn't having it: https://twitter.com/bchavez/status/1076929078597283840

Also, I'll try setting up ProcMon to monitor the file paths in the directory to see if I can pickup anything during the autopilot memory ramp up.

pranavkm · 2018-12-24T00:40:20Z

Do you see the leak when you turn off file watching: https://github.com/aspnet/AspNetCore/blob/release/2.2/src/Mvc/src/Microsoft.AspNetCore.Mvc.Razor/RazorViewEngineOptions.cs#L198-L236? As of 2.2, it's disabled by default in non-Development environments.

bchavez · 2018-12-24T02:29:12Z

Hi @pranavkm ,

I added the following code before .AddMvc and I still see the issue.

Let me know if I should be configuring it a different way.

With AllowRecompilingViewsOnFileChange = false, it seems the behavior persists.

Here's a 5-min MP4 video: https://files.bitarmory.com/screen_2600.mp4

The video shows how long each edit takes which progressively gets worse on each edit. At the very bottom of the video, there's a white-background area, that's the browser/website for the Razor page.

By edit 14 and 15, ASPNET seems to start lagging behind.
Edit 15 - 20, ASPNET really lags behind.
Particularly interesting is when I get to edit 30 notice how long it takes for edit 31 to appear in the browser after refresh. By edit 31, I've basically lost control of ASPNET.

📗 Green gutter color indicates changes saved in VS:
📙 Orange gutter color indicates changes edited (but not saved yet) in VS:
Flash of ⬜ white, indicates a browser refresh.

bchavez · 2018-12-24T19:55:27Z

Are we sure change tokens are being registered (or disposed of) correctly? And that we don't have any kind of daisy chaining of change registrations? If we had daisy chained callbacks it would seem consistent with what I was experiencing as continued subsequent edits took more time to appear in the browser.

Here's what I'm seeing inside CancellationTokenSource:

https://github.com/dotnet/coreclr/blob/418e0949164eb91e73f7884387bcf9661016ba9f/src/mscorlib/src/System/Threading/CancellationTokenSource.cs#L615-L637

There seems to be a never ending list of node = node.Next in the for loop (of type CallbackNode) in which our special thread appears to be burning CPU on. This is consistent with the !DumpHeap -stat stats where +CallbackNode appears to be the largest allocation:

000007fe75792c10 15268935    610757400 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fe757929c0 15268942    977212288 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fed4b48868 15268958    977213312 System.Action
000007fed4b75b20 15268971    977214144 System.Action`1[[System.Object, System.Private.CoreLib]]
000007fed4b923c0 28440798   2275263840 System.Threading.CancellationTokenSource+CallbackNode

Specifically, when node.ExecuteCallback() is called inside CancellationTokenSource for loop, it takes us back here:

https://github.com/aspnet/Extensions/blob/15d50130655f3c8d28e10ace41d4de04acf9eaa8/src/Microsoft.Extensions.Primitives/ChangeToken.cs#L31-L50

Following the jump, ultimately, we land here which causes the re-computation of routes:

https://github.com/aspnet/AspNetCore/blob/85e2147ff037cc9950249fbb9d095ad47ec4184a/src/Http/Routing/src/RouteValuesAddressScheme.cs#L77-L87

Rinse and repeat. This is the hot path the thread is on.

Also, I breakpointed the file change events here and didn't notice anything abnormal popping off during automatic monotonically increasing memory usage.

https://github.com/aspnet/Extensions/blob/9bc79b2f25a3724376d7af19617c33749a30ea3a/src/FileProviders/Physical/src/PhysicalFilesWatcher.cs#L84-L88

I'll continue to do more debugging. 🪲 🐛 🔨

PS. Fun fact, the node.Id in the CancellationTokenSource for loop is counting backwards as the for loop progresses. 😎

NodeID: 1340668
NodeID: 1340667
NodeID: 1340666
NodeID: 1340665
NodeID: 1340664
NodeID: 1340663
NodeID: 1340662
...

🏖️ 🎺 Beach Boys - Good Vibrations (Nick Warren bootleg)

davidfowl · 2018-12-24T21:04:38Z

Oh wait I see a problem. That code is supposed to be called once ever. The underlying code re-registers for triggering on its own and this does it as well.

bchavez · 2018-12-25T00:36:43Z

Yea, pretty interesting behavior. Now that I have the for loop in CancellationTokenSource:ExecuteCallbackHandlers() wired up I'm getting some pretty interesting behavior.

Every iteration of the for loop, when node.Id % 1000 = 0, I print a message. It appears, that only saving the Razor file can cause the callback registrations to double up. Here's the output:

node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 4000
node.Id: 3000
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 8000
node.Id: 7000
node.Id: 6000
node.Id: 5000
node.Id: 4000
node.Id: 3000
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 16000
node.Id: 15000
node.Id: 14000
node.Id: 13000
node.Id: 12000
node.Id: 11000
node.Id: 10000
node.Id: 9000
node.Id: 8000
node.Id: 7000
node.Id: 6000
node.Id: 5000
node.Id: 4000
node.Id: 3000
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 32000
node.Id: 31000
node.Id: 30000
node.Id: 29000
node.Id: 28000
node.Id: 27000
node.Id: 26000
node.Id: 25000
node.Id: 24000
node.Id: 23000
node.Id: 22000
node.Id: 21000
node.Id: 20000
node.Id: 19000
node.Id: 18000
node.Id: 17000
node.Id: 16000
node.Id: 15000
node.Id: 14000
node.Id: 13000
node.Id: 12000
node.Id: 11000
node.Id: 10000
node.Id: 9000
node.Id: 8000
node.Id: 7000
node.Id: 6000
node.Id: 5000
node.Id: 4000
node.Id: 3000
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.

Kinda neat. The situation can get out of hand pretty quick. So far, I'm pretty satisfied we have enough diagnostic information for the issue now. Y'all keepin' my debug skills freshhh 🍉. 😎

I can finally put my mind at rest; gonna to take a few day off now to enjoy the holidays. 🎄 🎄 🎄 🎄

Happy Holidays friends,
Brian Chavez

🚶‍♂️ 🚶‍♀️ Missing Persons - Walking in L.A.

rynowak · 2018-12-27T05:05:24Z

Wow thanks for the investigation here Brian. That's definitely a bug.

RonerCodes · 2019-01-04T00:28:01Z

Just throwing this in here that I'm also having the same issue. My dotnet.exe reached 3 GB's of RAM after wondering for like 15 minutes why my cshtml edits weren't even making it to the browser (browser kept reloading a very old copy!). I learned a lot reading this thread. Keep it up guys!

deepumi · 2019-01-04T13:25:56Z

I have the same issue in .net core 3.0. I cannot see changes when I refresh the .cshtml page.. (Markup and javascript changes.). I need to re-run the project.

Visual Studio version - 2019 preview
.Net Core 3.0
Windows 10 professional latest

Thank you

The issue here was that every time a Razor Page changed, we would subscribe an additional time to the endpoint change notifications. This means that if you tweaked a page 30 times, we would update the address table 31 times when you save the file. If you were doing a lot of editing then this would grow to a really large amount of computation. The fix is to use DataSourceDependentCache, which is an existing utility type we developed for this purpose. I'm not sure why it wasn't being used for this already. We're already using DataSourceDependentCache in a bunch of other places, and it's well tested. I also tweaked the stucture of this code to be more similar to EndpointNameAddressScheme. This involved some test changes that all seemed like good cleanup. The way this was being tested was a little wonky.

* Fix #6102 - Intense CPU utilization on page change The issue here was that every time a Razor Page changed, we would subscribe an additional time to the endpoint change notifications. This means that if you tweaked a page 30 times, we would update the address table 31 times when you save the file. If you were doing a lot of editing then this would grow to a really large amount of computation. The fix is to use DataSourceDependentCache, which is an existing utility type we developed for this purpose. I'm not sure why it wasn't being used for this already. We're already using DataSourceDependentCache in a bunch of other places, and it's well tested. I also tweaked the stucture of this code to be more similar to EndpointNameAddressScheme. This involved some test changes that all seemed like good cleanup. The way this was being tested was a little wonky. (cherry picked from commit a5658a8)

abdulkareemnalband · 2019-04-03T12:03:21Z

Is this fix available on 2.2 branch?

poke · 2019-04-03T12:06:29Z

@abdulkareemnalband Change 3e5b37f is included since version v2.2.2.

Eilon added the area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates label Jan 2, 2019

rynowak self-assigned this Jan 9, 2019

rynowak added this to the 3.0.0-preview2 milestone Jan 9, 2019

rynowak mentioned this issue Jan 9, 2019

Port fix for #6102 to 2.2.2 #6543

Closed

rynowak closed this as completed in a5658a8 Jan 14, 2019

rynowak added bug This issue describes a behavior which is not expected - a bug. Done This issue has been fixed labels Jan 14, 2019

yberstad mentioned this issue Mar 13, 2019

Asp.net Core not Collecting Garbage #1976

Closed

Alan-Lun mentioned this issue Oct 12, 2019

VS2019 How to Build a Web API2 for .Net Framework 4.8 #14932

Closed

ghost locked as resolved and limited conversation to collaborators Dec 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory Leak in new ASPNET 2.2 routing? #6102

Memory Leak in new ASPNET 2.2 routing? #6102

bchavez commented Dec 22, 2018 •

edited

Loading

rynowak commented Dec 22, 2018

bchavez commented Dec 22, 2018

davidfowl commented Dec 22, 2018 •

edited

Loading

bchavez commented Dec 23, 2018

davidfowl commented Dec 23, 2018

davidfowl commented Dec 23, 2018 •

edited

Loading

davidfowl commented Dec 23, 2018

bchavez commented Dec 23, 2018 •

edited

Loading

rynowak commented Dec 23, 2018 •

edited

Loading

bchavez commented Dec 24, 2018

pranavkm commented Dec 24, 2018

bchavez commented Dec 24, 2018

bchavez commented Dec 24, 2018 •

edited

Loading

davidfowl commented Dec 24, 2018 •

edited

Loading

bchavez commented Dec 25, 2018 •

edited

Loading

rynowak commented Dec 27, 2018

RonerCodes commented Jan 4, 2019

deepumi commented Jan 4, 2019

abdulkareemnalband commented Apr 3, 2019

poke commented Apr 3, 2019

Memory Leak in new ASPNET 2.2 routing? #6102

Memory Leak in new ASPNET 2.2 routing? #6102

Comments

bchavez commented Dec 22, 2018 • edited Loading

Describe the bug

To Reproduce

Expected behavior

Additional context

Operating System

IDE

rynowak commented Dec 22, 2018

bchavez commented Dec 22, 2018

davidfowl commented Dec 22, 2018 • edited Loading

bchavez commented Dec 23, 2018

dotnet new webapp -o fff

EnableEndpointRouting = false:

davidfowl commented Dec 23, 2018

davidfowl commented Dec 23, 2018 • edited Loading

davidfowl commented Dec 23, 2018

bchavez commented Dec 23, 2018 • edited Loading

rynowak commented Dec 23, 2018 • edited Loading

bchavez commented Dec 24, 2018

pranavkm commented Dec 24, 2018

bchavez commented Dec 24, 2018

bchavez commented Dec 24, 2018 • edited Loading

davidfowl commented Dec 24, 2018 • edited Loading

bchavez commented Dec 25, 2018 • edited Loading

rynowak commented Dec 27, 2018

RonerCodes commented Jan 4, 2019

deepumi commented Jan 4, 2019

abdulkareemnalband commented Apr 3, 2019

poke commented Apr 3, 2019

bchavez commented Dec 22, 2018 •

edited

Loading

davidfowl commented Dec 22, 2018 •

edited

Loading

`dotnet new webapp -o fff`

`EnableEndpointRouting = false`:

davidfowl commented Dec 23, 2018 •

edited

Loading

bchavez commented Dec 23, 2018 •

edited

Loading

rynowak commented Dec 23, 2018 •

edited

Loading

bchavez commented Dec 24, 2018 •

edited

Loading

davidfowl commented Dec 24, 2018 •

edited

Loading

bchavez commented Dec 25, 2018 •

edited

Loading