Plagued with 'System.OutOfMemoryException' errors in .NET 7 Preview 4

[littlehoneybunnytuttifruttipumpkinpie]

I'm running a basic but memory & CPU intensive .NET 7 app with an API, on a Web Server 2016, but after it runs for a while, I'm running into continual 'System.OutOfMemoryException' errors in various parts of the program, especially as the memory usage grows, but not sure if this is related to max RAM usage as it's less than but almost 95% full (works fine with .NET 6 and same & higher RAM usage).
The issue happens with .NET 7 Preview 4, while same program works fine with .NET 6. I have not tested previous .NET 7 preview versions.
The errors seem to appear in several/different parts of the program, each performing variety of functions unrelated to each other.
The errors are also repeating (in similar parts of the program as previously) after restarting the program and after it runs for a while, filling up memory.
The program mainly loads data from Zip files, uncompresses it and stores it in static Dictionary to be served via API. It does the same operation in a loop, so no difference in behavior over time, just the memory errors happen after a while and after RAM usage is increasing).

Some of the errors are showing as follows:

[error: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.SplitWithoutPostProcessing(ReadOnlySpan`1 sepList, ReadOnlySpan`1 lengthList, Int32 defaultLength, Int32 count)
   at System.String.SplitInternal(ReadOnlySpan`1 separators, Int32 count, StringSplitMyData MyData)...

[error: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.Ctor(ReadOnlySpan`1 value)
   at System.Span`1.ToString()
   at System.Text.ValueStringBuilder.ToString()
   at System.Number.FormatDouble(Double value, String format, NumberFormatInfo info)
   ...

[error: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Utils.Sys.CopyTo(Stream src, Stream dest) in ...

At least one example of the issue happens on the following line of code that operates on a Stream:
while ((cnt = src.Read(bytes, 0, bytes.Length)) != 0) dest.Write(bytes, 0, cnt);

Deployment machine:

• .NET 7 Preview 4
• Windows Server 2016
• Intel Xeon E5-2697 (x2)

(unable to test on other hardware at this time due to large RAM usage that I don't have elsewhere)
Not sure how to help debug this issue, while for now I'm going back to .NET 6 just to be able to run the program properly.

[littlehoneybunnytuttifruttipumpkinpie]

Update:
I was able to create reproducible scenario with a basic C# console program that results in similar (one of) 'OutOfMemoryException' errors:

The program basically runs and fills memory half-full before crashing on this line of code:
Test.static_string_dict[loop] = new string(test_string);

Note that my main program that I originally reported as crashing, crashes at memory about 90% full, so I don't think "half-full" memory is relevant, it just crashes at some point.
The specific line of code may also not be relevant, as my main program crashes on different/various lines of code, unrelated to this test.

When running again alongside .NET 6 version, only the .NET 7 version crashed while .NET 6 version continued running fine:

Here is the full code of the reproducible program:

using System.Text;

namespace MemTest
{
    internal class Test
    {
        internal static Dictionary<long, string> static_string_dict = new Dictionary<long, string>();
    }

    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                Dictionary<long, string> string_dict = new Dictionary<long, string>();

                GCMemoryInfo comp_info = GC.GetGCMemoryInfo();
                double ram_divider = 1024 * 1024 * 1024; //GB
                double ram_total = comp_info.TotalAvailableMemoryBytes / ram_divider;

                long loop = 0;
                string test_string = CreateTestString(100000000);

                for (; ; )
                {
                    int tick_count = Environment.TickCount;

                    loop++;
                    string_dict[loop] = new string(test_string);
                    Test.static_string_dict[loop] = new string(test_string);

                    Console.WriteLine("Loop completed in {0} ms.", Environment.TickCount - tick_count);

                    GC.Collect();
                    GC.WaitForPendingFinalizers();
                    //Task.Delay(5000).Wait();
                    comp_info = GC.GetGCMemoryInfo();
                    double ram_used = comp_info.MemoryLoadBytes / ram_divider;

                    Console.WriteLine("Used memory: {0} of {1} GB.", ram_used.ToString("0.##"), ram_total.ToString("0.##"));

                    //Clear RAM when over 98% full
                    if (ram_total != 0 && ram_used / ram_total >= 0.98)
                    {
                        Console.WriteLine("Memory nearly full. Clearing data.");
                        loop = 0;
                        string_dict = new Dictionary<long, string>();
                        Test.static_string_dict = new Dictionary<long, string>();

                        GC.Collect();
                        GC.WaitForPendingFinalizers();
                        //break;
                    }
                }

                Console.WriteLine("Program completed");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error: {ex.Message}\nStack:\n{ex.StackTrace}");
            }

            Console.WriteLine("Press any key to exit");
            Console.ReadKey();
        }


        static string CreateTestString(int components)
        {
            //creates custom strings for additional testing of compression not used in memory testing
            StringBuilder builder = new StringBuilder();
            for (int i = 1; i <= components; i++)
            {
                builder.Append(i).Append(',');
            }

            return builder.ToString();
        }

    }
}

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I'm running a basic but memory & CPU intensive .NET 7 app with an API, on a Web Server 2016, but after it runs for a while, I'm running into continual 'System.OutOfMemoryException' errors in various parts of the program, especially as the memory usage grows, but not sure if this is related to max RAM usage as it's less than but almost 95% full (works fine with .NET 6 and same & higher RAM usage).
The issue happens with .NET 7 Preview 4, while same program works fine with .NET 6. I have not tested previous .NET 7 preview versions.
The errors seem to appear in several/different parts of the program, each performing variety of functions unrelated to each other.
The errors are also repeating (in similar parts of the program as previously) after restarting the program and after it runs for a while, filling up memory.
The program mainly loads data from Zip files, uncompresses it and stores it in static Dictionary to be served via API. It does the same operation in a loop, so no difference in behavior over time, just the memory errors happen after a while and after RAM usage is increasing).

Some of the errors are showing as follows:

[error: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.SplitWithoutPostProcessing(ReadOnlySpan`1 sepList, ReadOnlySpan`1 lengthList, Int32 defaultLength, Int32 count)
   at System.String.SplitInternal(ReadOnlySpan`1 separators, Int32 count, StringSplitMyData MyData)...

[error: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.Ctor(ReadOnlySpan`1 value)
   at System.Span`1.ToString()
   at System.Text.ValueStringBuilder.ToString()
   at System.Number.FormatDouble(Double value, String format, NumberFormatInfo info)
   ...

[error: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Utils.Sys.CopyTo(Stream src, Stream dest) in ...

At least one example of the issue happens on the following line of code that operates on a Stream:
while ((cnt = src.Read(bytes, 0, bytes.Length)) != 0) dest.Write(bytes, 0, cnt);

Deployment machine:

• .NET 7 Preview 4
• Windows Server 2016
• Intel Xeon E5-2697 (x2)

(unable to test on other hardware at this time due to large RAM usage that I don't have elsewhere)
Not sure how to help debug this issue, while for now I'm going back to .NET 6 just to be able to run the program properly.

Author:	littlehoneybunnytuttifruttipumpkinpie
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[mangod9]

We have enabled GC regions functionality in preview3 which has changed some heuristics. Are you observing this behavior on machines with large available memory > 256gb or with lower available memory too?

Will try to repro..

[mangod9]

@littlehoneybunnytuttifruttipumpkinpie could you please try your repro after setting COMPlus_GCRegionRange=10700000000 increases the region reserved range to 1TB. By default it only reserves 256gb which might be the reason you are getting OOMs on large memory machines.

[mangod9]

Closing this based on the fix above. @littlehoneybunnytuttifruttipumpkinpie please validate on preview7 to check if its indeed fixed.