Assert failure: !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n"

[BruceForstall]

System.Net.Requests.Tests and System.Security.Cryptography.Pkcs.Tests

Many jobs, x64/arm64/arm32

https://dev.azure.com/dnceng/public/_build/results?buildId=1232250&view=ms.vss-test-web.build-test-results-tab

+ grep COMPlus
COMPlus_TieredCompilation=0
COMPlus_DbgMiniDumpName=/home/helixbot/dotnetbuild/dumps/coredump.%d.dmp
COMPlus_DbgEnableMiniDump=1
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/C2180A37/p
----- start Mon Jul 12 08:04:14 UTC 2021 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/C2180A37/p/dotnet exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/C2180A37/w/B3D40A47/e /datadisks/disk1/work/C2180A37/w/B3D40A47/e
  Discovering: System.Net.Requests.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Requests.Tests (found 334 of 353 test cases)
  Starting:    System.Net.Requests.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.FtpWebRequestTest.Ftp_AppendFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_LargeFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_CreateAndDelete [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFileSubDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_MakeAndRemoveDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"

Assert failure(PID 24366 [0x00005f2e], Thread: 24381 [0x5f3d]): !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n" "Possible cause: you used new (executable) to allocate the memory, but didn't use DeleteExecutable() to free it."
    File: /__w/1/s/src/coreclr/utilcode/clrhost_nodependencies.cpp Line: 296
    Image: /datadisks/disk1/work/C2180A37/p/dotnet

Gathering state for process 24366 dotnet
Crashing thread 00005f3d signal 00000006
Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.24366.dmp
Written 271179776 bytes (66206 pages) to core file
Dump successfully written
./RunTests.sh: line 162: 24366 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/C2180A37/w/B3D40A47/e
----- end Mon Jul 12 08:04:38 UTC 2021 ----- exit code 134 ----------------------------------------------------------

[BruceForstall]

@mangod9

[mangod9]

If this is a consistent failure for specific test category, might make sense for the owners to investigate if there is a smaller repro and if any allocations/pinvokes are causing a corruption.

[mangod9]

@karelz can some from system.net please take an initial look at the failures to determine if they might be corruption due to pinvokes here?

[stephentoub]

Again here:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-55499-merge-641ae0c0c3d241e0ac/System.Net.Requests.Tests/console.f66f1627.log?sv=2019-07-07&se=2021-08-02T02%3A22%3A55Z&sr=c&sp=rl&sig=oSYpPpupPfXa2ifp7JLYN4pM1Uk0xdYo8h%2B4xSxau90%3D

[mangod9]

Adding @janvorli as well in case this is related to the w^x work, assuming this started happening in the past day or so.

[janvorli]

I've found the issue, I'll create a PR with a fix in a minute.

[karelz]

Thanks @janvorli! Out of curiosity, what caused the problems? Was it w^x work or some churn in Networking/crypto?

[janvorli]

W^X work, I've forgotten to replace one call to c++ delete by an appropriate executable heap method.

[janvorli]

And it was in a place that's executed when we allocate a data structure from multiple threads and only one of them wins so the others free what they've allocated. That's why it was intermittent and I haven't caught it locally.

[am11]

@janvorli, fyi, enabling event log with DOTNET_EnableEventLog=1 (or COMPlus_EnableEventLog=1) raises this heap contamination assertion, even with a simple 'corerun helloworld.dll'. corehost has no issue with event log.

[JulieLeeMSFT]

@janvorli, this assert happened in runtime-coreclr superpmi-collect pipeline, e.g., in System.Text.Json.Tests.JsonDocumentTests.CheckUnparsable(json: "[ 1") test. Could you please take a look?

[jkotas]

This assert is a general sign of unmanaged heap corruption that can have many different root causes.

This is crashing in the superpmi tool. It is different test and scenario than this issue tracked originally. I have opened a new issue for it: #101708 .