[wasm] WBT test failure - SimpleSourceChangeRebuildTest.SimpleStringChangeInSource

[radical]

Failures 4/20-8/5 (incl. PRs):

Regression on 8/4 introduced in PR #72394

Day	Failures
8/5	6x PR (24x failures)
8/4	2x Official run + 6x PR (32x failures total)
7/19-8/2	1x PR #72394 (22x failures)

Fails on Windows and Linux.

Example: Rolling build, and log.

    Wasm.Build.NativeRebuild.Tests.SimpleSourceChangeRebuildTest.SimpleStringChangeInSource(buildArgs: BuildArgs { ProjectName = placeholder, Config = Release, AOT = True, ProjectFileContents = placeholder, ExtraBuildArgs =  }, nativeRelink: False, invariant: False, host: Chrome, id: "kfqswgdd.hbi") [FAIL]
      CompareStat failed:
      [Expected unchanged file: System.Private.CoreLib.dll.bc]
         old: FileStat { Exists = True, LastWriteTimeUtc = 8/4/2022 9:01:26 PM, Length = 16570448, FullPath = /datadisks/disk1/work/AEEC0971/w/AF8A095A/e/kfqswgdd.hbi/obj/Release/net7.0/browser-wasm/wasm/for-publish/System.Private.CoreLib.dll.bc }
         new: FileStat { Exists = True, LastWriteTimeUtc = 8/4/2022 9:02:39 PM, Length = 16570448, FullPath = /datadisks/disk1/work/AEEC0971/w/AF8A095A/e/kfqswgdd.hbi/obj/Release/net7.0/browser-wasm/wasm/for-publish/System.Private.CoreLib.dll.bc }

Some of the source assemblies are getting aot'ed again, when rebuilding with no changes. This seems to have been introduced in #72394 which disabled caching:

runtime/src/tasks/Common/FileCache.cs

Line 32 in bf362e8

//Enabled = true;

It didn't get caught because the WBT tests were failing to build because of #73312 .

let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('', 'SimpleStringChangeInSource', true, 'CompareStat failed', true);

cc @steveisok @mdh1418

[ghost]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Rolling build, and log.

    Wasm.Build.NativeRebuild.Tests.SimpleSourceChangeRebuildTest.SimpleStringChangeInSource(buildArgs: BuildArgs { ProjectName = placeholder, Config = Release, AOT = True, ProjectFileContents = placeholder, ExtraBuildArgs =  }, nativeRelink: False, invariant: False, host: Chrome, id: "kfqswgdd.hbi") [FAIL]
      CompareStat failed:
      [Expected unchanged file: System.Private.CoreLib.dll.bc]
         old: FileStat { Exists = True, LastWriteTimeUtc = 8/4/2022 9:01:26 PM, Length = 16570448, FullPath = /datadisks/disk1/work/AEEC0971/w/AF8A095A/e/kfqswgdd.hbi/obj/Release/net7.0/browser-wasm/wasm/for-publish/System.Private.CoreLib.dll.bc }
         new: FileStat { Exists = True, LastWriteTimeUtc = 8/4/2022 9:02:39 PM, Length = 16570448, FullPath = /datadisks/disk1/work/AEEC0971/w/AF8A095A/e/kfqswgdd.hbi/obj/Release/net7.0/browser-wasm/wasm/for-publish/System.Private.CoreLib.dll.bc }

Some of the source assemblies are getting aot'ed again, when rebuilding with no changes. This seems to have been introduced in #72394 which disabled caching:

runtime/src/tasks/Common/FileCache.cs

Line 32 in bf362e8

//Enabled = true;

It didn't get caught because the WBT tests were failing to build because of #73312 .

cc @steveisok @mdh1418

Author:	radical
Assignees:	radical
Labels:	arch-wasm, blocking-clean-ci, area-Build-mono, test-failure
Milestone:	-

[karelz]

Thanks @radical for quick fix! I removed blocking-clean-ci label, assuming the problem is now gone.