Proposal: PMI asm diffs using SuperPMI

[BruceForstall]

We now have jit-diff diff --pmi to use PMI for generating assembly code across a specified corpus of code, and analyzing the diffs assuming both baseline and diff compiler asm code is generated. This proposal is to introduce a mode to user SuperPMI to achieve the same results, but hopefully faster, eliminating incorrect diffs, and allowing for easier re-JIT of interesting methods.

I propose we add a --superpmi option, such that you would use jit-diff diff --pmi --superpmi. This specifies to still use PMI, but to accelerate it using SuperPMI. In this mode, you would be required to specify both a baseline and diff compiler (SuperPMI asm diffs require both). These steps would be executed:

• Run PMI over the specified corpus of code using the baseline compiler, in SuperPMI collection mode. We would not be generating textual asm output during this run.
• Do all the MC file cleanup and merging to create a single CODE.MCH file.
• Run SuperPMI in asm diff mode, passing both the baseline and diff JIT, and the CODE.MCH file, generating a DIFF.MCL file identifying all the functions that differ.
• If there are any diffs, generate textual ASM output (using COMPlus_JitDisasm) by running SuperPMI over just the DIFF.MCL methods, once for the baseline JIT, and once for the diff JIT.
• Run jit-analyze over the resulting ASM, to generate the diff statistics.

The resulting CODE.MCH and DIFF.MCL can remain for subsequent re-use and further investigations.

Note that we could also have a jit-diff diff --superpmi (without --pmi) for doing a similar thing but for crossgen-based asm diffs.

@dotnet/jit-contrib

[CarolEidt]

Sounds great to me!

[jashook]

I think that this can be improved to share the resulting mch files more broadly. I also believe that it can be can be generalized such that pmi is not the only aspect of the collection to run diffs on.

I can instead image a workflow like:

CI Workflow

1. Run SuperPMI collection (xarch)
   • superpmi collect - runtest.sh x64 checked
   • superpmi collect - pmi assemblies
   • superpmi collect - jitbench
2. Public resulting .mchs to* :
   • https://clrjit.blob.core.windows.net/superpmi/osx/x64/checked/latest-collection.mch
   • https://clrjit.blob.core.windows.net/superpmi/Linux/x64/checked/latest-collection.mch
   • https://clrjit.blob.core.windows.net/superpmi/Linux/armel/checked/latest-collection.mch
   • https://clrjit.blob.core.windows.net/superpmi/Linux/armhf/checked/latest-collection.mch
   • https://clrjit.blob.core.windows.net/superpmi/Linux/arm64/checked/latest-collection.mch
   • https://clrjit.blob.core.windows.net/superpmi/Windows_NT/x64/checked/latest-collection.mch
   • https://clrjit.blob.core.windows.net/superpmi/Windows_NT/x86/checked/latest-collection.mch
   • https://clrjit.blob.core.windows.net/superpmi/Windows_NT/arm/checked/latest-collection.mch
   • https://clrjit.blob.core.windows.net/superpmi/Windows_NT/arm64/checked/latest-collection.mch

Dev Workflow

1. Run jit-diff (or some tool) --superpmi arch build_type base jit diff jit
   • Pull down the latest-collection.mch
   • Run SuperPMI in asm diff mode, passing both the baseline and diff JIT, and the CODE.MCH file, generating a DIFF.MCL file identifying all the functions that differ.
   • If there are any diffs, generate textual ASM output (using COMPlus_JitDisasm) by running SuperPMI over just the DIFF.MCL methods, once for the baseline JIT, and once for the diff JIT.
   • Run jit-analyze over the resulting ASM, to generate the diff statistics.

*(hypothetical links) they do not go anywhere.

[jashook]

Note that either proposal requires coredistools to be ported to:

1. linux arm64
2. linux arm32
3. OSX
4. Windows arm32
5. Windows arm64

[BruceForstall]

@jashook These two proposals can be complementary. I would like to have the CI generate and store superpmi collections across multiple platforms/architectures, and have the dev workflow pull down the appropriate one for use in superpmi replay for assert checking or diffs. That proposal involves more moving parts, and working with the CI. My proposal would probably require building basic tooling in jitutils that could be leveraged for both cases, and only depends on the local machine, with no CI interaction. So hopefully it would be quicker to build and more robust.

Perhaps a jitutils tool that should come out of this is a "jit-spmi collect" tool that would collect whatever you want and put it in some file -- COLLECTION.MCH, say. Then "jit-spmi replay" could do assert checking using this, and "jit-spmi diff" or "jit-diff diff --superpmi" or something would use it for asm diffs.

[AndyAyersMS]

Some current, useful jit-diff functionality that might require extra work if we leverage SuperPMI:

• Reporting metric changes as a percentage of the metric for all methods.
• Reporting on special metrics (e.g. PerfScore).
• Summarizing metrics by assembly.

[BruceForstall]

Reporting metric changes as a percentage of the metric for all methods.

I've thought we needed to add that capability to SuperPMI asm diffs for a long time. For something like code size, where SuperPMI can itself compute the metric, it seems not too difficult. For metrics where only the JIT can compute the metric, we perhaps to teach the JIT how to report any/all metrics back to the runtime through the the JIT/EE interface, which would allow SuperPMI to capture them for subsequent use.