Run ILVerify on unit tests

[jonorossi]

We've been running PEVerify on the output of unit tests on Windows for .NET Framework builds for a very long time. Microsoft has been building Microsoft.DotNet.ILVerification under the CoreRT project, but that effort has now moved into the .NET runtime Git repo and looks to be coming out in .NET 5 as ILVerify and a library. It would probably be a good time to test out the tool to see if DP is in good shape and the tool doesn't have false positives, and ready us for running this tool for .NET Framework and Mono.

Right now there is no package on nuget.org, but there are builds on another feed as documented in the readme:

• https://github.com/dotnet/runtime/tree/master/src/coreclr/src/tools/ILVerify
• Move ILVerify tool to dotnet/runtime repo dotnet/runtime#13827

You can run this, but we'd probably want to use the library:

dotnet ilverify hello.dll -r "c:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.1.12\*.dll"

[stakx]

@jonorossi, I may be missing something, but ILVerify looks a little... unfinished. I don't expect that it can fully replace PEVerify in the near future, but I'll give it a try, and keep an eye on it.

[jonorossi]

I looked at the VerifierError enum and (wrongly?) assumed that was the full list from PEVerify, with only a few still commented out.

[stakx]

Gave ILVerify another go. It is now available as a dotnet tool that can be installed with

 > dotnet tool install --global dotnet-ilverify

Then you should be able to run it simply by invoking ilverify on the command line.

First problem is that ILVerify doesn't appear to be able to locate mscorlib:

> ilverify --verbose CastleDynProxy2.dll

Using input file '...\bin\Debug\net462\CastleDynProxy2.dll'

Error: Internal.IL.VerifierException: Assembly or module not found: mscorlib
   at ILVerify.Verifier.SetSystemModuleName(AssemblyName name)
   at ILVerify.Program.Run()
   at ILVerify.Program.Run(Options options)

Which is understandable given that the tool was written with .NET Core in mind, where there's no such thing as the GAC. But we'd be running the tool for net462 assemblies, since that's currently the only targeted platform that supports saving assemblies to disk... so this is unfortunate.

Providing the explicit path of the GAC's copy of mscorlib.dll, I got ILVerify to run a little further... but it still crashes:

> ilverify --reference "C:\Windows\Microsoft.NET\assembly\GAC...\mscorlib.dll" --verbose CastleDynProxy2.dll

Using input file '...\bin\Debug\net462\CastleDynProxy2.dll'

Using reference file 'C:\Windows\Microsoft.NET\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'

Verifying [DynamicProxyGenAssembly2]Castle.Proxies.ControlProxy..cctor
Error: System.NullReferenceException: Object reference not set to an instance of an object.
   at ILVerify.ILVerifyTypeSystemContext.VerifyModuleName(String simpleName, EcmaModule module)
   at ILVerify.ILVerifyTypeSystemContext.ResolveAssemblyOrNetmodule(String simpleName, String verificationName, IAssemblyDesc containingAssembly, Boolean throwIfNotFound)
   at ILVerify.ILVerifyTypeSystemContext.ResolveAssembly(AssemblyName name, Boolean throwIfNotFound)
   at Internal.TypeSystem.Ecma.EcmaModule.ResolveAssemblyReference(AssemblyReferenceHandle handle)
   at Internal.TypeSystem.Ecma.EcmaModule.EcmaObjectLookupHashtable.CreateValueFromKey(EntityHandle handle)
   at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey key)
   at Internal.TypeSystem.Ecma.EcmaModule.GetObject(EntityHandle handle, NotFoundBehavior notFoundBehavior)
   at Internal.TypeSystem.Ecma.EcmaModule.ResolveTypeReference(TypeReferenceHandle handle)
   at Internal.TypeSystem.Ecma.EcmaModule.EcmaObjectLookupHashtable.CreateValueFromKey(EntityHandle handle)
   at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey key)
   at Internal.TypeSystem.Ecma.EcmaModule.ResolveMemberReference(MemberReferenceHandle handle)
   at Internal.TypeSystem.Ecma.EcmaModule.EcmaObjectLookupHashtable.CreateValueFromKey(EntityHandle handle)
   at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey key)
   at Internal.TypeSystem.Ecma.EcmaModule.GetObject(EntityHandle handle, NotFoundBehavior notFoundBehavior)
   at Internal.IL.EcmaMethodIL.GetObject(Int32 token, NotFoundBehavior notFoundBehavior)
   at Internal.IL.ILImporter.ResolveToken(Int32 token)
   at Internal.IL.ILImporter.ImportLdToken(Int32 token)
   at Internal.IL.ILImporter.ImportBasicBlock(BasicBlock basicBlock)
   at Internal.IL.ILImporter.ImportBasicBlocks()
   at Internal.IL.ILImporter.Verify()
   at ILVerify.Verifier.VerifyMethod(EcmaModule module, MethodIL methodIL, MethodDefinitionHandle methodHandle)
   at ILVerify.Verifier.VerifyMethods(EcmaModule module, IEnumerable`1 methodHandles)+MoveNext()
   at ILVerify.Verifier.Verify(PEReader peReader, MethodDefinitionHandle methodHandle)+MoveNext()
   at ILVerify.Program.VerifyMethods(PEReader peReader, EcmaModule module, String path, Int32& numErrors, Int32& verifiedMethodCounter, Int32& methodCounter)
   at ILVerify.Program.VerifyAssembly(PEReader peReader, EcmaModule module, String path)
   at ILVerify.Program.Run()
   at ILVerify.Program.Run(Options options)

Could be related to the first problem, in either case: getting this to fly probably isn't going to be trivial. 🙂