OTLP exporter failed on .NET Framework

[muhaook]

Bug Report

Symptom

OTLP exporter failed on .Net framework. I found below exception in log:

[2022-12-19T20:04:59.3754190Z] [Error] EventSource=OpenTelemetry-Exporter-OpenTelemetryProtocol, Message=Unknown error in export method: System.MissingMethodException: Method not found: 'Google.Protobuf.ByteString Google.Protobuf.UnsafeByteOperations.UnsafeWrap(System.ReadOnlyMemory`1<Byte>)'.
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ActivityExtensions.ToOtlpSpan(Activity activity, SdkLimitOptions sdkLimitOptions)
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ActivityExtensions.AddBatch(ExportTraceServiceRequest request, SdkLimitOptions sdkLimitOptions, Resource processResource, Batch`1& activityBatch)
   at OpenTelemetry.Exporter.OtlpTraceExporter.Export(Batch`1& activityBatch) 

I am testing dotnet auto-instrumentation 0.5.1-beta.2. OTLP exporter on .Net works well.

as per my previous testing, OTLP exporter on 0.5.0 worked as well. so looks like this is a regression.

dotnet auto-instrumentation 0.5.1-beta.2 uses otel SDK 1.4.0 beta.3; while auto-instrumentation 0.5.0 uses otel SDK 1.3.1.

Runtime environment:

• OpenTelemetry Automatic Instrumentation version: 0.5.1-beta.2
• OS: Windows 10
• .NET version: .NET Framework 4.6.2

[pjanotti]

Hi @muhaook - we are working on an automated fix via #1646. for the time being, please, try the following workaround: add a binding redirect to the NuGet package System.Memory version 4.0.1.2 or higher, as done here.

For my reference are you using an old csproj style or the SDK csproj? The former is typically used for ASP.NET apps on .NET Framework.

[muhaook]

Thanks @pjanotti , I'll try the workaround.

Our sample is running on ASP.NET (Webform and MVC)+ WCF, the csproj is old style, not a SDK csproj.

[muhaook]

Hi @pjanotti , the workround works for most of my samples, but failed for one sample. The failed sample is hosted by IIS and targets at .net framework 4.6.2. Attached please find its web.config (changed to .txt so as to be able to upload)

Could not load file or assembly 'System.Memory' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

[FileLoadException: Could not load file or assembly 'System.Memory' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)]

[FileLoadException: Could not load file or assembly 'System.Memory, Version=4.0.1.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)]
   System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) +0
   System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) +481
   System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection) +128
   System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection) +22
   System.Reflection.Assembly.Load(String assemblyString) +34
   System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective) +48

[ConfigurationErrorsException: Could not load file or assembly 'System.Memory, Version=4.0.1.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)]
   System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective) +767
   System.Web.Configuration.CompilationSection.LoadAllAssembliesFromAppDomainBinDirectory() +256
   System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai) +58
   System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig) +287
   System.Web.Compilation.BuildManager.GetPreStartInitMethodsFromReferencedAssemblies() +70
   System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded) +137
   System.Web.Compilation.BuildManager.ExecutePreAppStart() +172
   System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +854

[HttpException (0x80004005): Could not load file or assembly 'System.Memory, Version=4.0.1.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)]
   System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +532
   System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +114
   System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +724

Web.config.txt

[Kielek]

@muhaook, could you please check https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/tag/v0.5.1-beta.3?

There is improvement related to loading different library version on .NET Framework. Please let us know if it help or no.

[pellared]

@muhaook please just make sure to use OpenTelemetry.DotNet.Auto.psm1 for installation as the script has some improvements as well (registers assemblies in GAC)

[muhaook]

Hi @Kielek @pellared , with v0.5.1-beta.3, I can not reproduce this issue, even without registering netfx assemblies in GAC.

I have 2 quick questions about registering / unregistering assemblies in GAC:

• Some assemblies below did not get added to GAC. is it because they are not strong named? any impact if they are not showing up in GAC?
  • netstandard.dll
  • grpc_csharp_ext.x64.dll
  • grpc_csharp_ext.x86.dll
• if I have an assembly already in GAC before installing dotnet auto-instrumentation, and this assembly is duplicated with an assembly under tracer-home/netfx/ folder. what will happen to it after installing dotnet auto-instrumentation? and what will happen to it after uninstalling auto-instrumentation? will it be removed from GAC by auto-instrumentation uninstall?

for info, I used this command to register assemblies in GAC:

        # Register .NET Framweworks dlls in GAC
        [System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a") | Out-Null
        $publish = New-Object System.EnterpriseServices.Internal.Publish 
        $dlls = Get-ChildItem -Path .\netfx\ -Filter *.dll -File
        foreach ($dll in $dlls) {
            $publish.GacInstall($dll.FullName)
        }

and I used command below to unregister assemblies from GAC:

    [System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a") | Out-Null
    $publish = New-Object System.EnterpriseServices.Internal.Publish 
    $dlls = Get-ChildItem -Path .\netfx\ -Filter *.dll -File
    foreach ($dll in $dlls) {
        $publish.GacRemove($dll.FullName)
    }

before auto-instrument install, my GAC:
gac.1.txt
after auto-instrument install, my GAC:
gac.2.txt

[Kielek]

@muhaook, closing this one.
GAC question moved to #1906

[ClumsyPenguin]

Same issue on .NET Framework 4.8

Installed OTel packages:
OpenTelemetry V1.6.0
OpenTelemetry.Api V1.6.0
OpenTelemetry.Api.ProviderBuilderExtensions V1.6.0
OpenTelemetry.Exporter.Console V1.6.0
OpenTelemetry.Exporter.OpenTelemetryProtocol_V1.6.0_
OpenTelemetry.Extensions.Hosting V1.6.0
OpenTelemetry.Instrumentation.AspNet V1.6.0-beta.1
OpenTelemetry.Instrumentation.AspNet.TelemetryHtppModule V1.6.0-beta.1
OpenTelemetry.Instrumentation.Http V1.5.1-beta.1
OpenTelemetry.Instrumentation.SqlClient V1.5.1-beta.1

My GRPC packages are:
Grpc.Core.Api V2.46.6
Grpc V2.46.6
Grpc.Core V2.46.6

System.IO.FileNotFoundException: Error loading native library. Not found in any of the possible locations: C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Grpc.Core\v4.0_2.0.0.0__d754f35622e28bad\grpc_csharp_ext.x64.dll,C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Grpc.Core\v4.0_2.0.0.0__d754f35622e28bad\runtimes/win-x64/native\grpc_csharp_ext.x64.dll,C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\Grpc.Core\v4.0_2.0.0.0__d754f35622e28bad\../..\runtimes/win-x64/native\grpc_csharp_ext.x64.dll
   at Grpc.Core.Internal.UnmanagedLibrary.FirstValidLibraryPath(String[] libraryPathAlternatives)
   at Grpc.Core.Internal.UnmanagedLibrary..ctor(String[] libraryPathAlternatives)
   at Grpc.Core.Internal.NativeExtension.LoadNativeMethodsUsingExplicitLoad()
   at Grpc.Core.Internal.NativeExtension.LoadNativeMethods()
   at Grpc.Core.Internal.NativeExtension..ctor()
   at Grpc.Core.Internal.NativeExtension.Get()
   at Grpc.Core.Internal.NativeMethods.Get()
   at Grpc.Core.GrpcEnvironment.GrpcNativeInit()
   at Grpc.Core.GrpcEnvironment..ctor()
   at Grpc.Core.GrpcEnvironment.AddRef()
   at Grpc.Core.Channel..ctor(String target, ChannelCredentials credentials, IEnumerable`1 options)
   at Grpc.Core.Channel..ctor(String target, ChannelCredentials credentials)
   at OpenTelemetry.Exporter.OtlpExporterOptionsExtensions.CreateChannel(OtlpExporterOptions options)
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.OtlpGrpcTraceExportClient..ctor(OtlpExporterOptions options, TraceServiceClient traceServiceClient)
   at OpenTelemetry.Exporter.OtlpExporterOptionsExtensions.GetTraceExportClient(OtlpExporterOptions options)
   at OpenTelemetry.Exporter.OtlpTraceExporter..ctor(OtlpExporterOptions exporterOptions, SdkLimitOptions sdkLimitOptions, IExportClient`1 exportClient)
   at OpenTelemetry.Trace.OtlpTraceExporterHelperExtensions.BuildOtlpExporterProcessor(OtlpExporterOptions exporterOptions, SdkLimitOptions sdkLimitOptions, IServiceProvider serviceProvider, Func`2 configureExporterInstance)
   at OpenTelemetry.Trace.OtlpTraceExporterHelperExtensions.<>c__DisplayClass2_0.<AddOtlpExporter>b__1(IServiceProvider sp)
   at OpenTelemetry.Trace.TracerProviderBuilderExtensions.<>c__DisplayClass8_0.<AddProcessor>b__0(IServiceProvider sp, TracerProviderBuilder builder)
   at OpenTelemetry.Trace.OpenTelemetryDependencyInjectionTracingServiceCollectionExtensions.ConfigureTracerProviderBuilderCallbackWrapper.ConfigureBuilder(IServiceProvider serviceProvider, TracerProviderBuilder tracerProviderBuilder)
   at OpenTelemetry.Trace.TracerProviderSdk..ctor(IServiceProvider serviceProvider, Boolean ownsServiceProvider)
   at OpenTelemetry.Trace.TracerProviderBuilderBase.Build()
   at OpenTelemetry.Trace.TracerProviderBuilderBase.InvokeBuild()
   at OpenTelemetry.Trace.TracerProviderBuilderExtensions.Build(TracerProviderBuilder tracerProviderBuilder)

That workaround mentioned here does not work for me unfortunately.