Version mismatch: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions'

[lorenzoiuri]

Bug Report

Symptom

Describe the bug
Instrumented pod crashes with error (note Version=6.0.0.0 and Version=3.1.0.0)

Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Uncaught exception during type initialization. (0x80131534)
File name: 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
 ---> System.TypeInitializationException: The type initializer for 'OpenTelemetry.AutoInstrumentation.Loader.Loader' threw an exception.
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at OpenTelemetry.Sdk.CreateTracerProviderBuilder()
   at OpenTelemetry.AutoInstrumentation.Instrumentation.Initialize() in /project/src/OpenTelemetry.AutoInstrumentation/Instrumentation.cs:line 67
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at OpenTelemetry.AutoInstrumentation.Loader.Loader.TryLoadManagedAssembly() in /project/src/OpenTelemetry.AutoInstrumentation.Loader/Loader.cs:line 72
   at OpenTelemetry.AutoInstrumentation.Loader.Loader..cctor() in /project/src/OpenTelemetry.AutoInstrumentation.Loader/Loader.cs:line 48
   --- End of inner exception stack trace ---
   at OpenTelemetry.AutoInstrumentation.Loader.Loader.AssemblyResolve_ManagedProfilerDependencies(Object sender, ResolveEventArgs args) in /project/src/OpenTelemetry.AutoInstrumentation.Loader/Loader.NetCore.cs:line 39
   at System.Runtime.Loader.AssemblyLoadContext.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name)
   at System.Runtime.Loader.AssemblyLoadContext.OnAssemblyResolve(RuntimeAssembly assembly, String assemblyFullName)
   at Microsoft.Extensions.Hosting.Host.CreateDefaultBuilder(String[] args)
   at Backend.API.Program.CreateHostBuilder(String[] args) in /builds/business/finopstool/code/webapp-costmanager-microservice/src/Backend.API/Program.cs:line 21
   at Backend.API.Program.Main(String[] args) in /builds/business/finopstool/code/webapp-costmanager-microservice/src/Backend.API/Program.cs:line 16

Expected behavior
The pod should not crash.

Screenshots
If applicable, add screenshots to help explain your problem.

Runtime environment (please complete the following information):

• OpenTelemetry Automatic Instrumentation version: 1.1.0
• OS: Linux Alpine musl
• .NET version: dotnet6

Additional context

We include the following dependencies in multiple subprojects

<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="6.0.0" />

We don't understand where the Version=3.1.0.0 is coming from: we include version 6.0.0 of Microsoft.Extensions.Hosting which is the only one which depends on Microsoft.Extensions.DependencyInjection.Abstractions, while the dotnet instrumentator depends on version 7.0.0: https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/main/src/Directory.Packages.props#L70.

We tried upgrading Microsoft.Extensions.Hosting from 6.0.0 to 7.0.0, with no changes.

We also tried including the OpenTelemetry.AutoInstrumentation NuGet packages, as described in the troubleshooting but the problem persists.

Edit: We are using the dotnet instrumentation via the opentelementry kubernetes operator.

Reproduce

Unfortunately we cannot provide the source code.

[Kielek]

@lorenzoiuri, I didn't have chance to start working on reproducing the issue. It will be great if you can create Minimal, Reproducible Example. It should not use your production code, but it should reproduce this in the mocked environment/scenario.

[Kielek]

@lorenzoiuri, I have created simple test application in Kielek@5ca51f1

Then executed build pipeline: https://github.com/Kielek/opentelemetry-dotnet-instrumentation/actions/runs/6824988834/job/18562149179 including Alpine. All tests passed.

Is there any chance that you create steps to reproduce as described in my previous message?

[lorenzoiuri]

Hi @Kielek, thank you for your support. We are working on the minimal example: it's not so easy because the code has many submodules, each importing nuget libraries.
Our efforts in recreating the bug bottom up (starting from a blank template) were not successful. We are now trying with a top down approach.

[lorenzoiuri]

Just to inform al those following this issue that I shared the minimal example with Kielek.

[Kielek]

@lorenzoiuri, please check the response in email. Please let me know if you are fine also to put shared code here, publicly.

[lorenzoiuri]

Hi Kielek, I received the reply and I'm trying to replicate your setup with instrument.sh. I'll let you known when we can publish the code. Thank you.

[federicobarera]

Same issue here. Where the project, or dependant projects, have direct dependencies on Microsoft.Extensions.* 6.0, we are having Could not load file or assembly issues. Adding OpenTelemetry.AutoInstrumentation nuget package does not solve the issue.

I am not familiar on how the injection mechanism work, but why a net6 project should hard reference Microsoft.Extensions 7.* packages?
Shouldn't be that depending on the target framework, the dependancy is >= [targetframework].*

[pjanotti]

The Could not load file or assembly issue typically happens in the following case: the AdditionalDeps moves the dll to a higher verion, let's say 7.0, however, the application has a dependency on an earlier version, let's say 6.0, and the early version is shipped with the application. At start up the runtime builds a list of the assemblies to be used by the app and for the dll in question the version from the application wins. However, when such dll is loaded the runtime reports an error because the dependencies that it built using AdditionalDeps + app dependencies should have loaded 7.0. If you remove from the app folder any dll for which you hit the error above you should be able to run your app with the instrumentation.

The Version=3.1.0.0 that is showing up is an artifact that the OpenTelemetry NuGet package that is built against that version, so its assembly reference is against 3.1.0.0 that's why it shows up in the error message. This is common for NuGet packages: they are built against the minimal version required.

The NuGet package, OpenTelemetry.AutoInstrumentation, is not intended to be used in conjunction with the AdditionalDeps and shared store. When using the NuGet package you should not have the DOTNET_ADDITIONAL_DEPS and DOTNET_SHARED_STORE configured and just use the instrument script to launch the application. In other words when using the NuGet package at build time, do not use the k8s operator to setup the instrumentation. One step that needs care when using the NuGet package is that the app should be build against the runtime that it is going to be running, e.g.: if you are going to run in Alpine but you are building on something else use a command like dotnet publish -c Release -r linux-musl-x64 --no-self-contained -o ../musl-dist/. If using the same platform to build and run the app you can simplify and use --use-current-runtime instead.

After all that said the usability for someone hitting this issue is not good. 🤔 We have to consider something less cumbersome path to get this working.

[Kielek]

@federicobarera, could you please create Minimal, Reproducible Example? It should include all steps you are using to build the package.

In the private channel, after a lot of attempts, we ware able to find the root cause of the issue described by @lorenzoiuri - including tests projects with dependencies to the folder with application.

[Kielek]

@federicobarera, I have started new issue for you #3168. Please put all details there.

I have received confirmation from @lorenzoiuri that the issue was related to the test dependencies. I am closing this issue.

[lorenzoiuri]

Hi, I just want to add that @Kielek suggested us to replace the commands:

dotnet build
dotnet publish -c release -o dist

with (example)

dotnet build .\src\Backend.API\Backend.API.csproj
dotnet publish .\src\Backend.API\Backend.API.csproj -c release -o dist

That solved the issue.

[pjanotti]

Nice finding @Kielek! The generated deps.json doesn't hint at that because it shows a chain of dependencies reaching the affected dlls. .NET 8 provides a warning for this type of publish:

warning NETSDK1194: The "--output" option isn't supported when building a solution.
Specifying a solution-level output path results in all projects copying outputs to the same directory,
which can lead to inconsistent builds.

Thanks @lorenzoiuri for providing the detailed repro.