An assembly specified in the application dependencies manifest was not found

[d-aguilera]

Bug Report

Symptom

When starting the "demo" Docker example, the Client and Server containers immediately stop with the following error:

An assembly specified in the application dependencies manifest (OpenTelemetry.AutoInstrumentation.AdditionalDeps.deps.json) was not found
  package: 'DnsClient', version: '1.4.0'
  path: 'lib/netstandard2.1/DnsClient.dll'

Expected behavior
All containers should start successfully.

Screenshots

Runtime environment:

• OpenTelemetry Automatic Instrumentation version: 0.5.0
• OS: Mac OS Monterey 12.6.2, M1 Pro, 32 GB RAM
• Docker Desktop 4.16.2 (95914)
• .NET version: defined by the Docker image

Additional context
In addition to getting this error with the Docker example running on my Mac, I also got this very same error on a Windows 11 Home VM running on Parallels on the Mac. In that case I used the Powershell script as instructed, and got the error after launching the app's exe.
During this attempt in Windows, I was able to workaround the issue by manually copying each and every DLL from C:\Program Files\OpenTelemetry .NET AutoInstrumentation\store\x64\net6.0 to the directory where the .exe was. So maybe this is just some environment variable missing? Because it seems it cannot resolve the location of these assemblies in the shared store (just guessing).

Reproduce

1. Enter make command
2. Wait until docker-compose downloads all images and starts all containers
3. Soon after that, 3 containers stop:

• sqlserver-1 ==> The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
• service-1 & client-1 ==> showing the error above.

I understand the SQL Server image error can be fixed by using a different image maybe. It's the .NET error that I'm concerned about.
Also I didn't found any OpenTelemetry log files in /var/log/opentelemetry/dotnet/

[d-aguilera]

About the SQL Server issue, I tried with Azure SQL Edge and it seems to work. Not sure if the app will be able to connect to it though but at least the container is up and running.

[d-aguilera]

I also tried with 6.0-jammy-arm64v8 for Client and Server, but it made no difference.

[d-aguilera]

More info:
When I remove

    env_file:
      - otel-dotnet.env # enables OpenTelemetry .NET Automatic Instrumentation

from docker-compose.yaml, both client and service start up and work correctly.
I know, these env vars are the whole purpose of the demo, but I just wanted to confirm the apps were able to run without instrumentation.

[d-aguilera]

[d-aguilera]

The readme says self-contained deployments are incompatible with automatic instrumentation, but I wanted to give it a try anyway to see if at least I could solve the missing dependencies problem this way. So I tried
dotnet publish -c Release -o out -r linux-arm64
and, as expected, it produced working Client and Service apps.

However, to my surprise, I'm seeing SQL traces in Grafana now, which I understand are automatically generated traces, correct?
Does this mean it was able to instrument a self-contained build?

[pellared]

I also tried with 6.0-jammy-arm64v8 for Client and Server, but it made no difference.

ARM is not supported. We tried it but we failed #1869

[pellared]

Thanks for reporting the problems that you have. Probably this issue has the same root cause as #1744 (comment)

[pellared]

The readme says self-contained deployments are incompatible with automatic instrumentation, but I wanted to give it a try anyway to see if at least I could solve the missing dependencies problem this way. So I tried dotnet publish -c Release -o out -r linux-arm64 and, as expected, it produced working Client and Service apps.

However, to my surprise, I'm seeing SQL traces in Grafana now, which I understand are automatically generated traces, correct? Does this mean it was able to instrument a self-contained build?

You are correct. Self-contained apps should be working. It is some other issue that we are trying to identify. See #2131 (comment)

I think we will fix the docs when we will better spot the issue.

[pellared]

Just a note that I was unable to reproduce the issue locally. I even tried

	docker-compose build --no-cache
	docker-compose up -d

I think this issue is not deterministic.

[rajkumar-rangaraj]

d-aguilera Could you please provide the following, it will help in troubleshooting an issue.

1. Get the complete list of environment variables from the problematic environment. e.g.,: printenv
2. Directory listing of OpenTelemetry Automatic Instrumentation package from the environment where the failure happens. e.g.,: tree opentelemetry-dotnet-instrumentation-linux-glibc or ls -R

[d-aguilera]

Here's the requested information:

Client:

• printenv
• cd /app; tree .

Service

• printenv
• cd /app; tree .

Common (both apps identical output):

• cd $OTEL_DOTNET_AUTO_HOME; tree .

[pjanotti]

I think this issue is not deterministic.

Perhaps because of this. However, this issue has some other specific steps where this may not be applicable.

[pjanotti]

@d-aguilera are you still getting the An assembly specified in the application dependencies manifest error as above? If that is the case could you please try the command that is failing with the following environment variables set:

COREHOST_TRACE=1
COREHOST_TRACEFILE=corehost_verbose_tracing.log
COREHOST_TRACE_VERBOSITY=4

these env vars are documented here. Running the command with these env vars should create the log file with some info to dig deeper into this.

[d-aguilera]

Got it @pjanotti, here's the log file:

corehost_verbose_tracing.log

[d-aguilera]

I see the following at line 115 of corehost_verbose_tracing.log:

-- arguments_t: env shared store: '/otel-dotnet-auto/store/arm64/net6.0'

This directory doesn't exist in the container.
Only x64/net6.0 and x86/net6.0 exist within /otel-dotnet-auto/store.
Maybe this is the reason why it fails to find the assemblies?

[pjanotti]

Well observed @d-aguilera this seems a good candidate for the issue. The arch/tfm are added by the host so as a cheap validation could you please rename /otel-dotnet-auto/store/x64/net6.0 to /otel-dotnet-auto/store/arm64/net6.0? The assemblies under the folder are MSIL so this shouldn't be a problem.

[d-aguilera]

That worked @pjanotti !

[d-aguilera]

and I'm seeing both manual and automatic traces in Grafana.

[d-aguilera]

This is the log file now in demo-service-1 :

corehost_verbose_tracing.log

[d-aguilera]

these env vars are documented here.

Maybe this should be mentioned in the troubleshooting doc. Or in the main readme perhaps.

[pjanotti]

This issue combines two distinct problems:

1. The lack of native ARM binaries prevents any bytecode instrumentation on any ARM architecture, including Apple M1 boxes. This will be tracked via Add arm64 build for native components #1865
2. The shared store doesn't build for the arm64 architecture. Currently we only cover x86 and x64 architectures but the list in the dotnet/runtime is much larger. Opened Add arm64 architecture to the shared store #2181

I will keep this open until we added the related information to the troubleshooting doc.

[d-aguilera]

Awesome guys. Thank you all very much.

1. The lack of native ARM binaries prevents any bytecode instrumentation on any ARM architecture

Does this mean a different kind of instrumentation was used in my case? Cause I'm seeing the traces

[pellared]

Awesome guys. Thank you all very much.

1. The lack of native ARM binaries prevents any bytecode instrumentation on any ARM architecture

Does this mean a different kind of instrumentation was used in my case? Cause I'm seeing the traces

The "source instrumentation type" is what is working in your scenario.

References:

• https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/main/docs/design.md#library-instrumentations
• https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/main/docs/config.md#instrumentations