Latest version of OpenTelemetry (1.4) is unusable due to dependency on System.Diagnostics.DiagnosticSource 7.0

[ThatRendle]

When using the latest version of the OpenTelemetry NuGet package (currently 1.4.0-beta3) an in-process function fails to start because OTel has taken a dependency on System.Diagnostics.DiagnosticSource 7.0.0, and I'm guessing the Host runtime is on 6.0.0 and won't let the package assemblies overwrite its own assemblies?

Investigative information

Please provide the following:

• Timestamp: 2022-11-25
• Function App version: v4

Repro steps

Provide the steps required to reproduce the problem:

1. Create a Function App
2. Add OpenTelemetry (see samples below)
3. Try and run locally with func or deploy to Azure

Packages:

<ItemGroup>
  <PackageReference Include="Azure.Monitor.OpenTelemetry.Exporter" Version="1.0.0-beta.5" />
  <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
  <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.1.3" />
  <PackageReference Include="OpenTelemetry" Version="1.4.0-beta.3" />
  <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.4.0-beta.3" />
  <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc9.9" />
  <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.9" />
  <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.9" />
  <PackageReference Include="System.Diagnostics.DiagnosticSource" Version="7.0.0" />
</ItemGroup>

Service setup:

private static void AddOpenTelemetry(IFunctionsHostBuilder builder)
{
    var attributes = new Dictionary<string, object>
    {
        ["host.name"] = Environment.MachineName,
        ["os.description"] = RuntimeInformation.OSDescription,
    };
    
    var resourceBuilder = ResourceBuilder.CreateEmpty()
        .AddService("OtelTestApp")
        .AddEnvironmentVariableDetector()
        .AddTelemetrySdk()
        .AddAttributes(attributes);

    builder.Services.AddOpenTelemetryTracing(builder =>
        {
            builder.SetResourceBuilder(resourceBuilder)
                .SetSampler(new AlwaysOnSampler())
                .AddSource("OtelTestApp.*")
                .AddSource("Microsoft.AspNetCore.Hosting")
                .AddHttpClientInstrumentation()
                .AddOtlpExporter(o =>
                {
                    o.Endpoint = new Uri("http://otel-collector.notarealurl-9dhf9w8.uksouth.azurecontainerapps.io/v1/traces");
                    o.ExportProcessorType = ExportProcessorType.Batch;
                    o.Protocol = OtlpExportProtocol.HttpProtobuf;
                });
        });
}

Expected behavior

Function App should start.

Actual behavior

Function app fails to start.

Error alert in Portal:

Exception when running locally:

System.IO.FileNotFoundException: Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
File name: 'System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
   at Microsoft.Extensions.DependencyInjection.TracerProviderBuilderServiceCollectionExtensions.ConfigureOpenTelemetryTracing(IServiceCollection services, Action`1 configure)
   at Microsoft.Extensions.DependencyInjection.OpenTelemetryServicesExtensions.AddOpenTelemetryTracing(IServiceCollection services, Action`1 configure)
   at OtelTestApp.Startup.AddOpenTelemetry(IFunctionsHostBuilder builder) in C:\Code\newday\otel\App\src\OtelTestApp\Startup.cs:line 48
   at OtelTestApp.Startup.Configure(IFunctionsHostBuilder builder) in C:\Code\newday\otel\App\src\OtelTestApp\Startup.cs:line 31
   at Microsoft.Azure.Functions.Extensions.DependencyInjection.FunctionsStartup.Configure(WebJobsBuilderContext context, IWebJobsBuilder builder)
   at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.ConfigureStartup(IWebJobsStartup startup, WebJobsBuilderContext context, IWebJobsBuilder builder) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs:line 162
   at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.ConfigureAndLogUserConfiguredServices(IWebJobsStartup startup, WebJobsBuilderContext context, IWebJobsBuilder builder, ILoggerFactory loggerFactory) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs:line 130
   at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.UseWebJobsStartup(IWebJobsBuilder builder, Type startupType, WebJobsBuilderContext context, ILoggerFactory loggerFactory) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs:line 115
   at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.UseExternalStartup(IWebJobsBuilder builder, IWebJobsStartupTypeLocator startupTypeLocator, WebJobsBuilderContext context, ILoggerFactory loggerFactory) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs:line 213
   at Microsoft.Azure.WebJobs.Script.ScriptHostBuilderExtensions.<>c__DisplayClass7_0.<AddScriptHostCore>b__1(HostBuilderContext context, IWebJobsBuilder webJobsBuilder) in /_/src/WebJobs.Script/ScriptHostBuilderExtensions.cs:line 211
   at Microsoft.Extensions.Hosting.WebJobsHostBuilderExtensions.<>c__DisplayClass5_0.<ConfigureWebJobs>b__1(HostBuilderContext context, IServiceCollection services) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsHostBuilderExtensions.cs:line 54
   at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptHostBuilder.BuildHost(Boolean skipHostStartup, Boolean skipHostConfigurationParsing) in /_/src/WebJobs.Script.WebHost/DefaultScriptHostBuilder.cs:line 59
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.BuildHost(Boolean skipHostStartup, Boolean skipHostJsonConfiguration) in /_/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs:line 582
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation, Int32 attemptCount, JobHostStartupMode startupMode) in /_/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs:line 277

Known workarounds

Using Isolated works OK, but that's a sub-optimal solution.

Related information

The current version of Azure.Monitor.OpenTelemetry.Exporter has a dependency on OpenTelemetry 1.4.0.

I appreciate that all the OTel 1.4 stuff is RC or beta right now, but it should go final soon and it would be good if Azure Functions supported it when it does.

[ramya894]

@markrendle We are investigating this issue and let you know the findings soon.

[alanwest]

@markrendle thanks for reporting this, I've noted this as well. I'm a maintainer of the OpenTelemetry .NET project.

@ramya894 The underlying incompatibility is actually System.Diagnostics.DiagnosticSource version 7.0.0 which OpenTelemetry .NET depends on.

I have a simple repro here https://github.com/alanwest/azure-function-diagnostic-source. My repro only depends on System.Diagnostics.DiagnosticSource and not OpenTelemetry .NET.

[fabiocav]

Hey @markrendle , thanks for opening this and apologies for the delayed response.

This is indeed a limitation of the in-proc model, where particularly for .NET shared assemblies/assemblies containing types exchanged with the user context, unification is required and must match the major version of the hosting environment.

For situations like this, Isolated is the recommendation, so I'd love to better understand what issues you have found there to make sure those are being tracked.

In some cases, we have the ability to update the host dependency across major versions after performing compatibility analysis. We'll take a closer look at System.Diagnostics.DiagnosticSource and see if that is a viable option here, but we can't make a commitment, as any breaking change has a potential impact on the in-proc model.

[cijothomas]

@fabiocav Just to reiterate that this issue is not just about OpenTelemetry users. Any app with DS 7.0 will face this. For example, if a library uses DS 7.0 to instrument (for metrics/traces), will face this issue irrespective of OpenTelemetry is enabled or not.

I can totally understand the limitation of the infra preventing this. Could you make this limitation documented so we can easily point people to the issue and the workaround/fix (move away from in-proc )?

(We have seen customers facing this issue when they use Microsoft.Extensions.Logging version 7.0 as well)

[RohitRanjanMS]

The issue is with the OpenTelemetry SDK >= 1.4. Please use 1.3.2 while we fix this issue.

A host error has occurred during startup operation '761f1320-de8e-4785-95f1-65d2edb421d2'.
Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
Value cannot be null. (Parameter 'provider')

[hejingkan2005]

@fabiocav , I also meet the same issue with Azure function after migrating OTel to 1.4.0-rc.3, may I have the update or plan of fix from Azure function?

[RohitRanjanMS]

Hi @hejingkan2005,
It will take some time before we can support Diagnostic Source 7. It's a big risk for us to support this as it may break some existing scenarios. We share this assembly across context boundaries, and this makes dependency on Diagnostic Source extremely vulnerable.
1.4 has additional features and I understand the frustration with the lack of support in Functions, but unfortunately, we won't be able to support this any time soon.

While this is an issue with .Net InProc, I would recommend .Net Isolated. As the worker in .Net Isolated is running in its own process, there is no dependency on Functions runtime.
Alternatively, you can still use 1.3 but there's risk involved as any critical/security issue won't be patched to 1.3 once we have a stable release of 1.4.

[alanwest]

I have successfully used FunctionsPreservedDependencies to get around this problem. I've updated my repro.

[lsl-2020]

Hi @fabiocav, just saw you've moved this to In-progress in Azure Functions - Host last week. Does that mean there are team working on it now? Thanks.

[cataggar]

Thanks @alanwest for that workaround! For the app I'm working on, I had to do:

  <ItemGroup>
    <!-- Allow the functions host to use newer versions. -->
    <FunctionsPreservedDependencies Include="System.Diagnostics.DiagnosticSource.dll" />
    <FunctionsPreservedDependencies Include="System.Text.Json.dll" />
    <FunctionsPreservedDependencies Include="System.Text.Encodings.Web.dll" />

Our tests are passing. Is there anything to be wary of?

[vasylstrogushr]

Workaround with FunctionsPreservedDependencies helps to run and start app but in my case not all activity traces (like http calls) are shown.

[erichkray]

Workaround with FunctionsPreservedDependencies helps to run and start app but in my case not all activity traces (like http calls) are shown.

From my experience, since the host and the function are on different versions of DiagnosticSource, the two operations do not get chained correctly.

[markti]

I am running into possibly a related issue... I am trying to deploy .NET 6.0 azure functions
I keep getting AZFD0005

Error code
AZFD0005
Level
Error
Message
Error configuring services in an external startup class.
Details
Microsoft.Azure.WebJobs.Script.ExternalStartupException : Error configuring services in an external startup class. ---> System.IO.FileNotFoundException : Could not load file or assembly 'Microsoft.Extensions.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified. at GitHubCrawler.Startup.Configure(IFunctionsHostBuilder builder) at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.ConfigureStartup(IWebJobsStartup startup,WebJobsBuilderContext context,IWebJobsBuilder builder) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 162 at

My csproj only references 6.0.0 version!

https://github.com/markti/open-source-strength/blob/main/src/dotnet/GitHubCrawler/GitHubCrawler/GitHubCrawler.csproj

[RassK]

@alanwest @cijothomas any updates on this?

OpenTelemetry 1.6.0 seems to run into this issue as well as S.D.DS is updated to 7.0.2 and host is looking for 7.0.0.

[cijothomas]

@alanwest @cijothomas any updates on this?

OpenTelemetry 1.6.0 seems to run into this issue as well as S.D.DS is updated to 7.0.2 and host is looking for 7.0.0.

I work in OpenTelemetry, so cannot comment on any updates from Azure Functions.
OpenTelemetry continuously updates DS versions. OTel 1.7.0 will bump DS to 8.0.

[RohitRanjanMS]

We will eventually bump-up DiagnosticSource when we move to .Net 8 for InProc. No ETA as of now.

https://techcommunity.microsoft.com/t5/apps-on-azure-blog/net-on-azure-functions-august-2023-roadmap-update/ba-p/3910098#:~:text=We%20are%20therefore%20also%20ready%20to%20confirm%20that%20we%20intend%20for%20.NET%208%20to%20be%20the%20last%20LTS%20release%20to%20receive%20in%2Dprocess%20model%20support%20in%20Azure%20Functions.

[Krumelur]

I don't fully follow what the correct workaround would now be.

• I had Otel.Exporter.Geneva 1.6.0 installed and would not get functions to work. All I got was a "File not found in OpenTelemtry.dll"
• I found this thread here and downgraded from 1.6.0 to 1.3.1 because I read somewhere that I should use a version lower than 1.4.x
• Now, I'm getting what @RohitRanjanMS mentions:

[2023-11-15T17:50:48.073Z] Executed 'Function1' (Failed, Id=7f124758-6107-45c1-be89-5bb090bede73, Duration=808ms)
[2023-11-15T17:50:48.074Z] System.Private.CoreLib: Exception while executing function: Function1. OpenTelemetry: Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

The project is Azure Functions using net6 as target framework.

What exactly is the recommended (temporary) fix?

[erichkray]

@Krumelur, what version of the OpenTelemetry library are you using?

As far as I am aware, 1.3.2 is the latest that works with the Azure Function Host on .NET 6. Honestly, the easiest way to work around this is to transition your project to an isolated process Azure Function and then you can use whatever version of Otel that you want.

[RohitRanjanMS]

yeah, you will not run into any of these issues with isolated. We just announced the support of .Net 8 in isolated.
https://azure.microsoft.com/en-us/updates/ga-azure-functions-supports-net-8-in-the-isolated-worker-model/

[shaikho]

I have successfully used FunctionsPreservedDependencies to get around this problem. I've updated my repro.

Worked for me in in-proc mode thanks man. @alanwest

[RohitRanjanMS]

We have released .NET 8 support for InProc, which should help address some of the concerns - https://azure.microsoft.com/en-us/updates/v2/azure-functions-supports-net8-using-in-process-model.

Recommendation is still to migrate to isolated model.