OpenTelemetry support on Azure Function App v3 (.Net Core 3.1)

[robece]

Question

Describe your environment.

Azure Function App v3 using .Net Core 3.1 working in Visual Studio 2019.

Describe any aspect of your environment relevant to the question.

This is my current project configuration:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
    <DockerFastModeProjectMountDirectory>/home/site/wwwroot</DockerFastModeProjectMountDirectory>
    <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
    <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Azure.Storage.Queues" Version="12.5.0" />
    <PackageReference Include="Microsoft.ApplicationInsights" Version="2.16.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.DependencyCollector" Version="2.16.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.3.1" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.SignalRService" Version="1.2.2" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="4.0.3" />
    <PackageReference Include="Microsoft.Extensions.Http" Version="3.1.10" />
    <PackageReference Include="Microsoft.Extensions.Http.Polly" Version="3.1.10" />
    <PackageReference Include="Microsoft.IdentityModel.Protocols" Version="6.8.0" />
    <PackageReference Include="Microsoft.IdentityModel.Protocols.OpenIdConnect" Version="6.8.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.11" />
    <PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.10.9" />
    <PackageReference Include="MongoDB.Driver" Version="2.11.4" />
    <PackageReference Include="OpenTelemetry.Exporter.Jaeger" Version="1.0.0-rc1.1" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc1.1" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc1.1" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc1.1" />
    <PackageReference Include="Polly" Version="7.2.1" />
    <PackageReference Include="SendGrid" Version="9.21.2" />
    <PackageReference Include="System.IdentityModel.Tokens.Jwt" Version="6.8.0" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

What are you trying to achieve?

I'm trying to initialize the distributed telemetry OpenTelemetry component in the Azure Function startup.cs.

Project.csproj:

    <PackageReference Include="OpenTelemetry.Exporter.Jaeger" Version="1.0.0-rc1.1" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc1.1" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc1.1" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc1.1" />

Startup.cs:

            builder.Services.AddOpenTelemetryTracing((builder) => builder
                .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("workflow"))               
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddJaegerExporter(jaegerOptions =>
                {
                    jaegerOptions.AgentHost = "localhost";
                    jaegerOptions.AgentPort = 6831;
                }));

I'm getting the following error:

[2020-11-21T00:59:37.448Z] A host error has occurred during startup operation 'a1099142-3c92-4e8b-aef2-1bd8e68bc0c1'.
[2020-11-21T00:59:37.450Z] func: Invalid host services. Microsoft.Azure.WebJobs.Script.WebHost: The following service registrations did not match the expected services:
[2020-11-21T00:59:37.450Z]   [Invalid] ServiceType: Microsoft.Extensions.Hosting.IHostedService, Lifetime: Singleton, ImplementationType: OpenTelemetry.Extensions.Hosting.Implementation.TelemetryHostedService, OpenTelemetry.Extensions.Hosting, Version=1.0.0.1, Culture=neutral, PublicKeyToken=7bd6737fe5b67e3c.
Value cannot be null. (Parameter 'provider')

What did you expect to see?

I ran this same pieces of code in other application: ASP.NET Core WebApp (not Azure Function) and is running successfully, I'm able to visualize the tracing in Jaeger, but I can't run it successfully in the Azure Function App project.

Additional Context

Add any other context about the problem here. If you followed an existing
documentation, please share the link to it.

I'm using this example as reference: https://github.com/open-telemetry/opentelemetry-dotnet/tree/master/examples/AspNetCore

Thank you,
RC

[cijothomas]

Azure functions has restrictions on running backgroundhostedservices, so you cannot use this package in Functions.

You can manually use opentelemetry

var openTelemetry = Sdk.CreateTracerProviderBuilder().
                AddSource("MyActivitySourceName")
                .SetSampler(new AlwaysOnSampler())
                .AddConsoleExporter()
                .Build();
            builder.Services.AddSingleton(openTelemetry);

(Functions has built-in integration with AzureApplicationInsights, and I am not sure how to setup something else)

[cijothomas]

Please try the above snippet and see if it works. I'll ask around for Azure Functions limitations.

[robece]

Thanks Cijo, I guess that should be an architecture limitation thinking in a serverless deployment, I found a related issue with hosted services on Azure Functions: Azure/azure-functions-host#5447 (comment), I already performed the suggested change, the error is gone but sadly no information is exposed in jaeger.

I will appreciate any workaround on this before decide to move to Azure Application Insights, I really like Open Telemetry and would like to implement distributed tracing on some containerized functions I have.

Thank you,
RC

[cijothomas]

no information is exposed in jaeger.

Could you share more on this - are you create activities using ActivitySource yourself, and they are not shown in Jaeger?
Or are you relying on the instrumentations like Sql/Http, and activities from these are not shown in Jaeger?

[robece]

Hi @cijothomas I have decided to move to Application Insights at least just for Functions/Durable Functions, I will continue working with Open Telemetry and hope to see soon more .net core samples with this kind of projects.

Thank you,
RC

[cijothomas]

#1803 closing as duplicate