Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grpc exporter doesn't follow dns change. #4813

Closed
pengweiqhca opened this issue Aug 30, 2023 · 5 comments
Closed

Grpc exporter doesn't follow dns change. #4813

pengweiqhca opened this issue Aug 30, 2023 · 5 comments
Labels
bug Something isn't working pkg:OpenTelemetry.Exporter.OpenTelemetryProtocol Issues related to OpenTelemetry.Exporter.OpenTelemetryProtocol NuGet package Stale Issues and pull requests which have been flagged for closing due to inactivity

Comments

@pengweiqhca
Copy link

pengweiqhca commented Aug 30, 2023

Bug Report

List of all OpenTelemetry NuGet
packages
and version that you are
using (e.g. OpenTelemetry 1.0.2):

  • OpenTelemetry.Exporter.OpenTelemetryProtocol 1.5.1

Runtime version (e.g. net462, net48, netcoreapp3.1, net6.0 etc. You can
find this information from the *.csproj file):

  • net6.0

Symptom

When i changed dns ip, grpc exporter still connected the old ip.

What is the expected behavior?

Grpc connect to new ip.

What is the actual behavior?

Grpc still connected the old ip long after i change dns.

Reproduce

var builder = WebApplication.CreateBuilder(args);

var meter = new Meter("Test");

var counter = meter.CreateObservableUpDownCounter("test.counter", () =>
{
    Console.WriteLine(DateTimeOffset.Now);

    return DateTimeOffset.Now.ToUnixTimeMilliseconds();
});

builder.Services
    .Configure<MetricReaderOptions>(options => options.PeriodicExportingMetricReaderOptions.ExportIntervalMilliseconds = 10000)
    .AddOpenTelemetry()
    .WithMetrics(mpb => mpb.AddMeter(meter.Name)
        .AddOtlpExporter(options => options.Endpoint = new Uri("http://collector.foo.com:4317")));

var app = builder.Build();

app.Run();
  1. Set collector.foo.com = ip1 in local host file.
  2. Open wireshark 1, start capturing TCP packet, set filter ip.addr == ip1 && tcp.port = 4317.
  3. Open wireshark 2, start capturing TCP packet, set filter ip.addr == ip2 && tcp.port = 4317.
  4. Run test code.
  5. TCP data will appear in wireshark 1.
  6. Modify collector.foo.com = ip2 in local host file.
  7. TCP data still in wireshark 1 rather than wireshark 2.

Additional Context

Grpc.Net use SocketsHttpHandler in .net6.0 and SocketsHttpHandler.PooledConnectionLifetime is InfiniteTimeSpan.

GrpcChannel.ForAddress(options.Endpoint) have no any options, so SocketsHttpHandler will use default settings. If grpc exporter is used every 1 minute, the http connection will never be disposed and always use the old connection, so there need to use the GrpcClientFactory instead.

Http exporter already uses HttpClientFactory.

@pengweiqhca pengweiqhca added the bug Something isn't working label Aug 30, 2023
pengweiqhca added a commit to pengweiqhca/opentelemetry-dotnet that referenced this issue Aug 30, 2023
@vishweshbankwar
Copy link
Member

@pengweiqhca I am not able to repro this behavior you are noticing (I do not see any interruption). looking at the doc as well, it should continue to work.

@JamesNK Could you please confirm? Reference: Grpc Channel Config for otlp exporter.

@JamesNK
Copy link
Contributor

JamesNK commented Sep 11, 2023

I'm not an expert at reacting to DNS changes.

Changing the code to use Grpc.Net.Client.ClientFactory is one option.

A simpler change would be to specify a SocketsHttpHandler with GrpcChannelOptions.HttpHandler. Then you can specify SocketsHttpHandler.PooledConnectionLifetime

@vishweshbankwar
Copy link
Member

@JamesNK Thanks! The behavior I am noticing is there is no interruption in connection during ip changes with this Grpc Channel Config (different than what reported on this issue). If I interpret this doc correctly, we dont have to do anything and it should automatically refresh. is that correct?

@JamesNK
Copy link
Contributor

JamesNK commented Sep 11, 2023

Once a connection is up and running, SocketsHttpHandler controls the connection lifetime. I think https://learn.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler.pooledconnectionlifetime?view=net-7.0#system-net-http-socketshttphandler-pooledconnectionlifetime is what you want to set.

we dont have to do anything and it should automatically refresh. is that correct?

I don't know. As I said, I'm not an expert in this area.

@vishweshbankwar vishweshbankwar added the pkg:OpenTelemetry.Exporter.OpenTelemetryProtocol Issues related to OpenTelemetry.Exporter.OpenTelemetryProtocol NuGet package label Sep 14, 2023
Copy link
Contributor

This issue was marked stale due to lack of activity and will be closed in 7 days. Commenting will instruct the bot to automatically remove the label. This bot runs once per day.

@github-actions github-actions bot added the Stale Issues and pull requests which have been flagged for closing due to inactivity label Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pkg:OpenTelemetry.Exporter.OpenTelemetryProtocol Issues related to OpenTelemetry.Exporter.OpenTelemetryProtocol NuGet package Stale Issues and pull requests which have been flagged for closing due to inactivity
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants