Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tracing not sharing context from ambassador? #731

Closed
phsiao opened this issue Jul 26, 2019 · 3 comments
Closed

tracing not sharing context from ambassador? #731

phsiao opened this issue Jul 26, 2019 · 3 comments
Assignees
Milestone

Comments

@phsiao
Copy link

phsiao commented Jul 26, 2019

I am using ambassador as ingress gateway for sleldon-core. This may not be a bug but I am hoping there are more insights from the community.

With the default configuration from template_model_tracing as well as enabling zipkin tracing using TracingService, I ended up with two separate sets of traces, one for ambassador (isolated) and another for seldon-core (including the model). I have verified that ambassador sets the headers to include the context.

I assume it does not work because ambassador only sent context using zipkin-style header (b3), while by default the jaeger java client that the engine uses assumes jaeger style header. I tried setting JAEGER_PROPAGATION according to the java client doc to 'b3' but it still leads to two sets of traces.

Appreciate help here.

@phsiao
Copy link
Author

phsiao commented Jul 26, 2019

Actually, just tried to use grpc instead of rest, and it works for grpc, just not for rest, that is,

seldon-core-api-tester -p contract.json tracing-example -n 1 --grpc

works, but without --grpc I ended up with two sets of traces.

@ukclivecox ukclivecox added this to the 1.0.x milestone Aug 24, 2019
@ukclivecox
Copy link
Contributor

That's interesting. We do use Jaeger in the engine. Not sure why gRPC would only produce 1 trace and REST not. Would need to check the code that starts the traces in both

grpc:

TracingClientInterceptor tracingInterceptor =
TracingClientInterceptor.newBuilder()
.withTracer(this.tracingProvider.getTracer())
.build();

rest:

if (tracingProvider.isActive()) {
Tracer tracer = tracingProvider.getTracer();
tracingSpan = tracer.buildSpan("/api/v0.1/predictions").start();
tracer.scopeManager().activate(tracingSpan);
}

Maybe io.opentracing.contrib.grpc.TracingClientInterceptor does something more intelligent?

@ukclivecox ukclivecox self-assigned this Nov 7, 2019
@ukclivecox ukclivecox assigned adriangonz and unassigned ukclivecox Nov 28, 2019
@ukclivecox ukclivecox modified the milestones: 1.0, 1.1 Dec 10, 2019
@ukclivecox
Copy link
Contributor

Lets test in new executor and reopen if still an issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants