-
Notifications
You must be signed in to change notification settings - Fork 834
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tracing not sharing context from ambassador? #731
Comments
Actually, just tried to use grpc instead of rest, and it works for grpc, just not for rest, that is, seldon-core-api-tester -p contract.json tracing-example -n 1 --grpc works, but without --grpc I ended up with two sets of traces. |
That's interesting. We do use Jaeger in the engine. Not sure why gRPC would only produce 1 trace and REST not. Would need to check the code that starts the traces in both grpc: seldon-core/engine/src/main/java/io/seldon/engine/grpc/GrpcChannelHandler.java Lines 31 to 34 in 978bbdd
rest: seldon-core/engine/src/main/java/io/seldon/engine/api/rest/RestClientController.java Lines 157 to 161 in 978bbdd
Maybe |
Lets test in new executor and reopen if still an issue |
I am using ambassador as ingress gateway for sleldon-core. This may not be a bug but I am hoping there are more insights from the community.
With the default configuration from
template_model_tracing
as well as enabling zipkin tracing using TracingService, I ended up with two separate sets of traces, one for ambassador (isolated) and another for seldon-core (including the model). I have verified that ambassador sets the headers to include the context.I assume it does not work because ambassador only sent context using zipkin-style header (b3), while by default the jaeger java client that the engine uses assumes jaeger style header. I tried setting
JAEGER_PROPAGATION
according to the java client doc to 'b3' but it still leads to two sets of traces.Appreciate help here.
The text was updated successfully, but these errors were encountered: