Enable tracing for development #832
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dandavison
commented
Oct 17, 2024
| .with_kind(SpanKind::Server) | ||
| .with_attributes([KeyValue::new("temporal.worker", true)]) | ||
| .start(&tracer); | ||
| }); |
There was a problem hiding this comment.
This span is being emitted, but the ones in workflow_machines.rs, managed_run.rs, workflow_stream.rs aren't; not sure why yet.
There was a problem hiding this comment.
I am guessing this has to do with separate runtimes.
I can't remember exactly how I had this working with a separate one previously. Might need to review those diffs some more
dandavison
commented
Oct 17, 2024
| .span_builder("apply_next_wft_from_history") | ||
| .with_kind(SpanKind::Server) | ||
| .with_attributes([KeyValue::new("temporal.worker", true)]) | ||
| .start(&tracer); |
There was a problem hiding this comment.
This span isn't being emitted; I'm not sure why currently.
dandavison
commented
Oct 17, 2024
| .span_builder("successful_completion") | ||
| .with_kind(SpanKind::Server) | ||
| .with_attributes([KeyValue::new("temporal.worker", true)]) | ||
| .start(&tracer); |
There was a problem hiding this comment.
This span isn't being emitted; I'm not sure why currently.
dandavison
commented
Oct 17, 2024
| .span_builder("process_completion") | ||
| .with_kind(SpanKind::Server) | ||
| .with_attributes([KeyValue::new("temporal.worker", true)]) | ||
| .start(&tracer); |
There was a problem hiding this comment.
This span isn't being emitted; I'm not sure why currently.
Sushisource
reviewed
Oct 23, 2024
Comment on lines
+222
to
+261
| let runtime = tokio::runtime::Builder::new_multi_thread() | ||
| .thread_name("telemetry") | ||
| .worker_threads(2) | ||
| .enable_all() | ||
| .build() | ||
| .unwrap(); | ||
|
|
||
| // create otel export layer | ||
| runtime.block_on(async { | ||
| let tracer_cfg = opentelemetry_sdk::trace::Config::default() | ||
| .with_resource(default_resource_instance().clone()); | ||
| let provider = opentelemetry_otlp::new_pipeline() | ||
| .tracing() | ||
| .with_exporter( | ||
| opentelemetry_otlp::new_exporter() | ||
| .tonic() | ||
| .with_endpoint("grpc://localhost:4317".to_string()), | ||
| // .with_metadata(MetadataMap::from_headers(headers.try_into()?)), | ||
| ) | ||
| .with_trace_config(tracer_cfg) | ||
| // Using install_simple instead for now because install_batch is not producing spans and is emitting this error message: | ||
| // OpenTelemetry trace error occurred. cannot send message to batch processor as the channel is closed | ||
| // .install_batch(opentelemetry_sdk::runtime::Tokio) | ||
| .install_simple() | ||
| .unwrap(); | ||
| opentelemetry::global::set_tracer_provider(provider.clone()); | ||
|
|
||
| let tracer = provider.tracer_builder("sdk-core").build(); | ||
| let opentelemetry = tracing_opentelemetry::layer().with_tracer(tracer); | ||
| // .with_filter(EnvFilter::new(&tracing.filter)) | ||
| export_layer = Some(opentelemetry); | ||
|
|
||
| let tracer = provider.tracer("sdk-core"); | ||
|
|
||
| let _span = tracer | ||
| .span_builder("telemetry_init") | ||
| .with_kind(SpanKind::Server) | ||
| .with_attributes([KeyValue::new("temporal.worker", true)]) | ||
| .start(&tracer); | ||
| }); |
There was a problem hiding this comment.
This all definitely needs to be protected behind a flag. (Which I guess you mentioned hehe)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP draft PR for discussion. All the spans added here are just temporary examples; the only thing I'll be proposing for merge is the configuration, behind a flag. One span is being emitted but the others aren't; see comments inline.