Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: OpenTelemetry log error occurred. cannot send message to batch processor as the channel is closed #1660

Open
NickLarsenNZ opened this issue Apr 11, 2024 · 5 comments
Labels
bug Something isn't working shutdown&runtime triage:todo Needs to be traiged.

Comments

@NickLarsenNZ
Copy link

NickLarsenNZ commented Apr 11, 2024

What happened?

When Tokio shuts down, I receive a lot of errors about a channel being closed:
This happens whether or not I attempt to shutdown (unset) the trace/log providers.

Here is a brief explanation as code:

#[tokio::main]
async fn main() {
    // initialise tracing/logging here

    opentelemetry::global::shutdown_tracer_provider();
    opentelemetry::global::shutdown_logger_provider();

    // Without this sleep, I get a different error:
    // OpenTelemetry log error occurred. Exporter otlp encountered the following errors: the grpc server returns error (Unknown error): , detailed error message: transport error
    tokio::time::sleep(Duration::from_secs(2)).await;

    // errors are printed at this point.
}

For a full (hopefully) reproducable version of the code and dependencies, see: https://gist.github.com/NickLarsenNZ/fbbb477230e9992e9a4aa2603edcf54f

I initially raised it in the #otel-rust Slack channel: https://cloud-native.slack.com/archives/C03GDP0H023/p1712590806651679

API Version

0.22.0

SDK Version

0.22.1

What Exporters are you seeing the problem on?

OTLP

Relevant log output

OpenTelemetry log error occurred. cannot send message to batch processor as the channel is closed.
@NickLarsenNZ NickLarsenNZ added bug Something isn't working triage:todo Needs to be traiged. labels Apr 11, 2024
@TommyCpp
Copy link
Contributor

The batch log processor has a background task that exporting the logs. When it's being droped the receiver in the background task will gets dropped too.

After this operation, every attempt to emit logs will result in such error.

@ramgdev
Copy link
Contributor

ramgdev commented Aug 20, 2024

I am seeing this too. Is it safe not to call opentelemetry::global::shutdown_tracer_provider(); when the application exits to avoid this error? What are the other implications of doing so?

@Oliboy50
Copy link

Oliboy50 commented Oct 18, 2024

so if we have this error, it means that the tracer provider has not been shutdown correctly before the application stopped?

I have this error even when I call opentelemetry::global::shutdown_tracer_provider() before ending my application, is it an expected behavior?

@Oliboy50
Copy link

Oliboy50 commented Oct 21, 2024

on my side, I realized that this error occurred after cloning the pipeline provider with something like:

let provider = otlp_pipeline
    .install_batch(opentelemetry_sdk::runtime::Tokio)
    .expect("Failed to build OTLP pipeline")
    .provider()
    .clone();

and then running opentelemetry::global::shutdown_tracer_provider(); would not shutdown everything (and therefore would gave me the error mentioned above) if the program ends while provider has not also been dropped

@francoposa
Copy link
Contributor

I have the same issue and it does not matter if it's cloned or I do everything in the same scope. It spams these errors no matter if I call shutdown on everything or not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working shutdown&runtime triage:todo Needs to be traiged.
Projects
None yet
Development

No branches or pull requests

6 participants