Emit kafka C library logs as tracing events

[armantigranyan]

I'm fairly new to Rust and I've been trying to emit the C library kafka logs as tracing events, similar to what we can do with rust-rdkafka logs if we set the tracing feature for the crate (https://github.com/fede1024/rust-rdkafka?tab=readme-ov-file#:~:text=rust%2Drdkafka%20uses%20the%20log%20crate%20to%20handle%20logging.%20Optionally%2C%20enable%20the%20tracing%20feature%20to%20emit%20tracing%20events%20as%20opposed%20to%20log%20records.)

i.e. I am able to see these logs as tracing events:

2024-03-27T20:00:42.645780Z TRACE foo: rdkafka::util: Destroyed client: REDACTED

What I want is these logs to be emitted as tracing events as well (sorry for REDACTED data, I'm not sure how much of this is sensitive):

%REDACTED|REDACTED|ASSIGN|rdkafka#consumer-X| [thrd:main]: Group "REDACTED": Consumer is terminating: treating assign as unassign
%REDACTED|REDACTED|CLEARASSIGN|rdkafka#consumer-X| [thrd:main]: Clearing current assignment of X partition(s)
etc.

Is that even possible? I came across this function rd_kafka_conf_set_log_cb but I'm not sure if I can get it to work from Rust.

[davidblewett]

The logs from rd_kafka_conf_set_log_cb are passed to this method on your context: https://docs.rs/rdkafka/latest/rdkafka/client/trait.ClientContext.html#method.log . You could implement a custom context that creates trace events (either instead of or in addition to the log crate handling).

[armantigranyan]

@davidblewett overriding the log method to use tracing instead of the log crate doesn't seem to change anything. Correct me if I'm wrong - wouldn't turning the tracing feature on (https://github.com/fede1024/rust-rdkafka?tab=readme-ov-file#:~:text=Optionally%2C%20enable%20the%20tracing%20feature%20to%20emit%20tracing%20events%20as%20opposed%20to%20log%20records) have achieved the same without the need to override the log method?

In any case, I'm still seeing these logs from the C library:

%REDACTED|REDACTED|ASSIGN|rdkafka#consumer-X| [thrd:main]: Group "REDACTED": Consumer is terminating: treating assign as unassign
%REDACTED|REDACTED|CLEARASSIGN|rdkafka#consumer-X| [thrd:main]: Clearing current assignment of X partition(s)
etc.

I'm wondering what the entry point for these logs is on the Rust side and how I can emit them as tracing events.

[davidblewett]

Sigh, you're right. I had forgotten about that. It was my understanding that the logging callback is the only spot. Are you passing in debug contexts perhaps? I would assume those would also flow through the log method though.

[armantigranyan]

I am passing in a custom context because I need to override some methods for ConsumerContext and more logging (all using the tracing crate) but that's about it :\

[trtt]

I think this PR can help: #655