You want to dynamically determine the topic when exporting with Kafka Exporter.

[pyama86]

Component(s)

exporter/kafka

Is your feature request related to a problem? Please describe.

When adding log information to Kafka, you separate the topics by log file or type. However, currently, to do this, you need to write many exporter definitions.

Describe the solution you'd like

You want to dynamically determine the topic from metadata, such as the file name obtained from the Filelog Receiver.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[shivanshuraj1333]

This looks like a reasonable request to me, if this gets triaged, would love to work on it.

[crobert-1]

Hello @pyama86, can you share a proposal for what the configuration of this option would look like?

[pyama86]

@crobert-1
Thank you for your reply. I'm hoping to be able to make definitions like the ones below.

receivers:
  filelog:
    include: [ /var/log/containers/*.log]
    operators:
      - type: json_parser

processors:
  # This processor is expected to extract k8s-related metadata from the logs,
  # and add it as attributes to the logs.
  k8sattributes:

exporters:
  kafka/custom:
    default_topic: default-topic
    brokers:
      - localhost:9092
    topic_from_attribute: k8s.app

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [k8sattributes]
      exporters: [kafka/custom]

It seems there's currently no way to access the attributes created by the processor, which would be very useful for various purposes.

[crobert-1]

The idea sounds good to me, but I do wonder a bit at implementation details. Do you plan on only allowing resource attribute names to be the source of value for topic_from_attribute?

[pyama86]

I think that users will likely want to input various values as needed, and the processor can generate or transform attributes accordingly, which seems to allow for reasonably flexible responses. On the other hand, I'm not very familiar with the implementation of the otel collector, so if there are any concerns or points that should be considered, presenting them would enable a deeper examination.

[crobert-1]

That's fair. I think initially we can restrict it to resource attributes as the source. The resource attribute name can be the value for topic_from_attribute, and then the resource attribute's value will be the topic. 👍

[pyama86]

Is there anything else I can contribute to this matter? I'm willing to help with anything if there is something I can do.

[crobert-1]

You're welcome to submit a PR if you're able! Otherwise it will depend on the priorities and availability of the component code owners, or other contributors.

[pyama86]

That's great! Whether it gets merged or not, I also believe that writing code is the best way to express what you want to do, so I'll give it a try. Thank you.