[receiver/pulsar] Make pulsar_receiver more configurable

[dao-jun]

Component(s)

receiver/pulsar

Is your feature request related to a problem? Please describe.

Currently, in pulsar_receiver, configuration items only supports specify a single topic name for traces/metrics/logs.

And traces/metrics/logs receiver has default topic name(otlp_spans for traces, otlp_logs for logs and otlp_metrics for metrics), say, if users specify a topic for traces, but couldn't specify topic for logs and metrics. If users want to start a metrics receiver pipeline, it will be failed, because otlp_spans otlp_logs and otlp_metrics is not pulsar topic name pattern.

A topic name pattern must comply with a certain patter(pulsar://tenant/namespace/topic_name, example: pulsar://public/default/otlp_spans).

Also, pulsar_receiver only provide a single encoding configuration for logs,traces and metrics. Say, users want to use jaeger_proto for traces, but it couldn't apply to metrics and logs.

In this conditions, users have to config as below:

receivers:
  pulsar/trace:
    encoding: jaeger_json
    topic: pulsar://public/default/otlp_spans
    #ignore other items...

  pulsar/log:
    encoding: otlp_proto
    topic: pulsar://public/default/otlp_logs
    #ignore other items...

  pulsar/metric:
    encoding: otlp_proto
    topic: pulsar://public/default/otlp_metrics
    #ignore other items...

service:
  pipeline:
    traces:
      receivers: [pulsar/trace]
      processors: [batch]
      exporters: [zipkin]

    logs:
      receivers: [pulsar/log]
      processors: [batch]
      exporters: [file]

    metrics:
      receivers: [pulsar/metric]
      processors: [batch]
      exporters: [prometheus]

This is relatively complex, so I hope to make pulsar_receiver more configurable.

Describe the solution you'd like

Provide extra configuration items:

trace:
  topic:
  encoding:
  subscription:
  consumer_name:
log:
  topic:
  encoding:
  subscription:
  consumer_name:
metric:
  topic:
  encoding:
  subscription:
  consumer_name:

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners for receiver/pulsar: @dmitryax @dao-jun. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmitryax]

@dao-jun with the suggested change users would have to repeat all the fields in all the signals sections even if they want to share the same configuration. So it doesn't seem like the suggested change always simplifies the configuration.

[dao-jun]

@dao-jun with the suggested change users would have to repeat all the fields in all the signals sections even if they want to share the same configuration. So it doesn't seem like the suggested change always simplifies the configuration.

@dmitryax The topic field cannot share with others, so it has to be removed. Since this field must be removed, it has already introduced breaking change, so I think we can take this opportunity to make the configuration items more clearly.

[dmitryax]

This makes the configuration interface inconsistent with kafka receiver and exporter. Ideally, they should be similar.

Maybe we can add logs_topic, metrics_topic and traces_topic if that's usually the only different field? We have a similar configuration in the clickhouse exporter. Then we could apply the same to the kafka components

[dao-jun]

This makes the configuration interface inconsistent with kafka receiver and exporter. Ideally, they should be similar.

Maybe we can add logs_topic, metrics_topic and traces_topic if that's usually the only different field? We have a similar configuration in the clickhouse exporter. Then we could apply the same to the kafka components

Pulsar exporter/receiver referred to kafka exporter/receiver when I developed them, but so far, I think there are some issues involved in this.

Unclear configuration items are a major issue, if the topic is not empty, it will apply to traces, metrics and logs pipeline, which means they will consume messages from a same topic! It will cause errors in the process of Unmarshal data, right?

[dmitryax]

Unclear configuration items are a major issue, if the topic is not empty, it will apply to traces, metrics and logs pipeline, which means they will consume messages from a same topic! It will cause errors in the process of Unmarshal data, right?

That's why I suggest replacing topic with metrics_topic, traces_topic and logs_topic. Does it make sense?

[dao-jun]

Unclear configuration items are a major issue, if the topic is not empty, it will apply to traces, metrics and logs pipeline, which means they will consume messages from a same topic! It will cause errors in the process of Unmarshal data, right?

That's why I suggest replacing topic with metrics_topic, traces_topic and logs_topic. Does it make sense?

it surely makes sense, but what for encoding,subscription and consumer_name? Also introduce trace_encoding,metric_encoding, log_encoding,trace_subscription,metric_subscription,log_subscription,trace_consumer_name,metric_consumer_name and log_consumer_name?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/pulsar: @dmitryax @dao-jun

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.