How to solve "error": "Throttle (0s), error: HTTP 503 \"Service Unavailable\" when using SplunkHECExporter to send data into Splunk?

[ericashi]

Component(s)

exporter/splunkhec

What happened?

Description

Currently, I am consuming data from Kafka Stream using kafkareceiver then send the data consumed into Splunk using splunkhec exporter. I am able to get the data into Splunk but there are times where error message is flagged. Please refer to the Log Output for the error log message.

May I get your assistance on the following questions, please?

1. Why does this error message flagged sometimes?
2. How can I solve this error message flagged?
3. Will this error message cause data loss?

Steps to Reproduce

Setup a config.yaml file with needed configurations and run the otelcol-contrib-collector image in Kubernetes environment.
Please refer to the "OpenTelemetry Collector configuration" section for the configurations.

Expected Result

I expected the data to always be pushed and will not display any error.

Actual Result

The data is pushed but will flagged the error message as shown in the Log Output section.

Collector version

v0.88.0

Environment information

Environment

Kubernetes container

OpenTelemetry Collector configuration

receivers:
  kafka:
    protocol_version: 2.0.0
    topic: <topic_name>
    brokers:
      - "<broker_link>"
    group_id: <group_id_name>
    encoding: json

processors:
  batch:
    send_batch_size: 10000
    timeout: 10s

exporters:
  splunk_hec:
    token: <HEC_Token>
    endpoint: <SPLUNK_Endpoint>
    sourcetype: <sourcetype_name>

  logging:
    verbosity: detailed

service:
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8885

  pipelines:
    logs:
      receivers: [kafka]
      processors: [batch]
      exporters: [logging, splunk_hec]

Log output

2023-11-05T16:24:47.334Z	info	exporterhelper/retry_sender.go:177	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "logs", "name": "splunk_hec", "error": "Throttle (0s), error: HTTP 503 \"Service Unavailable\"", "interval": "3.965794253s"}

Additional context

May I get your assistance on the following questions, please?

1. Why does this error message flagged sometimes?
2. How can I solve this error message flagged?
3. Will this error message cause data loss?

[github-actions]

Pinging code owners:

• exporter/splunkhec: @atoulme @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ericashi]

/label help-wanted question

[dmitryax]

Hi @ericashi,

Why does this error message flagged sometimes?

The error message says that the backend is unavailable, which in most cases means the Splunk backend cannot process data fast enough.

How can I solve this error message flagged?

This is a problem with the backend, not the collector. You need to make sure the backend can handle the amount of data that is being sent and not being throttled.

Will this error message cause data loss?

No, the retry mechanism is enabled by default. The retry succeeded if there is no following "max elapsed time expired" error message.

[crobert-1]

Hello @ericashi, have your questions been answered sufficiently here? If so, we can close this issue. Otherwise, feel free to let us know if there's any other information that would be helpful.

[ericashi]

Hi,

Thanks for the information provided. My questions has been answered.
We can proceed to close this ticket.

Regards,
Erica