Splunk HEC exporter sends empty log messages, which are disregarded with an error message

[rohits-splunk]

Component(s)

No response

Describe the issue you're reporting

I am trying to export logs from a window server using OTEL collector and HEC exporter , but getting the below error in windows event logs ::

1.6795869182428775e+09 error exporterhelper/queued_retry.go:401 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec", "error": "Permanent error: \"HTTP/1.1 400 Bad Request\\r\\nContent-Length: 27\\r\\nConnection: Keep-Alive\\r\\nContent-Type: application/json; charset=UTF-8\\r\\nDate: Thu, 23 Mar 2023 15:55:18 GMT\\r\\nServer: Splunkd\\r\\nVary: Authorization\\r\\nX-Content-Type-Options: nosniff\\r\\nX-Frame-Options: SAMEORIGIN\\r\\n\\r\\n{\\\"text\\\":\\\"No data\\\",\\\"code\\\":5}\"", "dropped_items": 1} go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:401 go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send go.opentelemetry.io/[email protected]/exporter/exporterhelper/logs.go:135 go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1 go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:205 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1 go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:61

In the hec endpoint i am using the same endpoint which i tried for sending the events through curl which worked perfectly. Not sure why the exporter is failing.

[atoulme]

Is Splunk also running on Windows?

[rohits-splunk]

No this is the Splunk cloud endpoint. Nothing running locally.

[atoulme]

Understood. We had that report - the "No data" error means that the log record itself is empty. This is a bug ; we should probably drop empty log records.

[rohits-splunk]

No the log record is not empty, i am sending the same log to signalfx endpoint and its reaching there fine. Just not on Splunk

[atoulme]

The error specifically says {"text":"No data","code":5} which points to absence of data in the log record event being sent, per https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/TroubleshootHTTPEventCollector.

[VihasMakwana]

@rohits-splunk can you attach the configurations you're using for splunk hec exporter? Are they default?

[rohits-splunk]

Hi @vihas-splunk , This is my splunk_hec exporter configuration ::
`exporters:

Traces

sapm:
access_token: "${SPLUNK_ACCESS_TOKEN}"
endpoint: "${SPLUNK_TRACE_URL}"

Metrics + Events

signalfx:
access_token: "XXXX"
api_url: http://ip-172-XX.us-west-1.compute.internal:6060/
ingest_url: http://ip-172-XX.us-west-1.compute.internal:9943/
# Use instead when sending to gateway
#api_url: http://${SPLUNK_GATEWAY_URL}:6060
#ingest_url: http://${SPLUNK_GATEWAY_URL}:9943
sync_host_metadata: true
correlation:

Logs

splunk_hec:
token: "bc48819e-41d2-XXX"
endpoint: "https://XXXX.splunkcloud.com:8088/services/collector/event"
tls:
insecure_skip_verify: true
source: "otel"
sourcetype: "otel"
index: main
health_check_enabled : true
export_raw: true`

[VihasMakwana]

is there any chance you can share the logs that you're ingesting?

[rohits-splunk]

@vihas-splunk Here you go ::
{"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "nonprod: some other log data"} {"message": "prod: some log data"} {"message": "prod: some log data"} "hec.log" 169L, 7223B {"message": "prod: some log data"} {"message": "prod: some log data"}

[VihasMakwana]

@atoulme can you please assign this to me, I have found the root cause.

[atoulme]

It’s yours!