Logs with category "ApplicationGatewayFirewallLog" fails in Azure Event Hub Receiver

[sigurdfalk]

Component(s)

receiver/azureeventhub

What happened?

Description

We are using this receiver to collect logs from some Azure resources, at this moment: AKS, ACR and Application Gateway (AGW) with WAFv2. So for AKS and ACR there are no issues, all logs enabled are being exported and searchable in Loki. However, for the AGW, we only se logs in the category ApplicationGatewayAccessLog even though also ApplicationGatewayFirewallLog is enabled. When we export logs to Log Analytics with the same diagnostic settings, all logs show up as expected, so it's not that the logs are missing.

In the OTEL Collector logs, we se a lot of this:

2023-10-28T10:08:34.593Z warn [email protected]/azureresourcelogs_unmarshaler.go:106 Invalid Timestamp {"kind": "receiver", "name": "azureeventhub", "data_type": "logs", "time": ""}

Steps to Reproduce

• Add diagnostic setting for Azure Application Gateway with ApplicationGatewayFirewallLog enabled
• Stream logs to EventHub
• Use azureeventhubreceiver to pick up logs from the EventHub

Expected Result

Logs going trough the OTEL pipeline and ending up being exported by our exporter (which is Loki in this case=

Actual Result

Lots of warnings in the OTEL collector logs saying:

2023-10-28T10:08:34.593Z warn [email protected]/azureresourcelogs_unmarshaler.go:106 Invalid Timestamp {"kind": "receiver", "name": "azureeventhub", "data_type": "logs", "time": ""}

We did not observe any logs being exported by the exporter

Collector version

v0.84.0

Environment information

Environment

OS: AKS v1.25.6
Installed via the OTEL Operator Helm chart

OpenTelemetry Collector configuration

receivers:
         azureeventhub:
          connection: xxx
          format: "azure"
          storage: file_storage     

      processors:
        batch:
        attributes/loki-azure:
          actions:
            - action: insert
              key: azure_category
              from_attribute: azure.category
            - action: insert
              key: loki.attribute.labels
              value: azure_category
        resource/loki-format-raw:
          attributes:
            - action: insert
              key: loki.format
              value: raw
              
      exporters:
        loki:
          endpoint: xxx
          headers:
            Authorization: xxx
            X-Scope-OrgID: xxx
          default_labels_enabled:
            exporter: false
            job: false
            instance: false
            level: false              
            
      service:
        pipelines:
          logs/eventhub:
            receivers:
              - azureeventhub
            processors:
              - batch
              - attributes/loki-azure
              - resource/loki-format-json
            exporters:
              - loki

Log output

No response

Additional context

Seems like Microsoft is not following their own standard in this particular log category. The filed "timestamp" should be "time" according to documentation. https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-diagnostics#access-log

[github-actions]

Pinging code owners:

• receiver/azureeventhub: @atoulme @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[tesharp]

This is similar to #27589. There is also other case like timestamp having different format like "11/09/2023 13:55:06"

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/azureeventhub: @atoulme @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

I believe this was likely fixed in #28805, but the change has not been released yet. We have a release scheduled for this week, so please try with the latest version when available.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/azureeventhub: @atoulme @djaglowski @cparkins

See Adding Labels via Comments if you do not have permissions to add labels yourself.