[receiver] Logs with category "ApplicationGatewayFirewallLog" fails in Azure Event Hub Receiver

[sigurdfalk]

Describe the bug

We are using this receiver to collect logs from some Azure resources, at this moment: AKS, ACR and Application Gateway (AGW) with WAFv2. So for AKS and ACR there are no issues, all logs enabled are being exported and searchable in Loki. However, for the AGW, we only se logs in the category ApplicationGatewayAccessLog even though also ApplicationGatewayFirewallLog is enabled. When we export logs to Log Analytics with the same diagnostic settings, all logs show up as expected, so it's not that the logs are missing.

In the OTEL Collector logs, we se a lot of this:

2023-10-28T10:08:34.593Z warn [email protected]/azureresourcelogs_unmarshaler.go:106 Invalid Timestamp {"kind": "receiver", "name": "azureeventhub", "data_type": "logs", "time": ""}

Steps to reproduce

• Add diagnostic setting for Azure Application Gateway with ApplicationGatewayFirewallLog enabled
• Stream logs to EventHub
• Use azureeventhubreceiver to pick up logs from the EventHub

What did you expect to see?

Logs going trough the OTEL pipeline and ending up being exported by our exporter (which is Loki in this case=

What did you see instead?

Lots of warnings in the OTEL collector logs saying:

2023-10-28T10:08:34.593Z warn [email protected]/azureresourcelogs_unmarshaler.go:106 Invalid Timestamp {"kind": "receiver", "name": "azureeventhub", "data_type": "logs", "time": ""}

We did not observe any logs being exported by the exporter

What version did you use?
Version: v0.84.0

What config did you use?

      receivers:
         azureeventhub:
          connection: xxx
          format: "azure"
          storage: file_storage     

      processors:
        batch:
        attributes/loki-azure:
          actions:
            - action: insert
              key: azure_category
              from_attribute: azure.category
            - action: insert
              key: loki.attribute.labels
              value: azure_category
        resource/loki-format-raw:
          attributes:
            - action: insert
              key: loki.format
              value: raw
              
      exporters:
        loki:
          endpoint: xxx
          headers:
            Authorization: xxx
            X-Scope-OrgID: xxx
          default_labels_enabled:
            exporter: false
            job: false
            instance: false
            level: false              
            
      service:
        pipelines:
          logs/eventhub:
            receivers:
              - azureeventhub
            processors:
              - batch
              - attributes/loki-azure
              - resource/loki-format-json
            exporters:
              - loki                          

Environment
OS: AKS v1.25.6
Installed via the OTEL Operator Helm chart

Additional context

Seems like Microsoft is not following their own standard in this particular log category. The filed "timestamp" should be "time" according to documentation. https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-diagnostics#access-log

[sigurdfalk]

Created in wrong repo, new issue here: open-telemetry/opentelemetry-collector-contrib#28806