[exporter/awsemf] Move CloudWatch Metric Stats Attribute to the bottom of EMF Logs entries

[mxiamxia]

Component(s)

exporter/awsemf

What happened?

Description

Cloudwatch metrics are being formatted and sent to AWS CW in EMF logs, however the metrics attributes today are being presented in a random order[code] in the log entries. For Cloudwatch customers, they can use LogsInsight to query these EMF logs but LogsInsights has the limitation that it can only scan maximumly 200 the event fields in a single Json Log entry. We've seen the EMF Log entries which have exceeded these 200 event fields limits when it includes the histogram bucket items(see atttached Latency MetricStats example) in EMF logs. CW Logs count each individual value in bucket as a single event fields. So we can easily reach the limit if there is a large bucket size included. We can't recommend our customers to reduce the bucket size during Instrumentation as it will reduce the accuracy of histogram metric used for their observability.

To mitigate this problem, we propose to put the Histogram MetricStats(Latency attribute as example) to the bottom of the EMF logs entries. So CW LogsInsight query will always be able to scan valuable data(Eg, metric dimensions) for the queries before reaching the event fields limit. This will require we change serialization implementation for PutLogsEvent API in EMFExporter.

Steps to Reproduce

n/a

Expected Result

{
    "EKS.Cluster": "petclinic-sampleApp",
    "K8s.Namespace": "default",
    "K8s.Node": "i-0102cd843cb286120",
    "K8s.Pod": "customers-service-b8699b487-4ltgk",
    "K8s.Workload": "customers-service",
    "aws.span.kind": "SERVER",
    "OTelLib": "AwsSpanMetricsProcessor",
    "Operation": "GET /owners",
    "Service": "customers-service",
    "Version": "1",
    "_aws": {
        "CloudWatchMetrics": [
            {
    ...
        ],
        "Timestamp": 1694724655040
        }
    },
    "Latency": {
        "Values": [
            7.180520493565249,
            7.498429295612942,
            8.177095129709611,
            8.53912606037064,
            9.311982999037598,
            9.724259708304764,
            10.154789466897915,
            10.604380406351545,
            11.073876437239882,
            12.076147885389684,
            13.752179963522824,
            14.996858591225857,
..............
            16.354190259419195,
            17.834370939375464,
            19.448519416609493,
            28.72208197426089,
            421.41224479444895,
            479.89947491922317
        ],
        "Counts": [
            3,
            3,
            1,
            3,
            1,
            2,
            2,
            1,
            2,
            1,
            2,
            1,
            1,
            1,
            1,
            1,
            1,
            1
        ],        
        "Max": 640.013708,
        "Min": 640.013708,
        "Count": 1,
        "Sum": 640.013708
    }

Actual Result

The current EMF logs attribute order

{
    "EKS.Cluster": "petclinic-sampleApp",
    "Latency": {
        "Values": [
            7.180520493565249,
            7.498429295612942,
            8.177095129709611,
            8.53912606037064,
            9.311982999037598,
            9.724259708304764,
              ................
            10.154789466897915,
            10.604380406351545,
            11.073876437239882,
            12.076147885389684,
            13.752179963522824,
            14.996858591225857,
            16.354190259419195,
            17.834370939375464,
            19.448519416609493,
            28.72208197426089,
            421.41224479444895,
            479.89947491922317
        ],
        "Counts": [
            3,
            3,
            1,
            3,
            1,
            2,
            2,
            1,
            2,
            1,
            2,
            1,
            1,
            1,
            1,
            1,
            1,
            1
        ],        
        "Max": 640.013708,
        "Min": 640.013708,
        "Count": 1,
        "Sum": 640.013708
    },
    "K8s.Namespace": "default",
    "K8s.Node": "i-0102cd843cb286120",
    "K8s.Pod": "customers-service-b8699b487-4ltgk",
    "K8s.Workload": "customers-service",
    "OTelLib": "AwsSpanMetricsProcessor",
    "Operation": "GET /owners",
    "Service": "customers-service",
    "Version": "1",
    "_aws": {
        "CloudWatchMetrics": [
            {
...
        ],
        "Timestamp": 1694724655040
    },
    "aws.span.kind": "SERVER"
}

Collector version

v0.85

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

## Actual Result
The current EMF logs attribute order

{
"EKS.Cluster": "petclinic-sampleApp",
"Error": {
"Values": [
0
],
"Counts": [
1
],
"Max": 0,
"Min": 0,
"Count": 1,
"Sum": 0
},
"Fault": {
"Values": [
0
],
"Counts": [
1
],
"Max": 0,
"Min": 0,
"Count": 1,
"Sum": 0
},
"K8s.Namespace": "default",
"K8s.Node": "i-0102cd843cb286120",
"K8s.Pod": "customers-service-b8699b487-4ltgk",
"K8s.Workload": "customers-service",
"Latency": {
"Values": [
7.180520493565249,
7.498429295612942,
8.177095129709611,
8.53912606037064,
9.311982999037598,
9.724259708304764,
10.154789466897915,
10.604380406351545,
11.073876437239882,
12.076147885389684,
13.752179963522824,
14.996858591225857,
16.354190259419195,
17.834370939375464,
19.448519416609493,
28.72208197426089,
421.41224479444895,
479.89947491922317
],
"Counts": [
3,
3,
1,
3,
1,
2,
2,
1,
2,
1,
2,
1,
1,
1,
1,
1,
1,
1
], "Max": 640.013708,
"Min": 640.013708,
"Count": 1,
"Sum": 640.013708
},
"OTelLib": "AwsSpanMetricsProcessor",
"Operation": "GET /owners",
"Service": "customers-service",
"Version": "1",
"_aws": {
"CloudWatchMetrics": [
{
...
],
"Timestamp": 1694724655040
},
"aws.span.kind": "SERVER"
}

### Collector version

v0.85

### Environment information

## Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")


### OpenTelemetry Collector configuration

_No response_

### Log output

_No response_

### Additional context

_No response_

[github-actions]

Pinging code owners for exporter/awsemf: @Aneurysm9 @shaochengwang @mxiamxia. See Adding Labels via Comments if you do not have permissions to add labels yourself.