Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support AWS Embedded Metric Format Version 0 #2

Merged
merged 4 commits into from
Apr 5, 2023
Merged

Add support AWS Embedded Metric Format Version 0 #2

merged 4 commits into from
Apr 5, 2023

Conversation

khanhntd
Copy link

@khanhntd khanhntd commented Apr 4, 2023 •
edited
Loading

Same as this PR #1

Description:
AWS Emf v0 supports timestamp with a string while aws emf v1 supports timestamp with an int. Therefore, if not changing, the metrics will not show up in CloudWatchMetrics. Updating the version and timestamp to be a string instead of integer

Link to tracking Issue:
N/A

Testing:
There are two tests I have done:

  • By running go test with awsemfexporter, the result is
khanhntd@88665a0eaf54 awsemfexporter % go test ./...
ok      github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter       2.409s
  • By using the following configuration for awsemfexporter with prometheus, here is the following result:
    image

Documentation:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html#CloudWatch_Embedded_Metric_Format_Specification_structure

@khanhntd khanhntd changed the title Emf Add support AWS Embedded Metric Format Version 0 Apr 4, 2023
@khanhntd khanhntd marked this pull request as ready for review April 4, 2023 21:25
ymtaye
ymtaye approved these changes Apr 4, 2023
View reviewed changes
Copy link

@ymtaye ymtaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

jefchien
jefchien approved these changes Apr 4, 2023
View reviewed changes
exporter/awsemfexporter/metric_translator.go
@@ -405,8 +405,8 @@ func translateCWMetricToEMF(cWMetric *cWMetrics, config *Config) *cwlogs.Event {
"Timestamp": 1668387032641
}
*/
fieldMap["Version"] = 0
fieldMap["Timestamp"] = cWMetric.timestampMs
fieldMap["Version"] = "0"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: If you're going to have a comment with an example above, keep it updated with your changes. Make version and timestamp strings.

@khanhntd khanhntd merged commit ee1e4f1 into main Apr 5, 2023
@khanhntd khanhntd deleted the emf branch April 5, 2023 00:39
@khanhntd khanhntd restored the emf branch April 5, 2023 16:37
@sky333999 sky333999 deleted the emf branch April 27, 2023 23:33
sky333999 pushed a commit that referenced this pull request Oct 28, 2024
@daidokoro @kentquirk @TylerHelmuth
[processor/transform] Add Conversion Function to OTTL for Exponential…
74b1048 
… Histo --> Histogram (open-telemetry#33824)

## Description

This PR adds a custom metric function to the transformprocessor to
convert exponential histograms to explicit histograms.

Link to tracking issue: Resolves open-telemetry#33827

**Function Name**
```
convert_exponential_histogram_to_explicit_histogram
```

**Arguments:**

- `distribution` (_upper, midpoint, uniform, random_)
- `ExplicitBoundaries: []float64`

**Usage example:**

```yaml
processors:
  transform:
    error_mode: propagate
    metric_statements:
    - context: metric
      statements:
        - convert_exponential_histogram_to_explicit_histogram("random", [10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0]) 
```

**Converts:**

```
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: response_time
     -> Description: 
     -> Unit: 
     -> DataType: ExponentialHistogram
     -> AggregationTemporality: Delta
ExponentialHistogramDataPoints #0
Data point attributes:
     -> metric_type: Str(timing)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-07-31 09:35:25.212037 +0000 UTC
Count: 44
Sum: 999.000000
Min: 40.000000
Max: 245.000000
Bucket (32.000000, 64.000000], Count: 10
Bucket (64.000000, 128.000000], Count: 22
Bucket (128.000000, 256.000000], Count: 12
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
```

**To:**

```
Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: response_time
     -> Description: 
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: Delta
HistogramDataPoints #0
Data point attributes:
     -> metric_type: Str(timing)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-07-30 21:37:07.830902 +0000 UTC
Count: 44
Sum: 999.000000
Min: 40.000000
Max: 245.000000
ExplicitBounds #0: 10.000000
ExplicitBounds #1: 20.000000
ExplicitBounds #2: 30.000000
ExplicitBounds #3: 40.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 60.000000
ExplicitBounds #6: 70.000000
ExplicitBounds #7: 80.000000
ExplicitBounds #8: 90.000000
ExplicitBounds #9: 100.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 2
Buckets #4, Count: 5
Buckets #5, Count: 0
Buckets #6, Count: 3
Buckets #7, Count: 7
Buckets #8, Count: 2
Buckets #9, Count: 4
Buckets #10, Count: 21
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
```

### Testing

- Several unit tests have been created. We have also tested by ingesting
and converting exponential histograms from the `statsdreceiver` as well
as directly via the `otlpreceiver` over grpc over several hours with a
large amount of data.

- We have clients that have been running this solution in production for
a number of weeks.

### Readme description:

### convert_exponential_hist_to_explicit_hist

`convert_exponential_hist_to_explicit_hist([ExplicitBounds])`

the `convert_exponential_hist_to_explicit_hist` function converts an
ExponentialHistogram to an Explicit (_normal_) Histogram.

`ExplicitBounds` is represents the list of bucket boundaries for the new
histogram. This argument is __required__ and __cannot be empty__.

__WARNING:__

The process of converting an ExponentialHistogram to an Explicit
Histogram is not perfect and may result in a loss of precision. It is
important to define an appropriate set of bucket boundaries to minimize
this loss. For example, selecting Boundaries that are too high or too
low may result histogram buckets that are too wide or too narrow,
respectively.

---------

Co-authored-by: Kent Quirk <[email protected]>
Co-authored-by: Tyler Helmuth <[email protected]>
zzhlogin pushed a commit to zzhlogin/opentelemetry-collector-contrib-aws that referenced this pull request Dec 19, 2024
@shivanthzen @mx-psi
Add histogram to telemetrygen (open-telemetry#36322)
9cb09d5 
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Generates simple histograms using telemetrygen
<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes

<!--Describe what testing was performed and which tests were added.-->
#### Testing
Test with a local otel collector with debug output

```
bin/telemetrygen metrics --metrics 5  --otlp-http --otlp-endpoint "localhost:4318"    --metric-type Histogram --otlp-insecure 
```
Output from debug Exporter: 
```
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.633365 +0000 UTC
Timestamp: 2024-11-13 16:22:51.633367 +0000 UTC
Count: 0
Sum: 0.000000
ExplicitBounds #0: 0.000000
ExplicitBounds amazon-contributing#1: 1.000000
ExplicitBounds amazon-contributing#2: 2.000000
ExplicitBounds amazon-contributing#3: 3.000000
ExplicitBounds amazon-contributing#4: 4.000000
Buckets #0, Count: 0
Buckets amazon-contributing#1, Count: 0
Buckets amazon-contributing#2, Count: 0
Buckets amazon-contributing#3, Count: 0
Buckets amazon-contributing#4, Count: 0
ResourceMetrics amazon-contributing#1
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.639942 +0000 UTC
Timestamp: 2024-11-13 16:22:51.639942 +0000 UTC
Count: 1
Sum: 1.000000
ExplicitBounds #0: 0.000000
ExplicitBounds amazon-contributing#1: 1.000000
ExplicitBounds amazon-contributing#2: 2.000000
ExplicitBounds amazon-contributing#3: 3.000000
ExplicitBounds amazon-contributing#4: 4.000000
Buckets #0, Count: 0
Buckets amazon-contributing#1, Count: 1
Buckets amazon-contributing#2, Count: 0
Buckets amazon-contributing#3, Count: 0
Buckets amazon-contributing#4, Count: 0
ResourceMetrics amazon-contributing#2
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.6404 +0000 UTC
Timestamp: 2024-11-13 16:22:51.640401 +0000 UTC
Count: 2
Sum: 4.000000
ExplicitBounds #0: 0.000000
ExplicitBounds amazon-contributing#1: 1.000000
ExplicitBounds amazon-contributing#2: 2.000000
ExplicitBounds amazon-contributing#3: 3.000000
ExplicitBounds amazon-contributing#4: 4.000000
Buckets #0, Count: 0
Buckets amazon-contributing#1, Count: 1
Buckets amazon-contributing#2, Count: 0
Buckets amazon-contributing#3, Count: 1
Buckets amazon-contributing#4, Count: 0
ResourceMetrics amazon-contributing#3
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.640729 +0000 UTC
Timestamp: 2024-11-13 16:22:51.640729 +0000 UTC
Count: 3
Sum: 3.000000
ExplicitBounds #0: 0.000000
ExplicitBounds amazon-contributing#1: 1.000000
ExplicitBounds amazon-contributing#2: 2.000000
ExplicitBounds amazon-contributing#3: 3.000000
ExplicitBounds amazon-contributing#4: 4.000000
Buckets #0, Count: 1
Buckets amazon-contributing#1, Count: 1
Buckets amazon-contributing#2, Count: 1
Buckets amazon-contributing#3, Count: 0
Buckets amazon-contributing#4, Count: 0
ResourceMetrics amazon-contributing#4
Resource SchemaURL: https://opentelemetry.io/schemas/1.13.0
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: gen
     -> Description:
     -> Unit:
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
StartTimestamp: 2024-11-13 16:22:50.641073 +0000 UTC
Timestamp: 2024-11-13 16:22:51.641073 +0000 UTC
Count: 4
Sum: 12.000000
ExplicitBounds #0: 0.000000
ExplicitBounds amazon-contributing#1: 1.000000
ExplicitBounds amazon-contributing#2: 2.000000
ExplicitBounds amazon-contributing#3: 3.000000
ExplicitBounds amazon-contributing#4: 4.000000
Buckets #0, Count: 0
Buckets amazon-contributing#1, Count: 0
Buckets amazon-contributing#2, Count: 1
Buckets amazon-contributing#3, Count: 2
Buckets amazon-contributing#4, Count: 1
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
```
<!--Describe the documentation added.-->
#### Documentation

<!--Please delete paragraphs that you did not use before submitting.-->

---------

Co-authored-by: Pablo Baeyens <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jefchien jefchien jefchien approved these changes

@ymtaye ymtaye ymtaye approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@khanhntd @ymtaye @jefchien @matianjun1