Add OTLP marshalling support for kinesis exporter #1

irisgve · 2020-10-13T00:12:15Z

Description:

Adding a Marshaller interface that allows encoding to different export formats
Add an OTLP marshaller with support for metrics
NOTE: we could add these otlp marshallers upstream in the core collector i.e. within https://github.com/open-telemetry/opentelemetry-collector/tree/v0.12.0/exporter/exporterhelper since this is essentially similar to the kafka exporter marshallers (potentially part of the submitting upstream PRs effort)

Testing:

Added unit tests for marshaller and otlp marshaller

Excaleo

LGTM, just remove the changes from factory.go since that'll come later and double check whether we need the Message struct to wrap the byte array.

exporter/kinesisexporter/factory.go

exporter/kinesisexporter/marshaller.go

Adding a Marshaller interface that allows encoding to different export formats Add an OTLP marshaller with support for metrics Co-authored-by: Iris Grace Endozo <[email protected]>

…ng (#9) * Add OTLP marshalling support for kinesis exporter (#1) Adding a Marshaller interface that allows encoding to different export formats Add an OTLP marshaller with support for metrics Co-authored-by: Iris Grace Endozo <[email protected]> * Swapped kinesis opencensus producer with omnition producer * Moved AWS/Kinesis setup to exporter.go * Delete unused Message struct (#3) Co-authored-by: Iris Grace Endozo <[email protected]> * Integrated marshaller into trace path * Adding tests for exporter and trace marshaling * Support sending out metrics to kinesis (#5) Co-authored-by: Iris Grace Endozo <[email protected]> * Added Start function to factory and logging of producer failures * Removed var * Changed order of config variables * Increase test coverage + some cleanups (#8) * Add tests + private methods * Add comments + fixes Co-authored-by: Iris Grace Endozo <[email protected]> * Add README.mdgst * Added parallel testing and import order fixes * Added pointer receivers * Opentelemetry to OpenTelemetry * Unused imports * Add license and fix import order * Fixed license * Fix linting shadowing error * Change readme descriptions * Use sensible values in example config * Added context validation in exporter * Move invalid context as const Co-authored-by: Iris Grace Endozo <[email protected]> Co-authored-by: Raymond Wang <[email protected]>

* Restructure buildCWMetric logic (#1) * Restructure code to remove duplicated logic * Update format * Improve function and variable names * Extract logic for dimension creation and add test * Implement minor fixes * Remove changes to go.sum * Implement tests for getCWMetrics * Implement tests for buildCWMetric * Format metric_translator_test.go * Run with gofmt -s * Disregard ordering of dimensions in test case * Perform dimension equality checking as a helper function

…ng (#9) * Add OTLP marshalling support for kinesis exporter (#1) Adding a Marshaller interface that allows encoding to different export formats Add an OTLP marshaller with support for metrics Co-authored-by: Iris Grace Endozo <[email protected]> * Swapped kinesis opencensus producer with omnition producer * Moved AWS/Kinesis setup to exporter.go * Delete unused Message struct (#3) Co-authored-by: Iris Grace Endozo <[email protected]> * Integrated marshaller into trace path * Adding tests for exporter and trace marshaling * Support sending out metrics to kinesis (#5) Co-authored-by: Iris Grace Endozo <[email protected]> * Added Start function to factory and logging of producer failures * Removed var * Changed order of config variables * Increase test coverage + some cleanups (#8) * Add tests + private methods * Add comments + fixes Co-authored-by: Iris Grace Endozo <[email protected]> * Add README.mdgst * Added parallel testing and import order fixes * Added pointer receivers * Opentelemetry to OpenTelemetry * Unused imports * Add license and fix import order * Fixed license * Fix linting shadowing error * Change readme descriptions * Use sensible values in example config * Added context validation in exporter * Move invalid context as const Co-authored-by: Iris Grace Endozo <[email protected]> Co-authored-by: Raymond Wang <[email protected]>

…etry#4626) Adds a Cloud Foundry metric receiver which reads metrics from Cloud Foundry Reverse Log Proxy Gateway. More details available in the `README.md`. `make gotidy` seems to have made plenty of subtle changes to `go.sum` files, not sure if this is normal. This PR contains the overall structure, documentation, implementation for config and factory, but does NOT contain the implementation of the receiver and does not enable the component, as that will come in separate PRs later. **Link to tracking Issue:** open-telemetry#5320 **Testing:** Unit tests. Manual testing was performed against Tanzu Application Service (TAS) versions 2.7, 2.8 and 2.11. Considered adding an integration test with mocked HTTP servers acting as endpoints where the HTTP server would provide a constant response (prerecorded from the real TAS traffic), but not sure if mocks would make more sense. **Documentation:** `README.md` and `doc.go` for the new receiver module were added.

…y#9224) * add vcenter vSAN collection * checkpoint on getting property collection working * checkpoint before integration test * dual receivers under root receiver pointer * checkpoint before updated mdatagen * use syslog receiver rather than tcplogreceiver * getting more performance counter refinements * remove unneccessary component addition * try to fix go.mod resolution issues * try to fix go.mod resolution issues pt 2 * addlicense * fix go.mod by fixing require directive * add readme for metrics * update readme * fix go.mod referring nonexistent version * add performance manager tests * more tests * add more attributes to virtual machines and host systems * add more attributes to virtual machines and host systems * spike changelog entry * fix go.mod in both places * fix go.mod in configschema * add // import github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vmwarevcenterreceiver to imports * add quotations * add to receiver lifecycle * remove extra go generate direction * fix typo of utilizaiton in metric description * small changes to interval id in performance queries to be more consistent * PR feedback including omitting company name prefix * PR feedback to not fail starting the component on potential network failures * minor grammar correction in vcenter readme * update expected metrics * update host_effective attribute value * remove PerformanceInterval customizability * add to codeowners * fix indentation on merge conflict * fix changelog entry place so its in the new components section * update to be on 0.49.0 of the collector' * add PR number to changelog * regenerate with newer version of mdatagen * move error log if unable to connect on start to receiver.Start() rather than scraper.Start() * fix test cases from last commit * minor update to config with tests * fix metric description * use utc for host vsan collection as well * update comments of public facing methods * return errors on getting clusters to the scraper errors * PR feedback #1 * instantiate new client if client is nil * update all descriptions to have punctuation * three more descs * move ensureReceiver up to once we validated as a config * some more PR feedback * looking into race conditions * run go tidy * fix import order and remove unneccessary mutex * remove mutex from struct * refactor client to responsible for knowing if the vsan endpoints are reachable * fix integration test referencing old var * change metrics.metrics => metrics.settings, update client pr feedback * remove vSAN collection temporarily * remove extra metric attributes for vSAN * remove vsan specific variables * clean up host PerfCounter disk latency metrics and fix some descriptions to better reflect interval * add 20s interval to extended documentation as needed * mdatagen fixes * add integration test metric scrape * fix import order * go up to 0.49.1 * gotidy * add replace directive for semconv * gotidy fixes * fix component not being on 0.50.0 * update to v0.50.1-0.20220429151328-041f39835df7 * use newer mdatagen * remove any logging functionality change && update documentation * fix integration test from flattening of config * fix scraper start not erroring if connection cannot be established * make scrapertest less flaky * format test json * Apply suggestions from code review Co-authored-by: Daniel Jaglowski <[email protected]> * adjust metric definition for vcenter.host.disk.throughput * remove comment and move pm level 2 metrics to appropriate section * try to be respective of datacenters * fix only vCenter server functionality * try building out a mock server for test coverage * make goporto * fix build issues * use latest mdatagen * add newlines to ends of xml recordings * fix integration test * moved around scrapererrors because now the receiver is datacenter dependent * try and do an audit of performance metrics and requests/responses * update testdata with correct units * make tidy * make tidy * update collector version * fix local testing code including modules * remove deprecated use of commonponenterror * pr feedback; add method of collection recording, return poweredOn/poweredOff VMs * remove content.json * fix description change in scraper_test.go * update collector version * bump replaced module; rebuild load tests * fix alibaba version auto localizing Co-authored-by: Daniel Jaglowski <[email protected]>

…emetry#24676) **Description:** The metadata.yml for the SSH check receiver currently documents a resource attribute containing the SSH endpoint but this is not emitted. This PR updates the receiver to include this resource attribute. **Link to tracking Issue:** open-telemetry#24441 **Testing:** Example collector config: ```yaml receivers: sshcheck: endpoint: 13.245.150.131:22 username: ec2-user key_file: /Users/dewald.dejager/.ssh/sandbox.pem collection_interval: 15s known_hosts: /Users/dewald.dejager/.ssh/known_hosts ignore_host_key: false resource_attributes: "ssh.endpoint": enabled: true exporters: logging: verbosity: detailed prometheus: endpoint: 0.0.0.0:8081 resource_to_telemetry_conversion: enabled: true service: pipelines: metrics: receivers: [sshcheck] exporters: [logging, prometheus] ``` The log output looks like this: ``` 2023-07-30T16:52:38.724+0200 info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 2, "data points": 2} 2023-07-30T16:52:38.724+0200 info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> ssh.endpoint: Str(13.245.150.131:22) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope otelcol/sshcheckreceiver 0.82.0-dev Metric #0 Descriptor: -> Name: sshcheck.duration -> Description: Measures the duration of SSH connection. -> Unit: ms -> DataType: Gauge NumberDataPoints #0 StartTimestamp: 2023-07-30 14:52:22.381672 +0000 UTC Timestamp: 2023-07-30 14:52:38.404003 +0000 UTC Value: 319 Metric #1 Descriptor: -> Name: sshcheck.status -> Description: 1 if the SSH client successfully connected, otherwise 0. -> Unit: 1 -> DataType: Sum -> IsMonotonic: false -> AggregationTemporality: Cumulative NumberDataPoints #0 StartTimestamp: 2023-07-30 14:52:22.381672 +0000 UTC Timestamp: 2023-07-30 14:52:38.404003 +0000 UTC Value: 1 ``` And the Prometheus metrics look like this: ``` # HELP sshcheck_duration Measures the duration of SSH connection. # TYPE sshcheck_duration gauge sshcheck_duration{ssh_endpoint="13.245.150.131:22"} 311 # HELP sshcheck_status 1 if the SSH client successfully connected, otherwise 0. # TYPE sshcheck_status gauge sshcheck_status{ssh_endpoint="13.245.150.131:22"} 1 ```

) **Description:** Adding command line argument `--status-code` to `telemetrygen traces`, which accepts `(Unset,Error,Ok)` (case sensitive) or the enum equivalent of `(0,1,2)`. Running ```shell telemetrygen traces --otlp-insecure --traces 1 --status-code 1 ``` against a minimal local collector yields ```txt 2023-07-29T21:27:57.862+0100 info ResourceSpans #0 Resource SchemaURL: https://opentelemetry.io/schemas/1.4.0 Resource attributes: -> service.name: Str(telemetrygen) ScopeSpans #0 ScopeSpans SchemaURL: InstrumentationScope telemetrygen Span #0 Trace ID : f6dc4be32c78b9999c69d504a79e68c1 Parent ID : 4e2cd6e0e90cf2ea ID : 20835413e32d26a5 Name : okey-dokey Kind : Server Start time : 2023-07-29 20:27:57.861602 +0000 UTC End time : 2023-07-29 20:27:57.861726 +0000 UTC Status code : Error Status message : Attributes: -> net.peer.ip: Str(1.2.3.4) -> peer.service: Str(telemetrygen-client) Span #1 Trace ID : f6dc4be32c78b9999c69d504a79e68c1 Parent ID : ID : 4e2cd6e0e90cf2ea Name : lets-go Kind : Client Start time : 2023-07-29 20:27:57.861584 +0000 UTC End time : 2023-07-29 20:27:57.861726 +0000 UTC Status code : Error Status message : Attributes: -> net.peer.ip: Str(1.2.3.4) -> peer.service: Str(telemetrygen-server) ``` and similarly (the string version) ```shell telemetrygen traces --otlp-insecure --traces 1 --status-code '"Ok"' ``` produces ```txt Resource SchemaURL: https://opentelemetry.io/schemas/1.4.0 Resource attributes: -> service.name: Str(telemetrygen) ScopeSpans #0 ScopeSpans SchemaURL: InstrumentationScope telemetrygen Span #0 Trace ID : dfd830da170acfe567b12f87685d7917 Parent ID : 8e15b390dc6a1ccc ID : 165c300130532072 Name : okey-dokey Kind : Server Start time : 2023-07-29 20:29:16.026965 +0000 UTC End time : 2023-07-29 20:29:16.027089 +0000 UTC Status code : Ok Status message : Attributes: -> net.peer.ip: Str(1.2.3.4) -> peer.service: Str(telemetrygen-client) Span #1 Trace ID : dfd830da170acfe567b12f87685d7917 Parent ID : ID : 8e15b390dc6a1ccc Name : lets-go Kind : Client Start time : 2023-07-29 20:29:16.026956 +0000 UTC End time : 2023-07-29 20:29:16.027089 +0000 UTC Status code : Ok Status message : Attributes: -> net.peer.ip: Str(1.2.3.4) -> peer.service: Str(telemetrygen-server) ``` The default is `Unset` which is the current behaviour. **Link to tracking Issue:** 24286 **Testing:** Added unit tests which covers both valid and invalid inputs. **Documentation:** Command line arguments are self documenting via the usage info in the flag. Co-authored-by: Pablo Baeyens <[email protected]>

irisgve force-pushed the kinesisexporter-metrics branch 9 times, most recently from 0e37979 to da4d255 Compare October 13, 2020 00:37

irisgve marked this pull request as ready for review October 13, 2020 00:40

irisgve requested a review from Excaleo October 13, 2020 00:54

irisgve self-assigned this Oct 13, 2020

Excaleo requested changes Oct 13, 2020

View reviewed changes

exporter/kinesisexporter/factory.go Outdated Show resolved Hide resolved

exporter/kinesisexporter/marshaller.go Show resolved Hide resolved

Add OTLP marshalling support for kinesis exporter

1857b6e

irisgve force-pushed the kinesisexporter-metrics branch from da4d255 to 1857b6e Compare October 13, 2020 02:35

Change up encoding + fixes

227aaa5

irisgve force-pushed the kinesisexporter-metrics branch from d28b8a5 to 227aaa5 Compare October 13, 2020 02:43

irisgve added 3 commits October 13, 2020 13:44

Rename to camel

9e9f1e1

Move back to export format

0723198

gofmt

63ca50c

irisgve requested a review from Excaleo October 13, 2020 02:46

Excaleo approved these changes Oct 13, 2020

View reviewed changes

irisgve merged commit 769c57e into kinesisexporter Oct 13, 2020

irisgve deleted the kinesisexporter-metrics branch October 13, 2020 02:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add OTLP marshalling support for kinesis exporter #1

Add OTLP marshalling support for kinesis exporter #1

irisgve commented Oct 13, 2020 •

edited

Loading

Excaleo left a comment

Add OTLP marshalling support for kinesis exporter #1

Add OTLP marshalling support for kinesis exporter #1

Conversation

irisgve commented Oct 13, 2020 • edited Loading

Excaleo left a comment

Choose a reason for hiding this comment

irisgve commented Oct 13, 2020 •

edited

Loading