[exporter/opensearch] Send logs to Opensearch #26475

jaehnri · 2023-09-05T20:41:16Z

Description:

Implementation of exporter to OpenSearch using opensearch-go library. As of now, this PR was heavily inspired by https://github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter.

By default, requests sent adhere to the OpenSearch Catalog schema for logs, but allows users to export using the Elastic Common Schema as well.

This PR also:

enables users to define the bulk_action between create and index
enables users to define the logs index without necessarily adhering to the new index naming conventions through the LogsIndex config.

Tracking Issue:

23611

Testing:

Integration

Successful round-trip to HTTP endpoint,
Permanent error during round-trip,
Retryable error response for first request, followed by successful response on retry,
Two retriable error responses, followed by successful response on second retry.

Manual

Authentication using configtls.TLSSetting (ca_file, cert_file, key_file)
Tested in EKS and K3s clusters running opni.

github-actions · 2023-09-28T05:20:11Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

jaehnri · 2023-10-02T19:37:05Z

@Aneurysm9 @bryan-aguilar friendly pinging for review here so the PR isn't closed for inactivity. Thanks!

bryan-aguilar · 2023-10-02T20:18:16Z

Pinging @MitchellGale @MaxKsyunz @YANG-DB as opensearch exporter codeowners.

bryan-aguilar · 2023-10-02T20:18:44Z

i have ran the integration tests, please check their success status and respond to any failures.

jaehnri · 2023-10-02T21:56:25Z

There were some issues regarding the changelog that should be addressed now. Can you run them again, please? Thanks!

mikaylathompson · 2023-10-13T21:21:39Z

This PR would be extremely useful for our use case as the client in the elasticsearch exporter blocks working with opensearch clusters.

If there's anything I can do to help push this PR across the finish line, please let me know. It looks like it's code complete and the tests are passing.

sumobrian · 2023-10-13T21:54:46Z

This PR is exactly what we need to adopt this package for an AWS solutions project. @Aneurysm9 is there anything that is preventing this PR from being merged?

bryan-aguilar · 2023-10-13T22:28:12Z

This PR is still awaiting approval from code owners @sumobrian

atoulme · 2023-10-22T22:08:38Z

Tagging code owners: @Aneurysm9 @MitchellGale @MaxKsyunz @YANG-DB

YANG-DB · 2023-10-22T23:40:54Z

Thanks for your support
We will push this to get merged ASAP

jaehnri · 2023-10-23T10:51:21Z

@atoulme @YANG-DB @bryan-aguilar rebased to fix conflicts. It should be good to go now.

YANG-DB · 2023-10-23T17:33:32Z

@Aneurysm9 - can you please review
Thanks 👍

Aneurysm9

I have reviewed primarily for structure and consistency with collector style and expectations. The notes here are not blocking as they do not affect the public API, but should be addressed before moving out of the development stage.

Aneurysm9 · 2023-10-23T17:56:01Z

exporter/opensearchexporter/encoder.go

+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package opensearchexporter // import "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opensearchexporter"


Nothing here is exported, which is a good thing. Please consider moving this to an internal package to minimize the likelihood of accidental changes to visibility increasing the public API surface area.

Aneurysm9 · 2023-10-23T17:58:37Z

exporter/opensearchexporter/factory.go

+	return exporterhelper.NewLogsExporter(ctx, set, cfg,
+		le.pushLogData,
+		exporterhelper.WithStart(le.Start),
+		exporterhelper.WithCapabilities(consumer.Capabilities{MutatesData: true}),


Is this necessary? If the exporter can avoid mutating data it can save a copy operation if it is used in a pipeline with multiple exporters.

Aneurysm9 · 2023-10-23T18:01:04Z

exporter/opensearchexporter/internal/objmodel/objmodel.go

+// Ingest Node is used. But either way, we try to present only well formed
+// document to OpenSearch.
+
+// nolint:errcheck


Can this be scoped to specific statements? Even if that requires duplication it's preferable to not allowing further unchecked errors to be introduced without discussion.

Aneurysm9 · 2023-10-23T18:02:23Z

exporter/opensearchexporter/log_bulk_indexer.go

+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package opensearchexporter // import "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opensearchexporter"


This may also be a candidate for moving to an internal package.

Aneurysm9 · 2023-10-23T18:04:46Z

exporter/opensearchexporter/sso_model.go

+	Links        []ssoSpanLinks    `json:"links,omitempty"`
+	Name         string            `json:"name"`
+	ParentSpanID string            `json:"parentSpanId"`
+	Resource     map[string]string `json:"resource,omitempty"`


Resource attributes can be more than just strings. How does this handle non-string-valued resource attributes?

jaehnri · 2023-10-24T19:19:00Z

For note: I will address the comments in another PR, including a refactor in the bulk indexer structure.

YANG-DB · 2023-10-26T23:16:09Z

Hi
Since the PR is labeled ready for merge - does it need another manual signoff or is it merged automatically??

bryan-aguilar · 2023-10-26T23:20:55Z

An @open-telemetry/collector-contrib-maintainer will come by and merge it when they get a chance. It is not done automatically.

bryan-aguilar · 2023-10-26T23:21:58Z

@jaehnri, please address merge conflicts though as that will prevent a maintainer from merging it in.

Signed-off-by: João Henri <[email protected]>

…space Signed-off-by: João Henri <[email protected]>

Signed-off-by: João Henri <[email protected]>

…bility Signed-off-by: João Henri <[email protected]>

Signed-off-by: João Henri <[email protected]>

jaehnri · 2023-10-26T23:41:16Z

@bryan-aguilar, done it.

mx-psi · 2023-10-27T09:05:09Z

I have reviewed primarily for structure and consistency with collector style and expectations. The notes here are not blocking as they do not affect the public API, but should be addressed before moving out of the development stage.

@jaehnri can you create a tracking issue for addressing the changes in @Aneurysm9's review? I will merge this now, but please link the issue here once you have created it :)

This fixes a merge conflict between #28594 and #26475

jaehnri · 2023-10-27T14:26:53Z

@mx-psi, here it is: #28666

## Description: <Describe what has changed.>  Implementation of exporter to OpenSearch using opensearch-go library. As of now, this PR was heavily inspired by https://github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter. By default, requests sent adhere to the OpenSearch Catalog [schema for logs](https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability/logs), but allows users to export using the Elastic Common Schema as well. This PR also: - enables users to define the `bulk_action` between `create` and `index` - enables users to define the logs index without necessarily adhering to the new [index naming conventions](opensearch-project/observability#1405) through the `LogsIndex` config. ## Tracking Issue: [23611](open-telemetry#23611) ## Testing: <Describe what testing was performed and which tests were added.> ### Integration - Successful round-trip to HTTP endpoint, - Permanent error during round-trip, - Retryable error response for first request, followed by successful response on retry, - Two retriable error responses, followed by successful response on second retry. ### Manual - Authentication using `configtls.TLSSetting` (`ca_file`, `cert_file`, `key_file`) - Tested in EKS and K3s clusters running [opni](https://github.com/rancher/opni). --------- Signed-off-by: João Henri <[email protected]> Signed-off-by: João Henri <[email protected]>

…8662) This fixes a merge conflict between open-telemetry#28594 and open-telemetry#26475

## Description: <Describe what has changed.>  Implementation of exporter to OpenSearch using opensearch-go library. As of now, this PR was heavily inspired by https://github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter. By default, requests sent adhere to the OpenSearch Catalog [schema for logs](https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability/logs), but allows users to export using the Elastic Common Schema as well. This PR also: - enables users to define the `bulk_action` between `create` and `index` - enables users to define the logs index without necessarily adhering to the new [index naming conventions](opensearch-project/observability#1405) through the `LogsIndex` config. ## Tracking Issue: [23611](open-telemetry#23611) ## Testing: <Describe what testing was performed and which tests were added.> ### Integration - Successful round-trip to HTTP endpoint, - Permanent error during round-trip, - Retryable error response for first request, followed by successful response on retry, - Two retriable error responses, followed by successful response on second retry. ### Manual - Authentication using `configtls.TLSSetting` (`ca_file`, `cert_file`, `key_file`) - Tested in EKS and K3s clusters running [opni](https://github.com/rancher/opni). --------- Signed-off-by: João Henri <[email protected]> Signed-off-by: João Henri <[email protected]>

…8662) This fixes a merge conflict between open-telemetry#28594 and open-telemetry#26475

jaehnri requested a review from Aneurysm9 as a code owner September 5, 2023 20:41

jaehnri requested a review from a team September 5, 2023 20:41

github-actions bot assigned Aneurysm9 Sep 5, 2023

github-actions bot added the exporter/opensearch label Sep 5, 2023

jaehnri force-pushed the opensearch-logs-exporter branch from 5c1bae6 to 2e579e0 Compare September 6, 2023 17:40

MaxKsyunz mentioned this pull request Sep 12, 2023

[exporter/opensearch] Add OpenSearchExporter to otelcontribcol #24668

Merged

github-actions bot added the Stale label Sep 28, 2023

jaehnri force-pushed the opensearch-logs-exporter branch 2 times, most recently from 54e12e8 to 59e1793 Compare October 2, 2023 19:30

bryan-aguilar removed the Stale label Oct 2, 2023

jaehnri force-pushed the opensearch-logs-exporter branch 2 times, most recently from e714108 to 9001616 Compare October 2, 2023 21:53

YANG-DB approved these changes Oct 22, 2023

View reviewed changes

jaehnri force-pushed the opensearch-logs-exporter branch from 9001616 to cf3ac2c Compare October 23, 2023 10:50

Aneurysm9 approved these changes Oct 23, 2023

View reviewed changes

Aneurysm9 added the ready to merge Code review completed; ready to merge by maintainers label Oct 23, 2023

github-actions bot mentioned this pull request Oct 24, 2023

Weekly Report: 2023-10-17 - 2023-10-24 #28557

Closed

jaehnri and others added 12 commits October 26, 2023 20:31

[exporter/opensearch]: Add first working version of log exporter

a7f981b

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch]: Add tests to config and factory

4b89411

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch]: Add integration tests to the log exporter

ecf5484

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch]: Make bulk action a configuration

2e5c74c

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch]: Make log exporter follow ss4o

dc5b547

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch]: Update README with logs and bulkaction

7cb5ac0

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch]: Generate LogsStability level

475d68e

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch] Support multiple encoding schemas for logs

232520e

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch] Use ss4o as the default encoding schema

396a6e3

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch] Allow use of logs_index instead of dataset-name…

edf868a

…space Signed-off-by: João Henri <[email protected]>

[exporter/opensearch]: Fix SSO schema selection

14bf8fa

Signed-off-by: João Henri <[email protected]>

[exporter/opensearch]: Add changelog for opensearch exporter log capa…

85b840c

…bility Signed-off-by: João Henri <[email protected]>

jaehnri force-pushed the opensearch-logs-exporter branch from 1d2d4a9 to 85b840c Compare October 26, 2023 23:32

[exporter/opensearch]: Use errors instead of multierr

7b9a5c3

Signed-off-by: João Henri <[email protected]>

mx-psi merged commit e8b0e2a into open-telemetry:main Oct 27, 2023
82 checks passed

github-actions bot added this to the next release milestone Oct 27, 2023

djaglowski mentioned this pull request Oct 27, 2023

[chore] Fix merge conflict with golden package move #28662

Merged

djaglowski added a commit that referenced this pull request Oct 27, 2023

[chore] Fix merge conflict with golden package move (#28662)

1f646ab

This fixes a merge conflict between #28594 and #26475

jmsnll pushed a commit to jmsnll/opentelemetry-collector-contrib that referenced this pull request Nov 12, 2023

[chore] Fix merge conflict with golden package move (open-telemetry#2…

daf7a31

…8662) This fixes a merge conflict between open-telemetry#28594 and open-telemetry#26475

RoryCrispin pushed a commit to ClickHouse/opentelemetry-collector-contrib that referenced this pull request Nov 24, 2023

[chore] Fix merge conflict with golden package move (open-telemetry#2…

ba08b5f

…8662) This fixes a merge conflict between open-telemetry#28594 and open-telemetry#26475

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter/opensearch] Send logs to Opensearch #26475

[exporter/opensearch] Send logs to Opensearch #26475

jaehnri commented Sep 5, 2023 •

edited

Loading

github-actions bot commented Sep 28, 2023

jaehnri commented Oct 2, 2023

bryan-aguilar commented Oct 2, 2023

bryan-aguilar commented Oct 2, 2023

jaehnri commented Oct 2, 2023

mikaylathompson commented Oct 13, 2023

sumobrian commented Oct 13, 2023

bryan-aguilar commented Oct 13, 2023

atoulme commented Oct 22, 2023 •

edited

Loading

YANG-DB commented Oct 22, 2023

jaehnri commented Oct 23, 2023

YANG-DB commented Oct 23, 2023

Aneurysm9 left a comment

Aneurysm9 Oct 23, 2023

Aneurysm9 Oct 23, 2023

Aneurysm9 Oct 23, 2023

Aneurysm9 Oct 23, 2023

Aneurysm9 Oct 23, 2023

jaehnri commented Oct 24, 2023

YANG-DB commented Oct 26, 2023

bryan-aguilar commented Oct 26, 2023

bryan-aguilar commented Oct 26, 2023

jaehnri commented Oct 26, 2023

mx-psi commented Oct 27, 2023

jaehnri commented Oct 27, 2023

[exporter/opensearch] Send logs to Opensearch #26475

[exporter/opensearch] Send logs to Opensearch #26475

Conversation

jaehnri commented Sep 5, 2023 • edited Loading

Description:

Tracking Issue:

Testing:

Integration

Manual

github-actions bot commented Sep 28, 2023

jaehnri commented Oct 2, 2023

bryan-aguilar commented Oct 2, 2023

bryan-aguilar commented Oct 2, 2023

jaehnri commented Oct 2, 2023

mikaylathompson commented Oct 13, 2023

sumobrian commented Oct 13, 2023

bryan-aguilar commented Oct 13, 2023

atoulme commented Oct 22, 2023 • edited Loading

YANG-DB commented Oct 22, 2023

jaehnri commented Oct 23, 2023

YANG-DB commented Oct 23, 2023

Aneurysm9 left a comment

Choose a reason for hiding this comment

Aneurysm9 Oct 23, 2023

Choose a reason for hiding this comment

Aneurysm9 Oct 23, 2023

Choose a reason for hiding this comment

Aneurysm9 Oct 23, 2023

Choose a reason for hiding this comment

Aneurysm9 Oct 23, 2023

Choose a reason for hiding this comment

Aneurysm9 Oct 23, 2023

Choose a reason for hiding this comment

jaehnri commented Oct 24, 2023

YANG-DB commented Oct 26, 2023

bryan-aguilar commented Oct 26, 2023

bryan-aguilar commented Oct 26, 2023

jaehnri commented Oct 26, 2023

mx-psi commented Oct 27, 2023

jaehnri commented Oct 27, 2023

jaehnri commented Sep 5, 2023 •

edited

Loading

atoulme commented Oct 22, 2023 •

edited

Loading