-
Notifications
You must be signed in to change notification settings - Fork 99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC] Observability indices naming standards & routing #1405
Comments
As I otherwise mentioned in opensearch-project/OpenSearch-Dashboards#3412.... Regarding index naming, I would like to see the naming structure updated to account for 'tenant' and 'version'. * Often RBAC rules are written in a fashion that clients, tenants, usergroups get access to a set of indexes matching NAME-*. namespace, especially where it would be placed in the index name, could be used in this fashion, but it would make writing RBAC rules a harder and I believe serves a slightly different function than what I am proposing ** As the schema will evolve it will be important to have versioning in the naming schema, which should be coupled with standard mapping definitions. As the schema evolves - so too would the mappings. As such a complete index (or per the proposal data-stream) name would be: |
## Description: <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Implementation of exporter to OpenSearch using opensearch-go library. As of now, this PR was heavily inspired by https://github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter. By default, requests sent adhere to the OpenSearch Catalog [schema for logs](https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability/logs), but allows users to export using the Elastic Common Schema as well. This PR also: - enables users to define the `bulk_action` between `create` and `index` - enables users to define the logs index without necessarily adhering to the new [index naming conventions](opensearch-project/observability#1405) through the `LogsIndex` config. ## Tracking Issue: [23611](#23611) ## Testing: <Describe what testing was performed and which tests were added.> ### Integration - Successful round-trip to HTTP endpoint, - Permanent error during round-trip, - Retryable error response for first request, followed by successful response on retry, - Two retriable error responses, followed by successful response on second retry. ### Manual - Authentication using `configtls.TLSSetting` (`ca_file`, `cert_file`, `key_file`) - Tested in EKS and K3s clusters running [opni](https://github.com/rancher/opni). --------- Signed-off-by: João Henri <[email protected]> Signed-off-by: João Henri <[email protected]>
## Description: <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Implementation of exporter to OpenSearch using opensearch-go library. As of now, this PR was heavily inspired by https://github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter. By default, requests sent adhere to the OpenSearch Catalog [schema for logs](https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability/logs), but allows users to export using the Elastic Common Schema as well. This PR also: - enables users to define the `bulk_action` between `create` and `index` - enables users to define the logs index without necessarily adhering to the new [index naming conventions](opensearch-project/observability#1405) through the `LogsIndex` config. ## Tracking Issue: [23611](open-telemetry#23611) ## Testing: <Describe what testing was performed and which tests were added.> ### Integration - Successful round-trip to HTTP endpoint, - Permanent error during round-trip, - Retryable error response for first request, followed by successful response on retry, - Two retriable error responses, followed by successful response on second retry. ### Manual - Authentication using `configtls.TLSSetting` (`ca_file`, `cert_file`, `key_file`) - Tested in EKS and K3s clusters running [opni](https://github.com/rancher/opni). --------- Signed-off-by: João Henri <[email protected]> Signed-off-by: João Henri <[email protected]>
## Description: <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> Implementation of exporter to OpenSearch using opensearch-go library. As of now, this PR was heavily inspired by https://github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter. By default, requests sent adhere to the OpenSearch Catalog [schema for logs](https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability/logs), but allows users to export using the Elastic Common Schema as well. This PR also: - enables users to define the `bulk_action` between `create` and `index` - enables users to define the logs index without necessarily adhering to the new [index naming conventions](opensearch-project/observability#1405) through the `LogsIndex` config. ## Tracking Issue: [23611](open-telemetry#23611) ## Testing: <Describe what testing was performed and which tests were added.> ### Integration - Successful round-trip to HTTP endpoint, - Permanent error during round-trip, - Retryable error response for first request, followed by successful response on retry, - Two retriable error responses, followed by successful response on second retry. ### Manual - Authentication using `configtls.TLSSetting` (`ca_file`, `cert_file`, `key_file`) - Tested in EKS and K3s clusters running [opni](https://github.com/rancher/opni). --------- Signed-off-by: João Henri <[email protected]> Signed-off-by: João Henri <[email protected]>
Problem statement
This document describes the index naming standard for ingestion of Observability signals - Traces, Metrics, Logs.
Currently there is no single coherent pattern to use for all Observability signals and potential data sources.
For example -
data-prepper
use their own index naming and structure to ingest Observability signals.data-prepper Indices:
otel-v1-apm-span-**
(Observability Trace mapping)otel-v1-apm-service-map
(Proprietary Index Mapping)The same goes for jaeger trace data type:
jaeger-span*
(Observability Trace mapping)This convention is also harder to manage regarding the index revolving for lifecycle management - this would be optimized using the
data_stream
layer supported by OpenSearch API.Today due to different index structure and non-standard naming patterns we cant create crosscutting queries that will correlate or aggregate information on top of different Observability data providers.
Proposal
We would use the next structure and naming patterns based on the following conventions :
data_stream
support for all Observability based standard indicesdata_stream
will encourage simple physical index management and query - each Observability index would actually be a data_stream:data_stream
concepts patterns and catalog. The next Observability index pattern will be followed:Index pattern will follow the next naming structure
{type}
-{dataset}
-{namespace}
sso_
schema convention )nginx.access
.This strategy will be defined by the two degrees of naming freedom:
dataset
andnamespace
For example a customer may want to route the nginx logs from two geographical areas into two different indices:
sso_logs-nginx-us
sso_logs-nginx-eu
This type of distinction also allows for creation of crosscutting queries by setting the next index query pattern
sso_logs-nginx-*
or by using a geographic based crosscutting querysso_logs-*-eu
.Data index routing
The ingestion component which is responsible for ingesting the Observability signals should route the data into the relevant indices.
The
sso_{type}-{dataset}-{namespace}
combination dictates the target index,{type}
is prefixed with thesso_
prefix into one of the supported type:sso_traces
sso_metrics
sso_logs
For example if within the ingested log contains the following section:
This indicates that the target index for this observability signal should be
sso_traces
-mysql
-prod
index that follows uses the traces schema mapping.Observability Index templates
With the expectation of multiple Observability data providers and the need to consolidate all to a single common schema - the Observability plugin will take the following responsibilities :
Note
It is important to mention here that these new capabilities would not change or prevent existing customer usage of the system and continue to allow proprietary usage.
In details
Logs Schema
see - #1403
Traces Schema
see - #1395
Metrics Schema
see - #1397
What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.
Note
Important to mention here that this new suggestion would not change or prevent existing customer usage of the system and continue to allow proprietary usage.
Do you have any additional context?
see opensearch-project/OpenSearch-Dashboards#3412
see https://opensearch.org/docs/latest/opensearch/data-streams/
see https://github.com/opensearch-project/data-prepper
The text was updated successfully, but these errors were encountered: