Skip to content

Commit

Permalink
feat(outputs.opensearch): Add OpenSearch output plugin (#11958)
Browse files Browse the repository at this point in the history 
Co-authored-by: Josh Powers <[email protected]>
  • Loading branch information
@mannukalra @powersj
mannukalra and powersj authored Sep 29, 2023
1 parent 188c8d7 commit 4e35ac8
Show file tree
Hide file tree
Showing 8 changed files with 1,864 additions and 0 deletions.
5 changes: 5 additions & 0 deletions plugins/outputs/all/opensearch.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
//go:build !custom || outputs || outputs.opensearch

package all

import _ "github.com/influxdata/telegraf/plugins/outputs/opensearch" // register plugin
352 changes: 352 additions & 0 deletions plugins/outputs/opensearch/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,352 @@
# OpenSearch Output Plugin

This plugin writes to [OpenSearch](https://opensearch.org/) via HTTP

It supports OpenSearch releases from 1 and 2. Future comparability with 1.x is
not guaranteed and instead will focus on 2.x support. Consider using the
existing Elasticsearch plugin for 1.x.

## Global configuration options <!-- @/docs/includes/plugin_config.md -->

In addition to the plugin-specific configuration settings, plugins support
additional global and plugin configuration settings. These settings are used to
modify metrics, tags, and field or create aliases and configure ordering, etc.
See the [CONFIGURATION.md][CONFIGURATION.md] for more details.

[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins

## Configuration

```toml @sample.conf
# Configuration for OpenSearch to send metrics to.
[[outputs.OpenSearch]]
## URLs
## The full HTTP endpoint URL for your OpenSearch instance. Multiple URLs can
## be specified as part of the same cluster, but only one URLs is used to
## write during each interval.
urls = ["http://node1.os.example.com:9200"]

## Index Name
## Target index name for metrics (OpenSearch will create if it not exists).
## This is a Golang template (see https://pkg.go.dev/text/template)
## You can also specify
## metric name (`{{.Name}}`), tag value (`{{.Tag "tag_name"}}`), field value (`{{.Field "feild_name"}}`)
## If the tag does not exist, the default tag value will be empty string "".
## the timestamp (`{{.Time.Format "xxxxxxxxx"}}`).
## For example: "telegraf-{{.Time.Format "2006-01-02"}}-{{.Tag "host"}}" would set it to telegraf-2023-07-27-HostName
index_name = ""

## Timeout
## OpenSearch client timeout
# timeout = "5s"

## Sniffer
## Set to true to ask OpenSearch a list of all cluster nodes,
## thus it is not necessary to list all nodes in the urls config option
# enable_sniffer = false

## GZIP Compression
## Set to true to enable gzip compression
# enable_gzip = false

## Health Check Interval
## Set the interval to check if the OpenSearch nodes are available
## Setting to "0s" will disable the health check (not recommended in production)
# health_check_interval = "10s"

## Set the timeout for periodic health checks.
# health_check_timeout = "1s"
## HTTP basic authentication details.
# username = ""
# password = ""
## HTTP bearer token authentication details
# auth_bearer_token = ""

## Optional TLS Config
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false

## Template Config
## Manage templates
## Set to true if you want telegraf to manage its index template.
## If enabled it will create a recommended index template for telegraf indexes
# manage_template = true

## Template Name
## The template name used for telegraf indexes
# template_name = "telegraf"

## Overwrite Templates
## Set to true if you want telegraf to overwrite an existing template
# overwrite_template = false

## Document ID
## If set to true a unique ID hash will be sent as
## sha256(concat(timestamp,measurement,series-hash)) string. It will enable
## data resend and update metric points avoiding duplicated metrics with
## different id's
# force_document_id = false

## Value Handling
## Specifies the handling of NaN and Inf values.
## This option can have the following values:
## none -- do not modify field-values (default); will produce an error
## if NaNs or infs are encountered
## drop -- drop fields containing NaNs or infs
## replace -- replace with the value in "float_replacement_value" (default: 0.0)
## NaNs and inf will be replaced with the given number, -inf with the negative of that number
# float_handling = "none"
# float_replacement_value = 0.0

## Pipeline Config
## To use a ingest pipeline, set this to the name of the pipeline you want to use.
# use_pipeline = "my_pipeline"

## Pipeline Name
## Additionally, you can specify a tag name using the notation (`{{.Tag "tag_name"}}`)
## which will be used as the pipeline name (e.g. "{{.Tag "os_pipeline"}}").
## If the tag does not exist, the default pipeline will be used as the pipeline.
## If no default pipeline is set, no pipeline is used for the metric.
# default_pipeline = ""
```

### Required parameters

* `urls`: A list containing the full HTTP URL of one or more nodes from your
OpenSearch instance.
* `index_name`: The target index for metrics. You can use the date format

For example: "telegraf-{{.Time.Format "2006-01-02"}}" would set it to
"telegraf-2023-07-27". You can also specify metric name (`{{ .Name }}`), tag
value (`{{ .Tag "tag_name" }}`), and field value (`{{ .Field "field_name" }}`).

If the tag does not exist, the default tag value will be empty string ""

## Permissions

If you are using authentication within your OpenSearch cluster, you need to
create an account and create a role with at least the manage role in the Cluster
Privileges category. Otherwise, your account will not be able to connect to your
OpenSearch cluster and send logs to your cluster. After that, you need to
add "create_index" and "write" permission to your specific index pattern.

## OpenSearch indexes and templates

### Indexes per time-frame

This plugin can manage indexes per time-frame, as commonly done in other tools
with OpenSearch. The timestamp of the metric collected will be used to decide
the index destination. For more information about this usage on OpenSearch,
check [the docs][1].

[1]: https://opensearch.org/docs/latest/

### Template management

Index templates are used in OpenSearch to define settings and mappings for
the indexes and how the fields should be analyzed. For more information on how
this works, see [the docs][2].

This plugin can create a working template for use with telegraf metrics. It uses
OpenSearch dynamic templates feature to set proper types for the tags and
metrics fields. If the template specified already exists, it will not overwrite
unless you configure this plugin to do so. Thus you can customize this template
after its creation if necessary.

Example of an index template created by telegraf on OpenSearch 2.x:

```json
{
"telegraf-2022.10.02" : {
"aliases" : { },
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"disk" : {
"properties" : {
"free" : {
"type" : "long"
},
"inodes_free" : {
"type" : "long"
},
"inodes_total" : {
"type" : "long"
},
"inodes_used" : {
"type" : "long"
},
"total" : {
"type" : "long"
},
"used" : {
"type" : "long"
},
"used_percent" : {
"type" : "float"
}
}
},
"measurement_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tag" : {
"properties" : {
"cpu" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"device" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"mode" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1664693522789",
"number_of_shards" : "1",
"number_of_replicas" : "1",
"uuid" : "TYugdmvsQfmxjzbGRJ8FIw",
"version" : {
"created" : "136247827"
},
"provided_name" : "telegraf-2022.10.02"
}
}
}
}

```

[2]: https://opensearch.org/docs/latest/opensearch/index-templates/

### Example events

This plugin will format the events in the following way:

```json
{
"@timestamp": "2017-01-01T00:00:00+00:00",
"measurement_name": "cpu",
"cpu": {
"usage_guest": 0,
"usage_guest_nice": 0,
"usage_idle": 71.85413456197966,
"usage_iowait": 0.256805341656516,
"usage_irq": 0,
"usage_nice": 0,
"usage_softirq": 0.2054442732579466,
"usage_steal": 0,
"usage_system": 15.04879301548127,
"usage_user": 12.634822807288275
},
"tag": {
"cpu": "cpu-total",
"host": "opensearhhost",
"dc": "datacenter1"
}
}
```

```json
{
"@timestamp": "2017-01-01T00:00:00+00:00",
"measurement_name": "system",
"system": {
"load1": 0.78,
"load15": 0.8,
"load5": 0.8,
"n_cpus": 2,
"n_users": 2
},
"tag": {
"host": "opensearhhost",
"dc": "datacenter1"
}
}
```

## Known issues

Integer values collected that are bigger than 2^63 and smaller than 1e21 (or in
this exact same window of their negative counterparts) are encoded by golang
JSON encoder in decimal format and that is not fully supported by OpenSearch
dynamic field mapping. This causes the metrics with such values to be dropped in
case a field mapping has not been created yet on the telegraf index. If that's
the case you will see an exception on OpenSearch side like this:

```json
{
"error": {
"root_cause": [
{"type": "mapper_parsing_exception", "reason": "failed to parse"}
],
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "illegal_state_exception",
"reason": "No matching token for number_type [BIG_INTEGER]"
}
},
"status": 400
}
```

The correct field mapping will be created on the telegraf index as soon as a
supported JSON value is received by OpenSearch, and subsequent insertions
will work because the field mapping will already exist.

This issue is caused by the way OpenSearch tries to detect integer fields,
and by how golang encodes numbers in JSON. There is no clear workaround for this
at the moment.
Loading

0 comments on commit 4e35ac8

Please sign in to comment.