feat(outputs.opensearch): opensearch output plugin #11958

mannukalra · 2022-10-07T06:42:45Z

Required for all PRs

Updated associated README.md.
Wrote appropriate unit tests.
Pull request title or commits are in conventional commit format

resolves #11345

mannukalra · 2022-10-07T06:49:44Z

#11345 All test cases passed with Opensearch 1.1.0 and 2.2.1.

Hipska

Although I know in the referenced FR it was decided to create a separate plugin, I'm wondering what is different to ES in this case, as it looks like the exact same API module is being used.

plugins/outputs/opensearch/README.md

plugins/outputs/opensearch/opensearch.go

mannukalra · 2022-10-14T13:11:49Z

Although I know in the referenced FR it was decided to create a separate plugin, I'm wondering what is different to ES in this case, as it looks like the exact same API module is being used.

Thought of using the opensearch-go client, but as of now there is no stable version which can support both 1.x and 2.x.
opensearch-project/opensearch-go#169

Maybe we can update the client in future.

plugins/outputs/opensearch/opensearch.go

plugins/outputs/opensearch/README.md

plugins/outputs/opensearch/opensearch.go

powersj · 2023-02-16T18:45:58Z

Thought of using the opensearch-go client, but as of now there is no stable version which can support both 1.x and 2.x. opensearch-project/opensearch-go#169

Maybe we can update the client in future.

We recently added the opensearch_query input plugin that uses that library, tested with both versions. In that PR we had a discussion about not using the github.com/olivere/elastic for any further. Is that because of comparability mode?

mannukalra · 2023-02-18T08:53:54Z

Thought of using the opensearch-go client, but as of now there is no stable version which can support both 1.x and 2.x. opensearch-project/opensearch-go#169
Maybe we can update the client in future.

We recently added the opensearch_query input plugin that uses that library, tested with both versions. In that PR we had a discussion about not using the github.com/olivere/elastic for any further. Is that because of comparability mode?

I was in opinion that opensearch-go/v2 is not backward compatible, but if it works with both will try to use this if you can confirm we need this change.

Hipska

I'm still in favour of using golang templates instead of something mixed and a lookalike of it.

plugins/outputs/opensearch/opensearch.go

powersj · 2023-02-21T16:48:47Z

I was in opinion that opensearch-go/v2 is not backward compatible, but if it works with both will try to use this if you can confirm we need this change.

We do not want any further plugins dependent on github.com/olivere/elastic

@NullOranje - when you looked at testing the new opensearch_query plugin you contributed, did you have to do anything special to test OpenSearch 1 and 2? Were there any compatibility issues?

NullOranje · 2023-02-21T17:01:09Z

@powerjs No, I didn't do anything differently other than test against a v1 container. I'm not sure what the breaking API changes might be in opensearch-go/v2, but the query API is the same, which is why it works

powersj · 2023-02-21T18:37:23Z

@mannukalra, what do you think about using opensearch-go/v2 and only targeting opensearch v2 and users can continue to use the elasticsearch plugin for v1?

NullOranje · 2023-02-21T19:21:01Z

That makes sense to me.

mannukalra · 2023-02-26T07:10:26Z

@mannukalra, what do you think about using opensearch-go/v2 and only targeting opensearch v2 and users can continue to use the elasticsearch plugin for v1?

Sure @powersj, that also sounds good, will try to make the change in the coming weeks.

Hipska · 2023-02-27T08:21:32Z

Can you mark the PR as draft in the meantime please?

powersj · 2023-04-04T14:12:17Z

@mannukalra,

Any progress on the updates to this PR? I know it is in draft, but trying to get an idea of the status of current PRs.

Thanks!

mannukalra · 2023-04-10T11:26:50Z

@mannukalra,

Any progress on the updates to this PR? I know it is in draft, but trying to get an idea of the status of current PRs.

Thanks!

@powersj, sorry it's been delayed a bit, will update shortly on the progress.

…th metric

Hipska

Great! Finally using the full power of GoLang templates!

Some minor remarks..

Hipska · 2023-09-18T07:01:00Z

plugins/outputs/opensearch/opensearch.go

+	// Determine if we should process NaN and inf values
+	valOptions := []string{"", "none", "drop", "replace"}
+	if err := choice.Check(o.FloatHandling, valOptions); err != nil {
+		return fmt.Errorf("config float_handling type %w", err)


Suggested change

return fmt.Errorf("config float_handling type %w", err)

return fmt.Errorf("config float_handling type: %w", err)

Hipska · 2023-09-18T07:01:13Z

plugins/outputs/opensearch/opensearch.go

+
+	indexTmpl, err := template.New("index").Parse(o.IndexName)
+	if err != nil {
+		return fmt.Errorf("error parsing indextemplate %w", err)


Suggested change

return fmt.Errorf("error parsing indextemplate %w", err)

return fmt.Errorf("error parsing index_name template: %w", err)

Hipska · 2023-09-18T07:02:52Z

plugins/outputs/opensearch/opensearch.go

+
+	pipelineTmpl, err := template.New("index").Parse(o.UsePipeline)
+	if err != nil {
+		return fmt.Errorf("error parsing pipelineTemplate for UsePipeline %w", err)


Suggested change

return fmt.Errorf("error parsing pipelineTemplate for UsePipeline %w", err)

return fmt.Errorf("error parsing use_pipeline template: %w", err)

Hipska · 2023-09-18T07:16:32Z

plugins/outputs/opensearch/README.md

+
+For example: "telegraf-{{.Time.Format "2006-01-02"}}" would set it to telegraf-2023-07-27
+You can also specify
+metric name (`{{Name}}`), tag value (`{{Tag "tag_name"}}`), field value (`{{Field "feild_name"}}`) 


Suggested change

metric name (`{{Name}}`), tag value (`{{Tag "tag_name"}}`), field value (`{{Field "feild_name"}}`)

metric name (`{{ .Name }}`), tag value (`{{ .Tag "tag_name" }}`), field value (`{{ .Field "feild_name" }}`)

Hipska · 2023-09-18T07:19:01Z

plugins/outputs/opensearch/sample.conf

+  ##   %Y - year (2016)
+  ##   %y - last two digits of year (00..99)
+  ##   %m - month (01..12)
+  ##   %d - day of month (e.g., 01)
+  ##   %H - hour (00..23)
+  ##   %V - week of the year (ISO week) (01..53)
+  ## For example: telegraf-%Y.%m.%d would set it to telegraf-2006-01-02


This is still the old way, please update.

Hipska · 2023-09-18T07:22:16Z

plugins/outputs/opensearch/sample.conf

+  ## Additionally, you can specify a tag name using the notation {{tag_name}}
+  ## which will be used as part of the index name: "telegraf-{{host}}-%Y.%m.%d"
+  ## If the tag does not exist, the default tag value will be used.


I would mention the possibility to use tags in the relevant section of "Index Name" and here just what happens if that tag doesn't exist..

It will be replaced with empty sting, have explained.
Just so you know- don't see a scope of handling the default_tag_value anymore with this implementation(have tried in test-cases, it only replaced with "" and not the value specified for DefaultTagValue), so removed DefaultTagValue field from struct. Let me know if I'm mistaken here and it can still work.

Hipska · 2023-09-18T07:23:48Z

plugins/outputs/opensearch/sample.conf

+  ## Additionally, you can specify a tag name using the notation {{tag_name}}
+  ## which will be used as part of the pipeline name (e.g. "{{es_pipeline}}").
+  ## If the tag does not exist, the default pipeline will be used as the
+  ## pipeline. If no default pipeline is set, no pipeline is used for the
+  ## metric.


Same here, explain the usage of tags directly in "Pipeline Config".

Hipska

Thanks for all the work!

srebhan · 2023-09-19T08:46:06Z

@mannukalra would you please be so kind to fix the CI tests before I'm doing the review!? It should be a make fmt.

I'm usually running make tidy && make fmt && make && make test && make vet && make check-deps && make lint && markdownlint ***/*.md before committing as this catches most of the CI issues...

powersj · 2023-09-27T20:29:02Z

I've cleaned up the PR so tests at least pass locally, will check CI shortly. I do want to do a review before we land as well.

Hipska · 2023-09-28T09:31:19Z

@powersj Still needs a make docs I assume.

telegraf-tiger · 2023-09-28T13:24:20Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

🥳 This pull request decreases the Telegraf binary size by -2.32 % for linux amd64 (new size: 197.3 MB, nightly size 202.0 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

srebhan

Thanks for all your effort @mannukalra and thanks for fixing the remaining issues @powersj!

mannukalra · 2023-09-30T06:57:32Z

Thanks for all your effort @mannukalra and thanks for fixing the remaining issues @powersj!

Thank you all for your support 😊

telegraf-tiger bot added feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins labels Oct 7, 2022

Hipska suggested changes Oct 11, 2022

View reviewed changes

Hipska reviewed Oct 14, 2022

View reviewed changes

plugins/outputs/opensearch/opensearch.go Outdated Show resolved Hide resolved

plugins/outputs/opensearch/opensearch.go Outdated Show resolved Hide resolved

plugins/outputs/opensearch/opensearch.go Outdated Show resolved Hide resolved

Hipska suggested changes Oct 17, 2022

View reviewed changes

Hipska suggested changes Oct 28, 2022

View reviewed changes

plugins/outputs/opensearch/opensearch.go Outdated Show resolved Hide resolved

plugins/outputs/opensearch/opensearch.go Outdated Show resolved Hide resolved

powersj self-assigned this Nov 17, 2022

powersj force-pushed the master branch from 606126b to cd00fb5 Compare January 26, 2023 20:49

powersj added the waiting for response waiting for response from contributor label Feb 16, 2023

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Feb 18, 2023

Hipska suggested changes Feb 20, 2023

View reviewed changes

Hipska requested review from Hipska and removed request for Hipska February 27, 2023 08:21

powersj marked this pull request as draft February 27, 2023 13:35

powersj added the waiting for response waiting for response from contributor label Apr 4, 2023

mannukalra closed this Apr 10, 2023

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Apr 10, 2023

mannukalra reopened this Apr 10, 2023

mannukalra added 4 commits September 13, 2023 14:22

review comments incorporation

1962fe1

minor corrections

0fda3e5

feat(outputs.opensearch): expected implementation of text template wi…

d03f648

…th metric

Merge branch 'influxdata:master' into master

aa7669d

Hipska suggested changes Sep 18, 2023

View reviewed changes

mannukalra added 3 commits September 18, 2023 17:50

feat(outputs.opensearch): review comments incorporation

eaa2865

Merge branch 'master' of https://github.com/mannukalra/telegraf

24aa6c6

log message correction

29afe38

Hipska approved these changes Sep 18, 2023

View reviewed changes

Hipska added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Sep 18, 2023

Hipska removed their assignment Sep 18, 2023

mannukalra requested a review from srebhan September 18, 2023 23:47

whitespace, go fmt, fix error passing

717189d

powersj self-requested a review September 27, 2023 20:29

remove long line from readme

c28e515

Hipska approved these changes Sep 27, 2023

View reviewed changes

make docs

5f24eb0

Hipska approved these changes Sep 28, 2023

View reviewed changes

srebhan approved these changes Sep 29, 2023

View reviewed changes

srebhan added the new plugin label Sep 29, 2023

srebhan merged commit 4e35ac8 into influxdata:master Sep 29, 2023
4 checks passed

github-actions bot added this to the v1.29.0 milestone Sep 29, 2023

RodrigoDornelles mentioned this pull request Nov 22, 2023

when will a new version with the opensearch output be released? #14332

Closed

Porkepix mentioned this pull request Dec 11, 2023

telegraf 1.29.0 Homebrew/homebrew-core#156969

Merged

	return fmt.Errorf("config float_handling type %w", err)
	return fmt.Errorf("config float_handling type: %w", err)

	return fmt.Errorf("error parsing indextemplate %w", err)
	return fmt.Errorf("error parsing index_name template: %w", err)

	return fmt.Errorf("error parsing pipelineTemplate for UsePipeline %w", err)
	return fmt.Errorf("error parsing use_pipeline template: %w", err)

	metric name (`{{Name}}`), tag value (`{{Tag "tag_name"}}`), field value (`{{Field "feild_name"}}`)
	metric name (`{{ .Name }}`), tag value (`{{ .Tag "tag_name" }}`), field value (`{{ .Field "feild_name" }}`)

feat(outputs.opensearch): opensearch output plugin #11958

feat(outputs.opensearch): opensearch output plugin #11958

Conversation

mannukalra commented Oct 7, 2022 • edited by Hipska Loading

Required for all PRs

mannukalra commented Oct 7, 2022

Hipska left a comment

Choose a reason for hiding this comment

mannukalra commented Oct 14, 2022 • edited Loading

powersj commented Feb 16, 2023

mannukalra commented Feb 18, 2023 • edited Loading

Hipska left a comment

Choose a reason for hiding this comment

powersj commented Feb 21, 2023

NullOranje commented Feb 21, 2023

powersj commented Feb 21, 2023

NullOranje commented Feb 21, 2023

mannukalra commented Feb 26, 2023

Hipska commented Feb 27, 2023

powersj commented Apr 4, 2023

mannukalra commented Apr 10, 2023

Hipska left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hipska left a comment

Choose a reason for hiding this comment

srebhan commented Sep 19, 2023

powersj commented Sep 27, 2023

Hipska commented Sep 28, 2023

telegraf-tiger bot commented Sep 28, 2023

Artifact URLs

srebhan left a comment

Choose a reason for hiding this comment

mannukalra commented Sep 30, 2023

mannukalra commented Oct 7, 2022 •

edited by Hipska

Loading

mannukalra commented Oct 14, 2022 •

edited

Loading

mannukalra commented Feb 18, 2023 •

edited

Loading