SC-20508 Add sematext exporter #2

Eromosele-SM · 2024-10-07T10:27:51Z

This PR adds metrics support for Sematext exporter

If a user enters the wrong region/ app_token it will throw an error

… with a different error. For some reason the recorded request body is nil

…xt format

otisg · 2024-10-09T06:27:25Z

@Eromosele-SM "This branch has conflicts that must be resolved"

exporter/sematextexporter/config.go

cmd/otelcontribcol/go.mod

exporter/sematextexporter/Makefile

exporter/sematextexporter/config.go

exporter/sematextexporter/writer.go

exporter/sematextexporter/writer_test.go

otisg · 2024-10-09T12:12:32Z

FWIW I saw 2 instances of the following in your merge of main into your fork:
c64a75e [chore] [exporter/sumologic] Use NewDefaultClientConfig instead of manually creating struct (open-telemetry#35547)

(this one for SL, and another for Mezmo)
Maybe something new to look into/use?

Eromosele-SM · 2024-10-09T13:40:44Z

FWIW I saw 2 instances of the following in your merge of main into your fork: c64a75e [chore] [exporter/sumologic] Use NewDefaultClientConfig instead of manually creating struct (open-telemetry#35547)

(this one for SL, and another for Mezmo) Maybe something new to look into/use?

Their createDefaultClientConfig() function returns a confighttp.ClientConfig but what our createDefaultClientConfig() function returns is a component.Config and there is no default for it

exporter/sematextexporter/writer.go

jackgopack4

I don't see any major issues here, some small suggestions. Make sure you get some pipelines/checks set up (you may be able to piggyback on upstream CI if you open a draft PR on open-telemetry/opentelemetry-collector-contrib

jackgopack4 · 2024-10-11T19:13:43Z

exporter/sematextexporter/writer_test.go

no specific comments on any of your tests, but it might be useful to open up a draft pr on the contrib repo to benefit from their pipelines they have set up for linting, test coverage, etc.

Yeah i will, once i am done with the logs support

exporter/sematextexporter/writer_test.go

exporter/sematextexporter/writer.go

exporter/sematextexporter/factory.go

exporter/sematextexporter/writer.go

exporter/sematextexporter/testdata/config.yaml

If hostname is not detected the exporter returns an error

…sent

otisg · 2024-10-25T08:02:44Z

exporter/sematextexporter/README.md

+* `endpoint` (required) HTTP/S destination for metric receivers. It is dependent on the region:
+    - US: `spm-receiver.sematext.com`
+    - EU: `spm-receiver.eu.sematext.com`
+* `timeout` (default = 5s) Timeout for requests


@Eromosele-SM make it match STA timeout?

exporter/sematextexporter/README.md

otisg · 2024-10-25T08:04:21Z

exporter/sematextexporter/README.md

+  * `US`
+  * `EU`
+* `payload_max_lines` (default = 10_000) Maximum number of lines allowed per HTTP POST request
+* `payload_max_bytes` (default = 10_000_000) Maximum number of bytes allowed per HTTP POST request


@Eromosele-SM I'd check these defaults internally with Sematext first.

Is this it?
https://sematext.com/docs/monitoring/custom-metrics/#:~:text=We%20recommend%20sending%20batches%20of%20between%20100%20and%20500%20lines%20of%20metrics.

That's the suggest for custom metrics. I'd ask STA devs what STA uses.

otisg · 2024-10-25T08:05:32Z

exporter/sematextexporter/README.md

+* `payload_max_bytes` (default = 10_000_000) Maximum number of bytes allowed per HTTP POST request
+* `metrics_schema` (default = telegraf-prometheus-v2) The chosen metrics schema to write; must be one of:
+    * `otel-v1`
+    * `telegraf-prometheus-v2`


@Eromosele-SM probably only one schema in the end. Also, this Exporter will end up shipping telemetry other than metrics. So maybe you want to structure the README in preparation for that. For example, this particular schema will be only for metrics.

I can always edit the readme when we have logs support ready?

Ofc. I hope the YAML is also structured in anticipation of Exporter sending other telemetry later (so other App tokens, other endpoints, although the region could probably remain general).

otisg

A few suggestions made...

Eromosele-SM · 2024-10-30T22:48:56Z

@boratanrikulu @otisg
Can you please take a look?
I made some changes

otisg

Just did a quick skip over the surface stuff, not actual code.

exporter/sematextexporter/README.md

exporter/sematextexporter/config.go

otisg · 2024-10-30T23:02:13Z

exporter/sematextexporter/config.go

+
+// Validate checks for invalid or missing entries in the configuration.
+func (cfg *Config) Validate() error {
+	if strings.ToLower(cfg.Region) != "eu" && strings.ToLower(cfg.Region) != "us" && strings.ToLower(cfg.Region) != "custom"{


Still not sure what "custom" is for. Esp. since in the line below you say only US and EU are allowed.

The custom is so that we can run tests on the exporter with a mockserver

AkhigbeEromo added 12 commits October 4, 2024 00:01

Worked on metrics support

7be09e3

Fixed all otel-related issues

56075f9

Add region to sematextexporter config

6cad40b

Rewrite test for config

8ed1a5a

Add comments for the region

af5ba97

Fix linter issues

6d2eff4

Vaidate app_token and region

929070f

If a user enters the wrong region/ app_token it will throw an error

Add sematext specifics for the metrics line

f958e8f

SC-20508 #comment Fix the timestamp issue but now the test is failing…

abf2930

… with a different error. For some reason the recorded request body is nil

Fix the issue with the test to ensure metrics line are sent in semate…

bf6e81b

…xt format

Fix lint issues

9d02ad7

Delete logging for test

2146176

Eromosele-SM marked this pull request as ready for review October 8, 2024 21:56

boratanrikulu requested changes Oct 9, 2024

View reviewed changes

Eromosele-SM force-pushed the add-sematext-exporter branch from bf683ff to 2146176 Compare October 9, 2024 11:35

boratanrikulu and others added 4 commits October 9, 2024 14:41

Merge remote-tracking branch 'origin' into add-sematext-exporter

6af2ce6

remove go.mod files

602ecbf

fix new-line issue

0290804

Update App_token to AppToken

b4788ae

AkhigbeEromo added 2 commits October 9, 2024 13:42

Clean up dirty code

649215f

Remove "influx" occurences

2e17dd5

Remove unneccessary comments

e358f65

jackgopack4 reviewed Oct 11, 2024

View reviewed changes

exporter/sematextexporter/writer.go Show resolved Hide resolved

jackgopack4 reviewed Oct 11, 2024

View reviewed changes

exporter/sematextexporter/writer.go Outdated Show resolved Hide resolved

jackgopack4 reviewed Oct 11, 2024

View reviewed changes

exporter/sematextexporter/factory.go Outdated Show resolved Hide resolved

Eromosele-SM changed the title ~~Add sematext exporter~~ SC:20508 Add sematext exporter Oct 14, 2024

otisg reviewed Oct 15, 2024

View reviewed changes

exporter/sematextexporter/writer.go Outdated Show resolved Hide resolved

otisg reviewed Oct 15, 2024

View reviewed changes

exporter/sematextexporter/testdata/config.yaml Outdated Show resolved Hide resolved

AkhigbeEromo and others added 9 commits October 16, 2024 14:56

Fix "No content" error

d8e5f14

Fix hostname issue

c7fb729

If hostname is not detected the exporter returns an error

Make telegraf-prometheus-v2 default metrics schema

daecc41

Change schema from otel-v1 to telegraf-prometheus-v2

683b038

Fix test because hostname changed

a8346ed

Reduce tags to 20 and make sure os.host and token tags are always pre…

02fec37

…sent

Make os.host dynamic in test

a85429a

Add README file

9317219

Change endpoint to metrics receiver endpoint

1f32a87

otisg reviewed Oct 25, 2024

View reviewed changes

exporter/sematextexporter/README.md Show resolved Hide resolved

otisg reviewed Oct 25, 2024

View reviewed changes

otisg approved these changes Oct 25, 2024

View reviewed changes

Eromosele-SM added 8 commits October 25, 2024 12:11

Region is now used to determine the endpoint

8d81a7d

Edit README.md

245725c

Update README.md

eeb42af

Refactor Exporter so that endpoints can be seperate

5d17fd7

Refactor Exporter

b4505b0

Correct a comment

98d51f4

Update README.md

1c87f19

Change defaults of payload

fcd0f90

Eromosele-SM requested review from otisg, jackgopack4 and boratanrikulu October 30, 2024 22:49

otisg reviewed Oct 30, 2024

View reviewed changes

Fix indentation issues

40581e7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SC-20508 Add sematext exporter #2

SC-20508 Add sematext exporter #2

Eromosele-SM commented Oct 7, 2024 •

edited

Loading

otisg commented Oct 9, 2024

otisg commented Oct 9, 2024

Eromosele-SM commented Oct 9, 2024

jackgopack4 left a comment

jackgopack4 Oct 11, 2024

Eromosele-SM Oct 15, 2024

otisg Oct 25, 2024

otisg Oct 25, 2024

Eromosele-SM Oct 25, 2024

otisg Oct 25, 2024

otisg Oct 25, 2024

Eromosele-SM Oct 25, 2024

otisg Oct 25, 2024

otisg left a comment

Eromosele-SM commented Oct 30, 2024

otisg left a comment

otisg Oct 30, 2024

Eromosele-SM Nov 5, 2024

SC-20508 Add sematext exporter #2

Are you sure you want to change the base?

SC-20508 Add sematext exporter #2

Conversation

Eromosele-SM commented Oct 7, 2024 • edited Loading

otisg commented Oct 9, 2024

otisg commented Oct 9, 2024

Eromosele-SM commented Oct 9, 2024

jackgopack4 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

otisg left a comment

Choose a reason for hiding this comment

Eromosele-SM commented Oct 30, 2024

otisg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eromosele-SM commented Oct 7, 2024 •

edited

Loading