Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SC-20508 Add sematext exporter #2

Open
wants to merge 40 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
7be09e3
Worked on metrics support
AkhigbeEromo Oct 3, 2024
56075f9
Fixed all otel-related issues
AkhigbeEromo Oct 4, 2024
6cad40b
Add region to sematextexporter config
AkhigbeEromo Oct 4, 2024
8ed1a5a
Rewrite test for config
AkhigbeEromo Oct 4, 2024
af5ba97
Add comments for the region
AkhigbeEromo Oct 4, 2024
6d2eff4
Fix linter issues
AkhigbeEromo Oct 4, 2024
929070f
Vaidate app_token and region
AkhigbeEromo Oct 5, 2024
f958e8f
Add sematext specifics for the metrics line
AkhigbeEromo Oct 7, 2024
abf2930
SC-20508 #comment Fix the timestamp issue but now the test is failing…
AkhigbeEromo Oct 8, 2024
bf6e81b
Fix the issue with the test to ensure metrics line are sent in semate…
AkhigbeEromo Oct 8, 2024
9d02ad7
Fix lint issues
AkhigbeEromo Oct 8, 2024
2146176
Delete logging for test
AkhigbeEromo Oct 8, 2024
6af2ce6
Merge remote-tracking branch 'origin' into add-sematext-exporter
boratanrikulu Oct 9, 2024
602ecbf
remove go.mod files
boratanrikulu Oct 9, 2024
0290804
fix new-line issue
boratanrikulu Oct 9, 2024
b4788ae
Update App_token to AppToken
AkhigbeEromo Oct 9, 2024
649215f
Clean up dirty code
AkhigbeEromo Oct 9, 2024
2e17dd5
Remove "influx" occurences
AkhigbeEromo Oct 9, 2024
e358f65
Remove unneccessary comments
AkhigbeEromo Oct 9, 2024
0cac0f9
Change writer_test function names to Pascal case
AkhigbeEromo Oct 15, 2024
89713a0
Remove unnecessary calculation
AkhigbeEromo Oct 15, 2024
be7f986
Put AppToken in a constant outside the function
AkhigbeEromo Oct 15, 2024
d8e5f14
Fix "No content" error
AkhigbeEromo Oct 16, 2024
c7fb729
Fix hostname issue
AkhigbeEromo Oct 16, 2024
daecc41
Make telegraf-prometheus-v2 default metrics schema
AkhigbeEromo Oct 22, 2024
683b038
Change schema from otel-v1 to telegraf-prometheus-v2
AkhigbeEromo Oct 23, 2024
a8346ed
Fix test because hostname changed
AkhigbeEromo Oct 23, 2024
02fec37
Reduce tags to 20 and make sure os.host and token tags are always pre…
AkhigbeEromo Oct 24, 2024
a85429a
Make os.host dynamic in test
AkhigbeEromo Oct 24, 2024
9317219
Add README file
Eromosele-SM Oct 25, 2024
1f32a87
Change endpoint to metrics receiver endpoint
Eromosele-SM Oct 25, 2024
8d81a7d
Region is now used to determine the endpoint
Eromosele-SM Oct 25, 2024
245725c
Edit README.md
Eromosele-SM Oct 28, 2024
eeb42af
Update README.md
Eromosele-SM Oct 28, 2024
5d17fd7
Refactor Exporter so that endpoints can be seperate
Eromosele-SM Oct 30, 2024
b4505b0
Refactor Exporter
Eromosele-SM Oct 30, 2024
98d51f4
Correct a comment
Eromosele-SM Oct 30, 2024
1c87f19
Update README.md
Eromosele-SM Oct 30, 2024
fcd0f90
Change defaults of payload
Eromosele-SM Oct 30, 2024
40581e7
Fix indentation issues
Eromosele-SM Nov 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions exporter/sematextexporter/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Sematext Exporter
<!-- status autogenerated section -->
<!-- end autogenerated section -->

This exporter supports sending metrics to [Sematext Cloud](https://sematext.com/) in Influx line protocol format

## Configuration

The following configuration options are supported:

* `endpoint` (required) HTTP/S destination for metric receivers. It is dependent on the region:
- US: `spm-receiver.sematext.com`
- EU: `spm-receiver.eu.sematext.com`
* `timeout` (default = 5s) Timeout for requests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Eromosele-SM make it match STA timeout?

* `AppToken` App token is the token of Sematext Monitoring App to which you want to send the metrics.
* `Region` Region specifies the Sematext region the user is operating in; must be one of:
Eromosele-SM marked this conversation as resolved.
Show resolved Hide resolved
* `US`
* `EU`
* `payload_max_lines` (default = 10_000) Maximum number of lines allowed per HTTP POST request
* `payload_max_bytes` (default = 10_000_000) Maximum number of bytes allowed per HTTP POST request
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Eromosele-SM I'd check these defaults internally with Sematext first.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the suggest for custom metrics. I'd ask STA devs what STA uses.

* `metrics_schema` (default = telegraf-prometheus-v2) The chosen metrics schema to write; must be one of:
* `otel-v1`
* `telegraf-prometheus-v2`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Eromosele-SM probably only one schema in the end. Also, this Exporter will end up shipping telemetry other than metrics. So maybe you want to structure the README in preparation for that. For example, this particular schema will be only for metrics.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can always edit the readme when we have logs support ready?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ofc. I hope the YAML is also structured in anticipation of Exporter sending other telemetry later (so other App tokens, other endpoints, although the region could probably remain general).

* `sending_queue` [details here](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.25.0/exporter/exporterhelper/README.md#configuration)
* `enabled` (default = true)
* `num_consumers` (default = 10) The number of consumers from the queue
* `queue_size` (default = 1000) Maximum number of batches allowed in queue at a given time
* `retry_on_failure` [details here](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.25.0/exporter/exporterhelper/README.md#configuration)
* `enabled` (default = true)
* `initial_interval` (default = 5s) Time to wait after the first failure before retrying
* `max_interval` (default = 30s) Upper bound on backoff interval
* `max_elapsed_time` (default = 120s) Maximum amount of time (including retries) spent trying to send a request/batch

The full list of settings exposed for this exporter are documented in [config.go](config.go).

Example:
```yaml
endpoint: spm-receiver.sematext.com
timeout: 500ms
sending_queue:
enabled: true
num_consumers: 3
queue_size: 10
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 3s
max_elapsed_time: 10s
region: US
app_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
metrics_schema: telegraf-prometheus-v2
payload_max_lines: 100
payload_max_bytes: 1000
```