Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OTel logs source doc to doc website. #3608

Merged
merged 52 commits into from
Apr 25, 2023
Merged
Changes from 40 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
6dcffb8
Add otel_logs_source doc.
carolxob Mar 28, 2023
4074a82
Content updates mainly to formatting.
carolxob Mar 28, 2023
42f0d4a
Converted list entries into tables and fixed formatting.
carolxob Mar 29, 2023
1e8121f
Minor updates.
carolxob Mar 29, 2023
a33e97d
Minor edits.
carolxob Mar 29, 2023
ef5c6f3
Minor edits.
carolxob Mar 29, 2023
4a4bb93
Minor edits.
carolxob Mar 29, 2023
dfa9265
Adjusted nav_order in metadata.
carolxob Mar 29, 2023
e0933e9
Content edits from tech review.
carolxob Apr 7, 2023
b624d08
Edit made based on technical review feedback.
carolxob Apr 11, 2023
cb494b6
Major revisions to table formatting and headers.
carolxob Apr 13, 2023
3b41ffc
Minor adjustements.
carolxob Apr 13, 2023
73a470e
Adjusted default values for items in the SSL table.
carolxob Apr 17, 2023
873af77
Reworded two options in the SSL section.
carolxob Apr 17, 2023
4e11722
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 20, 2023
24d450d
Minor changes.
carolxob Apr 24, 2023
199d20f
Minor changes.
carolxob Apr 24, 2023
f629ce9
Minor updates.
carolxob Apr 24, 2023
f3d04c2
Minor updates.
carolxob Apr 24, 2023
b644a6d
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
6251264
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
52136fc
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
2bb9ef6
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
cc0f5f9
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
35fabc2
Content updates from feedback.
carolxob Apr 24, 2023
96e832c
Minor update.
carolxob Apr 24, 2023
d85535f
Minor change.
carolxob Apr 24, 2023
9e81c14
Minor updates.
carolxob Apr 24, 2023
c7700e0
Minor updates.
carolxob Apr 24, 2023
6f1a769
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
fa0acaf
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
5ee99f6
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
27513ba
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
3a38abf
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
dae9551
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
b9ae337
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
c7c42dc
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
5502eea
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
1dccb68
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 24, 2023
eb42f79
Minor updates.
carolxob Apr 24, 2023
dd0d196
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
9a0c3f2
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
6102da5
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
91404d1
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
90f79e8
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
bf7d849
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
c33c64b
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
b4973b8
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
4e32bfc
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
49a1ce4
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
00d5222
Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md
carolxob Apr 25, 2023
9d1696d
Minor updates from editorial feedback.
carolxob Apr 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
carolxob marked this conversation as resolved.
Show resolved Hide resolved
layout: default
title: otel_logs_source
parent: Sources
grand_parent: Pipelines
nav_order: 25
---

# otel_logs_source


The `otel_logs_source` source is an OpenTelemetry source that follows the [OpenTelemetry Protocol Specification](https://github.com/open-telemetry/oteps/blob/master/text/0035-opentelemetry-protocol.md), receives logs from the OTel Collector in the form of `ExportLogsServiceRequest` records.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

This source supports the `OTLP/gRPC` protocol.
{: .note}

## Configuration

You can configure the `otel_logs_source` source with the following options.

| Option | Type | Description |
| :--- | :--- | :--- |
| port | int | Represents the port that the `otel_logs_source` is running on. Default value is `21892`. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
| path | string | Represents the path for sending unframed HTTP requests. You can use this option to support unframed gRPC with HTTP idiomatic path to a configurable path. The path should start with `/` and length should be at least 1. `/opentelemetry.proto.collector.logs.v1.LogsService/Export` endpoint is disabled for both gRPC and HTTP requests if the path is configured. The path can contain `${pipelineName}` placeholder, which is replaced with pipeline name. If the value is empty and `unframed_requests` is `true`, then the path that the source provides is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
carolxob marked this conversation as resolved.
Show resolved Hide resolved
| request_timeout | int | Represents the request timeout duration in milliseconds. Default value is `10000`. |
| health_check_service | Boolean | Enables the gRPC health check service under `grpc.health.v1/Health/Check`. Default value is `false`. |
| proto_reflection_service | Boolean | Enables a reflection service for Protobuf services (see [ProtoReflectionService](https://grpc.github.io/grpc-java/javadoc/io/grpc/protobuf/services/ProtoReflectionService.html) and [gRPC reflection](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md) documents). Default value is `false`. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
| unframed_requests | Boolean | Enables requests that are not framed using the gRPC wire protocol. Default value is `false`. |
| thread_count | int | The number of threads to keep in the `ScheduledThreadPool`. Default value is `500`. |
| max_connection_count | int | The maximum number of open connections allowed. Default value is `500`. |

### SSL

You can configure SSL in the `otel_logs_source` source with the following options.

| Option | Type | Description |
| :--- | :--- | :--- |
| ssl | Boolean | Enables TLS/SSL. Default value is `true`. |
| sslKeyCertChainFile | string | Represents the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. For example, see the Amazon S3 path `s3://<bucketName>/<path>`. Required if `ssl` is set to `true`. |
| sslKeyFile | string | Represents the SSL key file path or Amazon S3 path. For example, see the Amazon S3 path: `s3://<bucketName>/<path>`. Required if `ssl` is set to `true`. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
| useAcmCertForSSL | Boolean | Enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
| acmCertificateArn | string | Represents the ACM certificate ARN. ACM certificate take preference over Amazon S3 or local file system certificate. Required if `useAcmCertForSSL` is set to `true`. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
| awsRegion | string | Represents the AWS Region to use ACM or Amazon S3. Required if `useAcmCertForSSL` is set to `true` or `sslKeyCertChainFile` and `sslKeyFile` is the Amazon S3 path. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
carolxob marked this conversation as resolved.
Show resolved Hide resolved

## Usage

To get started, create a `pipeline.yaml` file and add the `otel_logs_source` source as source:
carolxob marked this conversation as resolved.
Show resolved Hide resolved

```
source:
- otel_logs_source:
```

## Metrics

You can use the following metrics with the `otel_logs_source`.
carolxob marked this conversation as resolved.
Show resolved Hide resolved

| Option | Type | Description |
| :--- | :--- | :--- |
| `requestTimeouts` | Counter | Measures the total number of requests that time out. |
| `requestsReceived` | Counter | Measures the total number of requests received by the `otel_logs_source`. |
carolxob marked this conversation as resolved.
Show resolved Hide resolved
| `badRequests` | Counter | Measures the total number of requests that could not be parsed. |
| `requestsTooLarge` | Counter | Measures the total number of requests that exceed the maximum allowed size. Indicates that the size of the data being written into the buffer is beyond the buffer's maximum capacity. |
| `internalServerError` | Counter | Measures the total number of requests that are erroneous due to errors other than `requestTimeouts` or `requestsTooLarge`. |
| `successRequests` | Counter | Measures the total number of requests successfully written to the buffer. |
| `payloadSize` | Distribution summary | Measures the distribution of all incoming payload sizes. |
| `requestProcessDuration` | Timer | Measures the duration of request processing. |