Skip to content

Commit

Permalink
Add trace parser docs. Clean up many other docs (open-telemetry#269)
Browse files Browse the repository at this point in the history 
* Add trace parser docs. Clean up many other docs

* retrigger checks
  • Loading branch information
@djaglowski
djaglowski authored Sep 17, 2021
1 parent bef2800 commit bc0bc79
Show file tree
Hide file tree
Showing 36 changed files with 378 additions and 308 deletions.
58 changes: 31 additions & 27 deletions docs/operators/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,38 +7,42 @@ For instance, a user may read lines from a file using the `file_input` operator.
## What operators are available?

Inputs:
- [File](/docs/operators/file_input.md)
- [Windows Event Log](/docs/operators/windows_eventlog_input.md)
- [TCP](/docs/operators/tcp_input.md)
- [UDP](/docs/operators/udp_input.md)
- [Journald](/docs/operators/journald_input.md)
- [Generate](/docs/operators/generate_input.md)
- [file_input](/docs/operators/file_input.md)
- [generate_input](/docs/operators/generate_input.md)
- [journald_input](/docs/operators/journald_input.md)
- [k8s_event_input](/docs/operators/k8s_event_input.md)
- [stdin](/docs/operators/stdin.md)
- [syslog_input](/docs/operators/syslog_input.md)
- [tcp_input](/docs/operators/tcp_input.md)
- [udp_input](/docs/operators/udp_input.md)
- [windows_eventlog_input](/docs/operators/windows_eventlog_input.md)

Parsers:
- [JSON](/docs/operators/json_parser.md)
- [Regex](/docs/operators/regex_parser.md)
- [Syslog](/docs/operators/syslog_parser.md)
- [Severity](/docs/operators/severity_parser.md)
- [Time](/docs/operators/time_parser.md)
- [csv_parser](/docs/operators/csv_parser.md)
- [json_parser](/docs/operators/json_parser.md)
- [regex_parser](/docs/operators/regex_parser.md)
- [syslog_parser](/docs/operators/syslog_parser.md)
- [severity_parser](/docs/operators/severity_parser.md)
- [time_parser](/docs/operators/time_parser.md)
- [trace_parser](/docs/operators/trace_parser.md)
- [uri_parser](/docs/operators/uri_parser.md)

Outputs:
- [Stdout](/docs/operators/stdout.md)
- [File](docs/operators/file_output.md)
- [file_output](docs/operators/file_output.md)
- [stdout](/docs/operators/stdout.md)

General purpose:
- [Add](/docs/operators/add.md)
- [Copy](/docs/operators/copy.md)
- [Flatten](/docs/operators/flatten.md)
- [Filter](/docs/operators/filter.md)
- [Host Metadata](/docs/operators/host_metadata.md)
- [Kubernetes Metadata Decorator](/docs/operators/k8s_metadata_decorator.md)
- [Metadata](/docs/operators/metadata.md)
- [Move](/docs/operators/move.md)
- [Rate Limit](/docs/operators/rate_limit.md)
- [Router](/docs/operators/router.md)
- [Recombine](/docs/operators/recombine.md)
- [Restructure](/docs/operators/restructure.md)
- [Remove](/docs/operators/remove.md)
- [Retain](/docs/operators/retain.md)
- [add](/docs/operators/add.md)
- [copy](/docs/operators/copy.md)
- [flatten](/docs/operators/flatten.md)
- [filter](/docs/operators/filter.md)
- [flatten](/docs/operators/flatten.md)
- [metadata](/docs/operators/metadata.md)
- [move](/docs/operators/move.md)
- [recombine](/docs/operators/recombine.md)
- [remove](/docs/operators/remove.md)
- [restructure](/docs/operators/restructure.md)
- [retain](/docs/operators/retain.md)
- [router](/docs/operators/router.md)

Or create your own [plugins](/docs/plugins.md) for a technology-specific use case.
14 changes: 7 additions & 7 deletions docs/operators/add.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ The `add` operator adds a value to an `entry`'s `body`, `attributes`, or `resour

### Configuration Fields

| Field | Default | Description |
| --- | --- | --- |
| `id` | `add` | A unique identifier for the operator |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries |
| `field` | required | The [field](/docs/types/field.md) to be added.
| `value` | required | `value` is either a static value or an [expression](https://github.com/open-telemetry/opentelemetry-log-collection/blob/main/docs/types/expression.md). If a value is specified, it will be added to each entry at the field defined by `field`. If an expression is specified, it will be evaluated for each entry and added at the field defined by `field`
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) |
| Field | Default | Description |
| --- | --- | --- |
| `id` | `add` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `field` | required | The [field](/docs/types/field.md) to be added. |
| `value` | required | `value` is either a static value or an [expression](https://github.com/open-telemetry/opentelemetry-log-collection/blob/main/docs/types/expression.md). If a value is specified, it will be added to each entry at the field defined by `field`. If an expression is specified, it will be evaluated for each entry and added at the field defined by `field`. |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). |
| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. |


Expand Down
14 changes: 7 additions & 7 deletions docs/operators/copy.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ The `copy` operator copies a value from one [field](/docs/types/field.md) to ano

### Configuration Fields

| Field | Default | Description |
| --- | --- | --- |
| `id` | `copy` | A unique identifier for the operator |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries |
| `from` | required | The [field](/docs/types/field.md) to copy the value of.
| `to` | required | The [field](/docs/types/field.md) to copy the value into.
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) |
| Field | Default | Description |
| --- | --- | --- |
| `id` | `copy` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `from` | required | The [field](/docs/types/field.md) from which the value should be copied. |
| `to` | required | The [field](/docs/types/field.md) to which the value should be copied. |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). |
| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. |

### Example Configurations:
Expand Down
24 changes: 12 additions & 12 deletions docs/operators/csv_parser.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,18 @@ The `csv_parser` operator parses the string-type field selected by `parse_from`

### Configuration Fields

| Field | Default | Description |
| --- | --- | --- |
| `id` | `csv_parser` | A unique identifier for the operator |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries |
| `header` | required | A string of delimited field names. The values in the delimited header will be used as keys |
| `delimiter` | `,` | A character that will be used as a delimiter. Values `\r` and `\n` cannot be used as a delimiter |
| `parse_from` | $body | A [field](/docs/types/field.md) that indicates the field to be parsed |
| `parse_to` | $body | A [field](/docs/types/field.md) that indicates the field to be parsed |
| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) |
| `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator |
| `severity` | `nil` | An optional [severity](/docs/types/severity.md) block which will parse a severity field before passing the entry to the output operator |
| Field | Default | Description |
| --- | --- | --- |
| `id` | `csv_parser` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `header` | required | A string of delimited field names. The values in the delimited header will be used as keys. |
| `delimiter` | `,` | A character that will be used as a delimiter. Values `\r` and `\n` cannot be used as a delimiter. |
| `parse_from` | $body | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | $body | The [field](/docs/types/field.md) to which the value will be parsed. |
| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). |
| `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator. |
| `severity` | `nil` | An optional [severity](/docs/types/severity.md) block which will parse a severity field before passing the entry to the output operator. |

### Example Configurations

Expand Down
38 changes: 19 additions & 19 deletions docs/operators/file_input.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,27 @@ The `file_input` operator reads logs from files. It will place the lines read in

### Configuration Fields

| Field | Default | Description |
| --- | --- | --- |
| `id` | `file_input` | A unique identifier for the operator |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries |
| `include` | required | A list of file glob patterns that match the file paths to be read |
| `exclude` | [] | A list of file glob patterns to exclude from reading |
| `poll_interval` | 200ms | The duration between filesystem polls |
| `multiline` | | A `multiline` configuration block. See below for details |
| `force_flush_period` | `500ms` | Time since last read of data from file, after which currently buffered log should be send to pipeline. Takes [duration](../types/duration.md) as value. Zero means waiting for new data forever |
| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry |
| `encoding` | `utf-8` | The encoding of the file being read. See the list of supported encodings below for available options |
| `include_file_name` | `true` | Whether to add the file name as the attribute `file.name` |
| `include_file_path` | `false` | Whether to add the file path as the attribute `file.path` |
| `include_file_name_resolved` | `false` | Whether to add the file name after symlinks resolution as the attribute `file.name.resolved` |
| `include_file_path_resolved` | `false` | Whether to add the file path after symlinks resolution as the attribute `file.path.resolved` |
| `start_at` | `end` | At startup, where to start reading logs from the file. Options are `beginning` or `end` |
| Field | Default | Description |
| --- | --- | --- |
| `id` | `file_input` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `include` | required | A list of file glob patterns that match the file paths to be read. |
| `exclude` | [] | A list of file glob patterns to exclude from reading. |
| `poll_interval` | 200ms | The duration between filesystem polls. |
| `multiline` | | A `multiline` configuration block. See below for details. |
| `force_flush_period` | `500ms` | Time since last read of data from file, after which currently buffered log should be send to pipeline. Takes [duration](../types/duration.md) as value. Zero means waiting for new data forever. |
| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry. |
| `encoding` | `utf-8` | The encoding of the file being read. See the list of supported encodings below for available options. |
| `include_file_name` | `true` | Whether to add the file name as the attribute `file.name`. |
| `include_file_path` | `false` | Whether to add the file path as the attribute `file.path`. |
| `include_file_name_resolved` | `false` | Whether to add the file name after symlinks resolution as the attribute `file.name.resolved`. |
| `include_file_path_resolved` | `false` | Whether to add the file path after symlinks resolution as the attribute `file.path.resolved`. |
| `start_at` | `end` | At startup, where to start reading logs from the file. Options are `beginning` or `end`. |
| `fingerprint_size` | `1kb` | The number of bytes with which to identify a file. The first bytes in the file are used as the fingerprint. Decreasing this value at any point will cause existing fingerprints to forgotten, meaning that all files will be read from the beginning (one time). |
| `max_log_size` | `1MiB` | The maximum size of a log entry to read before failing. Protects against reading large amounts of data into memory |
| `max_log_size` | `1MiB` | The maximum size of a log entry to read before failing. Protects against reading large amounts of data into memory |.
| `max_concurrent_files` | 1024 | The maximum number of log files from which logs will be read concurrently (minimum = 2). If the number of files matched in the `include` pattern exceeds half of this number, then files will be processed in batches. One batch will be processed per `poll_interval`. |
| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes |
| `resource` | {} | A map of `key: value` pairs to add to the entry's resource |
| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes. |
| `resource` | {} | A map of `key: value` pairs to add to the entry's resource. |

Note that by default, no logs will be read unless the monitored file is actively being written to because `start_at` defaults to `end`.

Expand Down
Loading

0 comments on commit bc0bc79

Please sign in to comment.