From bc0bc797a977ba8c18cc5b22b0d47bb2b24e6605 Mon Sep 17 00:00:00 2001 From: Daniel Jaglowski Date: Fri, 17 Sep 2021 09:31:20 -0400 Subject: [PATCH] Add trace parser docs. Clean up many other docs (#269) * Add trace parser docs. Clean up many other docs * retrigger checks --- docs/operators/README.md | 58 +++++++++++++----------- docs/operators/add.md | 14 +++--- docs/operators/copy.md | 14 +++--- docs/operators/csv_parser.md | 24 +++++----- docs/operators/file_input.md | 38 ++++++++-------- docs/operators/file_output.md | 10 ++-- docs/operators/filter.md | 10 ++-- docs/operators/flatten.md | 12 ++--- docs/operators/generate_input.md | 16 +++---- docs/operators/journald_input.md | 24 +++++----- docs/operators/json_parser.md | 20 ++++---- docs/operators/k8s_event_input.md | 20 ++++---- docs/operators/metadata.md | 14 +++--- docs/operators/move.md | 14 +++--- docs/operators/recombine.md | 20 ++++---- docs/operators/regex_parser.md | 22 ++++----- docs/operators/remove.md | 12 ++--- docs/operators/restructure.md | 12 ++--- docs/operators/retain.md | 12 ++--- docs/operators/router.md | 20 ++++---- docs/operators/severity_parser.md | 20 ++++---- docs/operators/stanza_input.md | 10 ++-- docs/operators/stdin.md | 10 ++-- docs/operators/stdout.md | 6 +-- docs/operators/syslog_input.md | 12 ++--- docs/operators/syslog_parser.md | 18 ++++---- docs/operators/tcp_input.md | 26 +++++------ docs/operators/time_parser.md | 20 ++++---- docs/operators/trace_parser.md | 19 ++++++++ docs/operators/udp_input.md | 22 ++++----- docs/operators/uri_parser.md | 30 ++++++------ docs/operators/windows_eventlog_input.md | 22 ++++----- docs/types/entry.md | 14 +++--- docs/types/severity.md | 12 ++--- docs/types/timestamp.md | 12 ++--- docs/types/trace.md | 47 +++++++++++++++++++ 36 files changed, 378 insertions(+), 308 deletions(-) create mode 100644 docs/operators/trace_parser.md create mode 100644 docs/types/trace.md diff --git a/docs/operators/README.md b/docs/operators/README.md index 56880f74..ef5a9315 100644 --- a/docs/operators/README.md +++ b/docs/operators/README.md @@ -7,38 +7,42 @@ For instance, a user may read lines from a file using the `file_input` operator. ## What operators are available? Inputs: -- [File](/docs/operators/file_input.md) -- [Windows Event Log](/docs/operators/windows_eventlog_input.md) -- [TCP](/docs/operators/tcp_input.md) -- [UDP](/docs/operators/udp_input.md) -- [Journald](/docs/operators/journald_input.md) -- [Generate](/docs/operators/generate_input.md) +- [file_input](/docs/operators/file_input.md) +- [generate_input](/docs/operators/generate_input.md) +- [journald_input](/docs/operators/journald_input.md) +- [k8s_event_input](/docs/operators/k8s_event_input.md) +- [stdin](/docs/operators/stdin.md) +- [syslog_input](/docs/operators/syslog_input.md) +- [tcp_input](/docs/operators/tcp_input.md) +- [udp_input](/docs/operators/udp_input.md) +- [windows_eventlog_input](/docs/operators/windows_eventlog_input.md) Parsers: -- [JSON](/docs/operators/json_parser.md) -- [Regex](/docs/operators/regex_parser.md) -- [Syslog](/docs/operators/syslog_parser.md) -- [Severity](/docs/operators/severity_parser.md) -- [Time](/docs/operators/time_parser.md) +- [csv_parser](/docs/operators/csv_parser.md) +- [json_parser](/docs/operators/json_parser.md) +- [regex_parser](/docs/operators/regex_parser.md) +- [syslog_parser](/docs/operators/syslog_parser.md) +- [severity_parser](/docs/operators/severity_parser.md) +- [time_parser](/docs/operators/time_parser.md) +- [trace_parser](/docs/operators/trace_parser.md) +- [uri_parser](/docs/operators/uri_parser.md) Outputs: -- [Stdout](/docs/operators/stdout.md) -- [File](docs/operators/file_output.md) +- [file_output](docs/operators/file_output.md) +- [stdout](/docs/operators/stdout.md) General purpose: -- [Add](/docs/operators/add.md) -- [Copy](/docs/operators/copy.md) -- [Flatten](/docs/operators/flatten.md) -- [Filter](/docs/operators/filter.md) -- [Host Metadata](/docs/operators/host_metadata.md) -- [Kubernetes Metadata Decorator](/docs/operators/k8s_metadata_decorator.md) -- [Metadata](/docs/operators/metadata.md) -- [Move](/docs/operators/move.md) -- [Rate Limit](/docs/operators/rate_limit.md) -- [Router](/docs/operators/router.md) -- [Recombine](/docs/operators/recombine.md) -- [Restructure](/docs/operators/restructure.md) -- [Remove](/docs/operators/remove.md) -- [Retain](/docs/operators/retain.md) +- [add](/docs/operators/add.md) +- [copy](/docs/operators/copy.md) +- [flatten](/docs/operators/flatten.md) +- [filter](/docs/operators/filter.md) +- [flatten](/docs/operators/flatten.md) +- [metadata](/docs/operators/metadata.md) +- [move](/docs/operators/move.md) +- [recombine](/docs/operators/recombine.md) +- [remove](/docs/operators/remove.md) +- [restructure](/docs/operators/restructure.md) +- [retain](/docs/operators/retain.md) +- [router](/docs/operators/router.md) Or create your own [plugins](/docs/plugins.md) for a technology-specific use case. diff --git a/docs/operators/add.md b/docs/operators/add.md index 8a768d22..57f3d5d3 100644 --- a/docs/operators/add.md +++ b/docs/operators/add.md @@ -4,13 +4,13 @@ The `add` operator adds a value to an `entry`'s `body`, `attributes`, or `resour ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `add` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `field` | required | The [field](/docs/types/field.md) to be added. -| `value` | required | `value` is either a static value or an [expression](https://github.com/open-telemetry/opentelemetry-log-collection/blob/main/docs/types/expression.md). If a value is specified, it will be added to each entry at the field defined by `field`. If an expression is specified, it will be evaluated for each entry and added at the field defined by `field` -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `add` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `field` | required | The [field](/docs/types/field.md) to be added. | +| `value` | required | `value` is either a static value or an [expression](https://github.com/open-telemetry/opentelemetry-log-collection/blob/main/docs/types/expression.md). If a value is specified, it will be added to each entry at the field defined by `field`. If an expression is specified, it will be evaluated for each entry and added at the field defined by `field`. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | diff --git a/docs/operators/copy.md b/docs/operators/copy.md index f4128487..6fc9fe2a 100644 --- a/docs/operators/copy.md +++ b/docs/operators/copy.md @@ -4,13 +4,13 @@ The `copy` operator copies a value from one [field](/docs/types/field.md) to ano ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `copy` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `from` | required | The [field](/docs/types/field.md) to copy the value of. -| `to` | required | The [field](/docs/types/field.md) to copy the value into. -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `copy` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `from` | required | The [field](/docs/types/field.md) from which the value should be copied. | +| `to` | required | The [field](/docs/types/field.md) to which the value should be copied. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | ### Example Configurations: diff --git a/docs/operators/csv_parser.md b/docs/operators/csv_parser.md index 02ef5457..81ba2d9c 100644 --- a/docs/operators/csv_parser.md +++ b/docs/operators/csv_parser.md @@ -4,18 +4,18 @@ The `csv_parser` operator parses the string-type field selected by `parse_from` ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `csv_parser` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `header` | required | A string of delimited field names. The values in the delimited header will be used as keys | -| `delimiter` | `,` | A character that will be used as a delimiter. Values `\r` and `\n` cannot be used as a delimiter | -| `parse_from` | $body | A [field](/docs/types/field.md) that indicates the field to be parsed | -| `parse_to` | $body | A [field](/docs/types/field.md) that indicates the field to be parsed | -| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | -| `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator | -| `severity` | `nil` | An optional [severity](/docs/types/severity.md) block which will parse a severity field before passing the entry to the output operator | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `csv_parser` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `header` | required | A string of delimited field names. The values in the delimited header will be used as keys. | +| `delimiter` | `,` | A character that will be used as a delimiter. Values `\r` and `\n` cannot be used as a delimiter. | +| `parse_from` | $body | The [field](/docs/types/field.md) from which the value will be parsed. | +| `parse_to` | $body | The [field](/docs/types/field.md) to which the value will be parsed. | +| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | +| `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator. | +| `severity` | `nil` | An optional [severity](/docs/types/severity.md) block which will parse a severity field before passing the entry to the output operator. | ### Example Configurations diff --git a/docs/operators/file_input.md b/docs/operators/file_input.md index 93759061..5bf815d7 100644 --- a/docs/operators/file_input.md +++ b/docs/operators/file_input.md @@ -4,27 +4,27 @@ The `file_input` operator reads logs from files. It will place the lines read in ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `file_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `include` | required | A list of file glob patterns that match the file paths to be read | -| `exclude` | [] | A list of file glob patterns to exclude from reading | -| `poll_interval` | 200ms | The duration between filesystem polls | -| `multiline` | | A `multiline` configuration block. See below for details | -| `force_flush_period` | `500ms` | Time since last read of data from file, after which currently buffered log should be send to pipeline. Takes [duration](../types/duration.md) as value. Zero means waiting for new data forever | -| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry | -| `encoding` | `utf-8` | The encoding of the file being read. See the list of supported encodings below for available options | -| `include_file_name` | `true` | Whether to add the file name as the attribute `file.name` | -| `include_file_path` | `false` | Whether to add the file path as the attribute `file.path` | -| `include_file_name_resolved` | `false` | Whether to add the file name after symlinks resolution as the attribute `file.name.resolved` | -| `include_file_path_resolved` | `false` | Whether to add the file path after symlinks resolution as the attribute `file.path.resolved` | -| `start_at` | `end` | At startup, where to start reading logs from the file. Options are `beginning` or `end` | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `file_input` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `include` | required | A list of file glob patterns that match the file paths to be read. | +| `exclude` | [] | A list of file glob patterns to exclude from reading. | +| `poll_interval` | 200ms | The duration between filesystem polls. | +| `multiline` | | A `multiline` configuration block. See below for details. | +| `force_flush_period` | `500ms` | Time since last read of data from file, after which currently buffered log should be send to pipeline. Takes [duration](../types/duration.md) as value. Zero means waiting for new data forever. | +| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry. | +| `encoding` | `utf-8` | The encoding of the file being read. See the list of supported encodings below for available options. | +| `include_file_name` | `true` | Whether to add the file name as the attribute `file.name`. | +| `include_file_path` | `false` | Whether to add the file path as the attribute `file.path`. | +| `include_file_name_resolved` | `false` | Whether to add the file name after symlinks resolution as the attribute `file.name.resolved`. | +| `include_file_path_resolved` | `false` | Whether to add the file path after symlinks resolution as the attribute `file.path.resolved`. | +| `start_at` | `end` | At startup, where to start reading logs from the file. Options are `beginning` or `end`. | | `fingerprint_size` | `1kb` | The number of bytes with which to identify a file. The first bytes in the file are used as the fingerprint. Decreasing this value at any point will cause existing fingerprints to forgotten, meaning that all files will be read from the beginning (one time). | -| `max_log_size` | `1MiB` | The maximum size of a log entry to read before failing. Protects against reading large amounts of data into memory | +| `max_log_size` | `1MiB` | The maximum size of a log entry to read before failing. Protects against reading large amounts of data into memory |. | `max_concurrent_files` | 1024 | The maximum number of log files from which logs will be read concurrently (minimum = 2). If the number of files matched in the `include` pattern exceeds half of this number, then files will be processed in batches. One batch will be processed per `poll_interval`. | -| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes | -| `resource` | {} | A map of `key: value` pairs to add to the entry's resource | +| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes. | +| `resource` | {} | A map of `key: value` pairs to add to the entry's resource. | Note that by default, no logs will be read unless the monitored file is actively being written to because `start_at` defaults to `end`. diff --git a/docs/operators/file_output.md b/docs/operators/file_output.md index 6e1fb21c..a5b1fca4 100644 --- a/docs/operators/file_output.md +++ b/docs/operators/file_output.md @@ -4,11 +4,11 @@ The `file_output` operator will write log entries to a file. By default, they wi ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `file_output` | A unique identifier for the operator | -| `path` | required | A path to write the entries to | -| `format` | | A [go template](https://golang.org/pkg/text/template/) that will be used to render each entry into a log line | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `file_output` | A unique identifier for the operator. | +| `path` | required | The file path to which entries will be written. | +| `format` | | A [go template](https://golang.org/pkg/text/template/) that will be used to render each entry into a log line. | ### Example Configurations diff --git a/docs/operators/filter.md b/docs/operators/filter.md index 950fd425..88151bff 100644 --- a/docs/operators/filter.md +++ b/docs/operators/filter.md @@ -4,11 +4,11 @@ The `filter` operator filters incoming entries that match an expression. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `filter` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `expr` | required | Incoming entries that match this [expression](/docs/types/expression.md) will be dropped | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `filter` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `expr` | required | Incoming entries that match this [expression](/docs/types/expression.md) will be dropped. | | `drop_ratio` | 1.0 | The probability a matching entry is dropped (used for sampling). A value of 1.0 will drop 100% of matching entries, while a value of 0.0 will drop 0%. | ### Examples diff --git a/docs/operators/flatten.md b/docs/operators/flatten.md index eac1547c..3a1851d2 100644 --- a/docs/operators/flatten.md +++ b/docs/operators/flatten.md @@ -5,12 +5,12 @@ The operator only flattens a single level deep. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `flatten` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `field` | required | The [field](/docs/types/field.md) to be flattened. | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `flatten` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `field` | required | The [field](/docs/types/field.md) to be flattened. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | ### Example Configurations: diff --git a/docs/operators/generate_input.md b/docs/operators/generate_input.md index 57841458..640fc74e 100644 --- a/docs/operators/generate_input.md +++ b/docs/operators/generate_input.md @@ -5,14 +5,14 @@ coupled with the [`rate_limit`](/docs/operators/rate_limit.md) operator. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `generate_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `write_to` | `$body` | A [field](/docs/types/field.md) that will be set to the path of the file the entry was read from | -| `entry` | | A [entry](/docs/types/entry.md) log entry to repeatedly generate | -| `count` | 0 | The number of entries to generate before stopping. A value of 0 indicates unlimited | -| `static` | `false` | If true, the timestamp of the entry will remain static after each invocation | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `generate_input` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `write_to` | `$body` | A [field](/docs/types/field.md) that will be set to the path of the file the entry was read from. | +| `entry` | | A [entry](/docs/types/entry.md) log entry to repeatedly generate. | +| `count` | 0 | The number of entries to generate before stopping. A value of 0 indicates unlimited. | +| `static` | `false` | If true, the timestamp of the entry will remain static after each invocation. | ### Example Configurations diff --git a/docs/operators/journald_input.md b/docs/operators/journald_input.md index 1e3336b0..6c444e29 100644 --- a/docs/operators/journald_input.md +++ b/docs/operators/journald_input.md @@ -8,18 +8,18 @@ The `journald_input` operator will use the `__REALTIME_TIMESTAMP` field of the j ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `journald_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `directory` | | A directory containing journal files to read entries from | -| `files` | | A list of journal files to read entries from | -| `units` | | A list of units to read entries from | -| `priority` | `info` | Filter output by message priorities or priority ranges | -| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry | -| `start_at` | `end` | At startup, where to start reading logs from the file. Options are `beginning` or `end` | -| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes | -| `resource` | {} | A map of `key: value` pairs to add to the entry's resource | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `journald_input` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `directory` | | A directory containing journal files to read entries from. | +| `files` | | A list of journal files to read entries from. | +| `units` | | A list of units to read entries from. | +| `priority` | `info` | Filter output by message priorities or priority ranges. | +| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry. | +| `start_at` | `end` | At startup, where to start reading logs from the file. Options are `beginning` or `end`. | +| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes. | +| `resource` | {} | A map of `key: value` pairs to add to the entry's resource. | ### Example Configurations ```yaml diff --git a/docs/operators/json_parser.md b/docs/operators/json_parser.md index 9fbb135a..6fe3f219 100644 --- a/docs/operators/json_parser.md +++ b/docs/operators/json_parser.md @@ -4,17 +4,17 @@ The `json_parser` operator parses the string-type field selected by `parse_from` ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `json_parser` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `parse_from` | `$body` | A [field](/docs/types/field.md) that indicates the field that should be parsed | -| `parse_to` | `$body` | A [field](/docs/types/field.md) that indicates the field to which values will be parsed | -| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `json_parser` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `parse_from` | `$body` | The [field](/docs/types/field.md) from which the value will be parsed. | +| `parse_to` | `$body` | The [field](/docs/types/field.md) to which the value will be parsed. | +| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | -| `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator | -| `severity` | `nil` | An optional [severity](/docs/types/severity.md) block which will parse a severity field before passing the entry to the output operator | +| `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator. | +| `severity` | `nil` | An optional [severity](/docs/types/severity.md) block which will parse a severity field before passing the entry to the output operator. | ### Example Configurations diff --git a/docs/operators/k8s_event_input.md b/docs/operators/k8s_event_input.md index 556142ef..13953660 100644 --- a/docs/operators/k8s_event_input.md +++ b/docs/operators/k8s_event_input.md @@ -5,16 +5,16 @@ Kubernetes API, and currently requires that Stanza is running inside a Kubernete ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `k8s_event_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `namespaces` | All namespaces | An array of namespaces to collect events from. | -| `discover_namespaces` | `true` | If true, the operator will regularly poll for new namespaces to include | -| `discovery_interval ` | `1m` | The interval at which the operator searches for new namespaces to follow | -| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry | -| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes | -| `resource` | {} | A map of `key: value` pairs to add to the entry's resource | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `k8s_event_input` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `namespaces` | All namespaces | An array of namespaces to collect events from.. | +| `discover_namespaces` | `true` | If true, the operator will regularly poll for new namespaces to include. | +| `discovery_interval ` | `1m` | The interval at which the operator searches for new namespaces to follow. | +| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry. | +| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes. | +| `resource` | {} | A map of `key: value` pairs to add to the entry's resource. | ### Example Configurations diff --git a/docs/operators/metadata.md b/docs/operators/metadata.md index b5a013f9..4e734b59 100644 --- a/docs/operators/metadata.md +++ b/docs/operators/metadata.md @@ -4,13 +4,13 @@ The `metadata` operator adds attributes to incoming entries. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `metadata` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes | -| `resource` | {} | A map of `key: value` pairs to add to the entry's resource | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `metadata` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes. | +| `resource` | {} | A map of `key: value` pairs to add to the entry's resource. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | Inside the label values, an [expression](/docs/types/expression.md) surrounded by `EXPR()` will be replaced with the evaluated form of the expression. The entry's body can be accessed diff --git a/docs/operators/move.md b/docs/operators/move.md index 33c4ac83..5ca3d8c2 100644 --- a/docs/operators/move.md +++ b/docs/operators/move.md @@ -4,13 +4,13 @@ The `move` operator moves (or renames) a field from one location to another. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `move` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `from` | required | The [field](/docs/types/field.md) to move the value out of. -| `to` | required | The [field](/docs/types/field.md) to move the value into. -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `move` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `from` | required | The [field](/docs/types/field.md) from which the value will be moved. | +| `to` | required | The [field](/docs/types/field.md) to which the value will be moved. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | ### Example Configurations: diff --git a/docs/operators/recombine.md b/docs/operators/recombine.md index bfd1a2bb..c2d5bae7 100644 --- a/docs/operators/recombine.md +++ b/docs/operators/recombine.md @@ -4,16 +4,16 @@ The `recombine` operator combines consecutive logs into single logs based on sim ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `metadata` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | -| `is_first_entry` | | An [expression](/docs/types/expression.md) that returns true if the entry being processed is the first entry in a multiline series | -| `is_last_entry` | | An [expression](/docs/types/expression.md) that returns true if the entry being processed is the last entry in a multiline series | -| `combine_field` | required | The [field](/docs/types/field.md) from all the entries that will recombined with newlines | -| `max_batch_size` | 1000 | The maximum number of consecutive entries that will be combined into a single entry | -| `overwrite_with` | `oldest` | Whether to use the fields from the `oldest` or the `newest` entry for all the fields that are not combined with newlines | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `metadata` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | +| `is_first_entry` | | An [expression](/docs/types/expression.md) that returns true if the entry being processed is the first entry in a multiline series. | +| `is_last_entry` | | An [expression](/docs/types/expression.md) that returns true if the entry being processed is the last entry in a multiline series. | +| `combine_field` | required | The [field](/docs/types/field.md) from all the entries that will recombined with newlines. | +| `max_batch_size` | 1000 | The maximum number of consecutive entries that will be combined into a single entry. | +| `overwrite_with` | `oldest` | Whether to use the fields from the `oldest` or the `newest` entry for all the fields that are not combined with newlines. | Exactly one of `is_first_entry` and `is_last_entry` must be specified. diff --git a/docs/operators/regex_parser.md b/docs/operators/regex_parser.md index 4194a29c..c275a25b 100644 --- a/docs/operators/regex_parser.md +++ b/docs/operators/regex_parser.md @@ -8,18 +8,18 @@ This operator makes use of [Go regular expression](https://github.com/google/re2 ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `regex_parser` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `regex` | required | A [Go regular expression](https://github.com/google/re2/wiki/Syntax). The named capture groups will be extracted as fields in the parsed body. | -| `parse_from` | `$body` | A [field](/docs/types/field.md) that indicates the field from which values should be parsed | -| `parse_to` | `$body` | A [field](/docs/types/field.md) that indicates the field to which values will be parsed | -| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `regex_parser` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `regex` | required | A [Go regular expression](https://github.com/google/re2/wiki/Syntax). The named capture groups will be extracted as fields in the parsed body. | +| `parse_from` | `$body` | The [field](/docs/types/field.md) from which the value will be parsed. | +| `parse_to` | `$body` | The [field](/docs/types/field.md) to which the value will be parsed. | +| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | -| `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator | -| `severity` | `nil` | An optional [severity](/docs/types/severity.md) block which will parse a severity field before passing the entry to the output operator | +| `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator. | +| `severity` | `nil` | An optional [severity](/docs/types/severity.md) block which will parse a severity field before passing the entry to the output operator. | ### Example Configurations diff --git a/docs/operators/remove.md b/docs/operators/remove.md index bbd81bc6..652e119d 100644 --- a/docs/operators/remove.md +++ b/docs/operators/remove.md @@ -4,12 +4,12 @@ The `remove` operator removes a field from a record. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `remove` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `field` | required | The [field](/docs/types/field.md) to remove. if '$attributes' or '$resource' is specified, all fields of that type will be removed. -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `remove` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `field` | required | The [field](/docs/types/field.md) to remove. if '$attributes' or '$resource' is specified, all fields of that type will be removed. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | ### Example Configurations: diff --git a/docs/operators/restructure.md b/docs/operators/restructure.md index ee498e7c..ea120ee2 100644 --- a/docs/operators/restructure.md +++ b/docs/operators/restructure.md @@ -7,12 +7,12 @@ they are defined. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `restructure` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `ops` | required | A list of ops. The available op types are defined below | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `restructure` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `ops` | required | A list of ops. The available op types are defined below. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | ### Op types diff --git a/docs/operators/retain.md b/docs/operators/retain.md index b805bdfd..82e33c1b 100644 --- a/docs/operators/retain.md +++ b/docs/operators/retain.md @@ -4,12 +4,12 @@ The `retain` operator keeps the specified list of fields, and removes the rest. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `retain` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `fields` | required | A list of [fields](/docs/types/field.md) to be kept. | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `retain` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `fields` | required | A list of [fields](/docs/types/field.md) to be kept. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. |
NOTE: If no fields in a group (attributes, resource, or body) are specified, that entire group will be retained. diff --git a/docs/operators/router.md b/docs/operators/router.md index 13343e89..75a9559b 100644 --- a/docs/operators/router.md +++ b/docs/operators/router.md @@ -10,19 +10,19 @@ An entry that does not match any of the routes is dropped and not processed furt ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `router` | A unique identifier for the operator | -| `routes` | required | A list of routes. See below for details | -| `default` | | The operator(s) that will receive any entries not matched by any of the routes | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `router` | A unique identifier for the operator. | +| `routes` | required | A list of routes. See below for details. | +| `default` | | The operator(s) that will receive any entries not matched by any of the routes. | #### Route configuration -| Field | Default | Description | -| --- | --- | --- | -| `output` | required | The connected operator(s) that will receive all outbound entries for this route | -| `expr` | required | An [expression](/docs/types/expression.md) that returns a boolean. The body of the routed entry is available as `$` | -| `attributes` | {} | A map of `key: value` pairs to add to an entry that matches the route | +| Field | Default | Description | +| --- | --- | --- | +| `output` | required | The connected operator(s) that will receive all outbound entries for this route. | +| `expr` | required | An [expression](/docs/types/expression.md) that returns a boolean. The body of the routed entry is available as `$`. | +| `attributes` | {} | A map of `key: value` pairs to add to an entry that matches the route. | ### Examples diff --git a/docs/operators/severity_parser.md b/docs/operators/severity_parser.md index 1da51b6e..07e6556b 100644 --- a/docs/operators/severity_parser.md +++ b/docs/operators/severity_parser.md @@ -4,16 +4,16 @@ The `severity_parser` operator sets the severity on an entry by parsing a value ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | required | A unique identifier for the operator | -| `output` | required | The `id` for the operator to send parsed entries to | -| `parse_from` | required | A [field](/docs/types/field.md) that indicates the field to be parsed as JSON | -| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | -| `preset` | `default` | A predefined set of values that should be interpreted at specific severity levels | -| `mapping` | | A formatted set of values that should be interpreted as severity levels. | -| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `severity_parser` | A unique identifier for the operator. | +| `output` | Next in pipeline | The `id` for the operator to send parsed entries to. | +| `parse_from` | required | The [field](/docs/types/field.md) from which the value will be parsed. | +| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | +| `preset` | `default` | A predefined set of values that should be interpreted at specific severity levels. | +| `mapping` | | A formatted set of values that should be interpreted as severity levels. | +| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | ### Example Configurations diff --git a/docs/operators/stanza_input.md b/docs/operators/stanza_input.md index 4140f0ff..61033271 100644 --- a/docs/operators/stanza_input.md +++ b/docs/operators/stanza_input.md @@ -6,11 +6,11 @@ Care should be taken when doing any additional processing of logs coming from th ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `stanza_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `buffer_size` | 100 | The number of entries to buffer before dropping entries because we aren't processing fast enough | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `stanza_input` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `buffer_size` | 100 | The number of entries to buffer before dropping entries because we aren't processing fast enough. | ### Example Configurations diff --git a/docs/operators/stdin.md b/docs/operators/stdin.md index 2382410b..e886852e 100644 --- a/docs/operators/stdin.md +++ b/docs/operators/stdin.md @@ -4,11 +4,11 @@ The `stdin` generates entries from lines written to stdin. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `generate_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `write_to` | `$body` | A [field](/docs/types/field.md) that will be set to the path of the file the entry was read from | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `stdin` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `write_to` | `$body` | A [field](/docs/types/field.md) that will be set to the path of the file the entry was read from. | ### Example Configurations diff --git a/docs/operators/stdout.md b/docs/operators/stdout.md index ffb7b622..cbd2d668 100644 --- a/docs/operators/stdout.md +++ b/docs/operators/stdout.md @@ -5,9 +5,9 @@ or running one-time batch processing jobs. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | required | A unique identifier for the operator | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `stdout` | A unique identifier for the operator. | ### Example Configurations diff --git a/docs/operators/syslog_input.md b/docs/operators/syslog_input.md index efebc8a4..596bad91 100644 --- a/docs/operators/syslog_input.md +++ b/docs/operators/syslog_input.md @@ -4,15 +4,15 @@ The `syslog_input` operator listens for syslog format logs from UDP/TCP packages ### Configuration Fields -| Field | Default | Description | -| ---------- | ---------------- | ------------------------------------------------------------ | -| `id` | `syslog_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `syslog_input` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | | `tcp` | {} | A [tcp_input config](./tcp_input.md#configuration-fields) to defined syslog_parser operator. | | `udp` | {} | A [udp_input config](./udp_input.md#configuration-fields) to defined syslog_parser operator. | | `syslog` | required | A [syslog parser config](./syslog_parser.md#configuration-fields) to defined syslog_parser operator. | -| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes | -| `resource` | {} | A map of `key: value` pairs to add to the entry's resource | +| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes. | +| `resource` | {} | A map of `key: value` pairs to add to the entry's resource. | diff --git a/docs/operators/syslog_parser.md b/docs/operators/syslog_parser.md index 0f861174..83dec741 100644 --- a/docs/operators/syslog_parser.md +++ b/docs/operators/syslog_parser.md @@ -4,15 +4,15 @@ The `syslog_parser` operator parses the string-type field selected by `parse_fro ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `syslog_parser` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `parse_from` | `$body` | A [field](/docs/types/field.md) that indicates the field from which value should be parsed | -| `parse_to` | `$body` | A [field](/docs/types/field.md) that indicates the field to which values will be parsed | -| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | -| `protocol` | required | The protocol to parse the syslog messages as. Options are `rfc3164` and `rfc5424` | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `syslog_parser` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `parse_from` | `$body` | The [field](/docs/types/field.md) from which the value will be parsed. | +| `parse_to` | `$body` | The [field](/docs/types/field.md) to which the value will be parsed. | +| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | +| `protocol` | required | The protocol to parse the syslog messages as. Options are `rfc3164` and `rfc5424`. | | `location` | `UTC` | The geographic location (timezone) to use when parsing the timestamp (Syslog RFC 3164 only). The available locations depend on the local IANA Time Zone database. [This page](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) contains many examples, such as `America/New_York`. | | `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator | | `severity` | `nil` | An optional [severity](/docs/types/severity.md) block which will parse a severity field before passing the entry to the output operator | diff --git a/docs/operators/tcp_input.md b/docs/operators/tcp_input.md index 05d1d36d..1bd83312 100644 --- a/docs/operators/tcp_input.md +++ b/docs/operators/tcp_input.md @@ -4,19 +4,19 @@ The `tcp_input` operator listens for logs on one or more TCP connections. The op ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `tcp_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `max_log_size` | `1MiB` | The maximum size of a log entry to read before failing. Protects against reading large amounts of data into memory | -| `listen_address` | required | A listen address of the form `:` | -| `tls` | nil | An optional `TLS` configuration (see the TLS configuration section) | -| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry | -| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes | -| `resource` | {} | A map of `key: value` pairs to add to the entry's resource | -| `add_attributes` | false | Adds `net.*` attributes according to [semantic convention][https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md#general-network-connection-attributes] | -| `multiline` | | A `multiline` configuration block. See below for details | -| `encoding` | `utf-8` | The encoding of the file being read. See the list of supported encodings below for available options | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `tcp_input` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `max_log_size` | `1MiB` | The maximum size of a log entry to read before failing. Protects against reading large amounts of data into memory. | +| `listen_address` | required | A listen address of the form `:`. | +| `tls` | nil | An optional `TLS` configuration (see the TLS configuration section). | +| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry. | +| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes. | +| `resource` | {} | A map of `key: value` pairs to add to the entry's resource. | +| `add_attributes` | false | Adds `net.*` attributes according to [semantic convention][https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md#general-network-connection-attributes]. | +| `multiline` | | A `multiline` configuration block. See below for details. | +| `encoding` | `utf-8` | The encoding of the file being read. See the list of supported encodings below for available options. | #### TLS Configuration diff --git a/docs/operators/time_parser.md b/docs/operators/time_parser.md index 1e9f8301..efadfd2d 100644 --- a/docs/operators/time_parser.md +++ b/docs/operators/time_parser.md @@ -4,16 +4,16 @@ The `time_parser` operator sets the timestamp on an entry by parsing a value fro ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | required | A unique identifier for the operator | -| `output` | required | The connected operator(s) that will receive all outbound entries | -| `parse_from` | required | A [field](/docs/types/field.md) that indicates the field to be parsed as JSON | -| `layout_type` | `strptime` | The type of timestamp. Valid values are `strptime`, `gotime`, and `epoch` | -| `layout` | required | The exact layout of the timestamp to be parsed | -| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | -| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `time_parser` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `parse_from` | required | The [field](/docs/types/field.md) from which the value will be parsed. | +| `layout_type` | `strptime` | The type of timestamp. Valid values are `strptime`, `gotime`, and `epoch`. | +| `layout` | required | The exact layout of the timestamp to be parsed. | +| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | +| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | ### Example Configurations diff --git a/docs/operators/trace_parser.md b/docs/operators/trace_parser.md new file mode 100644 index 00000000..5fafb8ea --- /dev/null +++ b/docs/operators/trace_parser.md @@ -0,0 +1,19 @@ +## `trace_parser` operator + +The `trace_parser` operator sets the trace on an entry by parsing a value from the body. + +### Configuration Fields + +| Field | Default | Description | +| --- | --- | --- | +| `id` | `trace_parser` | A unique identifier for the operator. | +| `output` | Next in pipeline | The `id` for the operator to send parsed entries to. | +| `trace_id.parse_from` | `trace_id` | A [field](/docs/types/field.md) that indicates the field to be parsed as a trace ID. | +| `span_id.parse_from` | `span_id` | A [field](/docs/types/field.md) that indicates the field to be parsed as a span ID. | +| `trace_flags.parse_from` | `trace_flags` | A [field](/docs/types/field.md) that indicates the field to be parsed as trace flags. | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | + + +### Example Configurations + +Several detailed examples are available [here](/docs/types/trace.md). diff --git a/docs/operators/udp_input.md b/docs/operators/udp_input.md index 5fc8a5ce..1244261b 100644 --- a/docs/operators/udp_input.md +++ b/docs/operators/udp_input.md @@ -4,17 +4,17 @@ The `udp_input` operator listens for logs from UDP packets. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `udp_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `listen_address` | required | A listen address of the form `:` | -| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry | -| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes | -| `resource` | {} | A map of `key: value` pairs to add to the entry's resource | -| `add_attributes` | false | Adds `net.*` attributes according to [semantic convention][https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md#general-network-connection-attributes] | -| `multiline` | | A `multiline` configuration block. See below for details | -| `encoding` | `utf-8` | The encoding of the file being read. See the list of supported encodings below for available options | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `udp_input` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `listen_address` | required | A listen address of the form `:`. | +| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry. | +| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes. | +| `resource` | {} | A map of `key: value` pairs to add to the entry's resource. | +| `add_attributes` | false | Adds `net.*` attributes according to [semantic convention][https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md#general-network-connection-attributes]. | +| `multiline` | | A `multiline` configuration block. See below for details. | +| `encoding` | `utf-8` | The encoding of the file being read. See the list of supported encodings below for available options. | #### `multiline` configuration diff --git a/docs/operators/uri_parser.md b/docs/operators/uri_parser.md index b36cf22d..740ea200 100644 --- a/docs/operators/uri_parser.md +++ b/docs/operators/uri_parser.md @@ -13,14 +13,14 @@ The `uri_parser` operator parses the string-type field selected by `parse_from` ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `uri_parser` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `parse_from` | `$body` | A [field](/docs/types/field.md) that indicates the field to be parsed as JSON | -| `parse_to` | `$body` | A [field](/docs/types/field.md) that indicates the field to be parsed as JSON | -| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) | -| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md) | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `uri_parser` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `parse_from` | `$body` | The [field](/docs/types/field.md) from which the value will be parsed. | +| `parse_to` | `$body` | The [field](/docs/types/field.md) to which the value will be parsed. | +| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). | +| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). | | `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. | @@ -28,13 +28,13 @@ The `uri_parser` operator parses the string-type field selected by `parse_from` The following fields are returned. Empty fields are not returned. -| Field | Type | Example | Description | -| --- | --- | --- | --- | -| scheme | `string` | `"http"` | [URI Scheme](https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml). HTTP, HTTPS, FTP, etc. | -| user | `string` | `"dev"` | [Userinfo](https://tools.ietf.org/html/rfc3986#section-3.2) username. Password is always ignored. | -| host | `string` | `"golang.org"` | The [hostname](https://tools.ietf.org/html/rfc3986#section-3.2.2) such as `www.example.com`, `example.com`, `example`. A scheme is required in order to parse the `host` field. | -| port | `string` | `"8443"` | The [port](https://tools.ietf.org/html/rfc3986#section-3.2.3) the request is sent to. A scheme is required in order to parse the `port` field. | -| path | `string` | `"/v1/app"` | URI request [path](https://tools.ietf.org/html/rfc3986#section-3.3). | +| Field | Type | Example | Description | +| --- | --- | --- | --- | +| scheme | `string` | `"http"` | [URI Scheme](https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml). HTTP, HTTPS, FTP, etc. | +| user | `string` | `"dev"` | [Userinfo](https://tools.ietf.org/html/rfc3986#section-3.2) username. Password is always ignored. | +| host | `string` | `"golang.org"` | The [hostname](https://tools.ietf.org/html/rfc3986#section-3.2.2) such as `www.example.com`, `example.com`, `example`. A scheme is required in order to parse the `host` field. | +| port | `string` | `"8443"` | The [port](https://tools.ietf.org/html/rfc3986#section-3.2.3) the request is sent to. A scheme is required in order to parse the `port` field. | +| path | `string` | `"/v1/app"` | URI request [path](https://tools.ietf.org/html/rfc3986#section-3.3). | | query | `map[string][]string` | `"query":{"user":["admin"]}` | Parsed URI [query string](https://tools.ietf.org/html/rfc3986#section-3.4). | diff --git a/docs/operators/windows_eventlog_input.md b/docs/operators/windows_eventlog_input.md index d1191e9e..b8cabc64 100644 --- a/docs/operators/windows_eventlog_input.md +++ b/docs/operators/windows_eventlog_input.md @@ -4,17 +4,17 @@ The `windows_eventlog_input` operator reads logs from the windows event log API. ### Configuration Fields -| Field | Default | Description | -| --- | --- | --- | -| `id` | `windows_eventlog_input` | A unique identifier for the operator | -| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries | -| `channel` | required | The windows event log channel to monitor | -| `max_reads` | 100 | The maximum number of bodies read into memory, before beginning a new batch | -| `start_at` | `end` | On first startup, where to start reading logs from the API. Options are `beginning` or `end` | -| `poll_interval` | 1s | The interval at which the channel is checked for new log entries. This check begins again after all new bodies have been read | -| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry | -| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes | -| `resource` | {} | A map of `key: value` pairs to add to the entry's resource | +| Field | Default | Description | +| --- | --- | --- | +| `id` | `windows_eventlog_input` | A unique identifier for the operator. | +| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. | +| `channel` | required | The windows event log channel to monitor. | +| `max_reads` | 100 | The maximum number of bodies read into memory, before beginning a new batch. | +| `start_at` | `end` | On first startup, where to start reading logs from the API. Options are `beginning` or `end`. | +| `poll_interval` | 1s | The interval at which the channel is checked for new log entries. This check begins again after all new bodies have been read. | +| `write_to` | `$body` | The body [field](/docs/types/field.md) written to when creating a new log entry. | +| `attributes` | {} | A map of `key: value` pairs to add to the entry's attributes. | +| `resource` | {} | A map of `key: value` pairs to add to the entry's resource. | ### Example Configurations diff --git a/docs/types/entry.md b/docs/types/entry.md index 4dcf0a6d..dbfd6ee5 100644 --- a/docs/types/entry.md +++ b/docs/types/entry.md @@ -3,13 +3,13 @@ Entry is the base representation of log data as it moves through a pipeline. All operators either create, modify, or consume entries. ## Structure -| Field | Description | -| --- | --- | -| `timestamp` | The timestamp associated with the log (RFC 3339). | -| `severity` | The [severity](/docs/types/field.md) of the log. | -| `severity_text` | The original text that was interpreted as a [severity](/docs/types/field.md). | -| `resource` | A map of key/value pairs that describe the resource from which the log originated. | -| `attributes` | A map of key/value pairs that provide additional context to the log. This value is often used by a consumer to filter logs. | +| Field | Description | +| --- | --- | +| `timestamp` | The timestamp associated with the log (RFC 3339). | +| `severity` | The [severity](/docs/types/field.md) of the log. | +| `severity_text` | The original text that was interpreted as a [severity](/docs/types/field.md). | +| `resource` | A map of key/value pairs that describe the resource from which the log originated. | +| `attributes` | A map of key/value pairs that provide additional context to the log. This value is often used by a consumer to filter logs. | | `body` | The contents of the log. This value is often modified and restructured in the pipeline. It may be a string, number, or object. | diff --git a/docs/types/severity.md b/docs/types/severity.md index 0ac739f5..da5521a3 100644 --- a/docs/types/severity.md +++ b/docs/types/severity.md @@ -8,12 +8,12 @@ Severity is represented as a number from 1 to 24. The meaning of these severity Parser operators can parse a severity and attach the resulting value to a log entry. -| Field | Default | Description | -| --- | --- | --- | -| `parse_from` | required | A [field](/docs/types/field.md) that indicates the field to be parsed as JSON | -| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) | -| `preset` | `default` | A predefined set of values that should be interpretted at specific severity levels | -| `mapping` | | A custom set of values that should be interpretted at designated severity levels | +| Field | Default | Description | +| --- | --- | --- | +| `parse_from` | required | The [field](/docs/types/field.md) from which the value will be parsed. | +| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). | +| `preset` | `default` | A predefined set of values that should be interpretted at specific severity levels. | +| `mapping` | | A custom set of values that should be interpretted at designated severity levels. | ### How severity `mapping` works diff --git a/docs/types/timestamp.md b/docs/types/timestamp.md index cfe9ec37..50f1f84c 100644 --- a/docs/types/timestamp.md +++ b/docs/types/timestamp.md @@ -2,12 +2,12 @@ Parser operators can parse a timestamp and attach the resulting time value to a log entry. -| Field | Default | Description | -| --- | --- | --- | -| `parse_from` | required | A [field](/docs/types/field.md) that indicates the field to be parsed as JSON | -| `layout_type` | `strptime` | The type of timestamp. Valid values are `strptime`, `gotime`, and `epoch` | -| `layout` | required | The exact layout of the timestamp to be parsed | -| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md) | +| Field | Default | Description | +| --- | --- | --- | +| `parse_from` | required | The [field](/docs/types/field.md) from which the value will be parsed. | +| `layout_type` | `strptime` | The type of timestamp. Valid values are `strptime`, `gotime`, and `epoch`. | +| `layout` | required | The exact layout of the timestamp to be parsed. | +| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). | | `location` | `Local` | The geographic location (timezone) to use when parsing a timestamp that does not include a timezone. The available locations depend on the local IANA Time Zone database. [This page](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) contains many examples, such as `America/New_York`. | diff --git a/docs/types/trace.md b/docs/types/trace.md new file mode 100644 index 00000000..d95cc75c --- /dev/null +++ b/docs/types/trace.md @@ -0,0 +1,47 @@ +## Trace Parsing + +Traces context fields are defined in the [OpenTelemetry Logs Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#trace-context-fields). + + +### `trace` parsing parameters + +Parser operators can parse a trace context and attach the resulting values to a log entry. + +| Field | Default | Description | +| --- | --- | --- | +| `trace_id.parse_from` | `trace_id` | A [field](/docs/types/field.md) that indicates the field to be parsed as a trace ID. | +| `span_id.parse_from` | `span_id` | A [field](/docs/types/field.md) that indicates the field to be parsed as a span ID. | +| `trace_flags.parse_from` | `trace_flags` | A [field](/docs/types/field.md) that indicates the field to be parsed as trace flags. | + + +### How to use trace parsing + +All parser operators, such as [`regex_parser`](/docs/operators/regex_parser.md) support these fields inside of a `trace` block. + +If a `trace` block is specified, the parser operator will perform the trace parsing _after_ performing its other parsing actions, but _before_ passing the entry to the specified output operator. + +```yaml +- type: regex_parser + regexp: '^TraceID=(?P\S*) SpanID=(?P\S*) TraceFlags=(?P\d*)' + trace: + trace_id: + parse_from: trace_id + span_id: + parse_from: span_id + trace_flags: + parse_from: trace_flags +``` + +--- + +As a special case, the [`trace_parser`](/docs/operators/trace_parser.md) operator supports these fields inline. This is because trace parsing is the primary purpose of the operator. + +```yaml +- type: trace_parser + trace_id: + parse_from: trace_id + span_id: + parse_from: span_id + trace_flags: + parse_from: trace_flags +```