Skip to content

Commit

Permalink
Remove '$' from field syntax (#364)
Browse files Browse the repository at this point in the history
* Remove '$' from field syntax

* Enforce rule that body fields must start with 'body'
  • Loading branch information
djaglowski authored Mar 24, 2022
1 parent 1e603b9 commit fd659de
Show file tree
Hide file tree
Showing 124 changed files with 501 additions and 437 deletions.
10 changes: 5 additions & 5 deletions docs/operators/add.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Add a value to the body using an expression
```yaml
- type: add
field: key2
value: EXPR($.key1 + "_suffix")
value: EXPR(body.key1 + "_suffix")
```
<table>
Expand Down Expand Up @@ -150,7 +150,7 @@ Add a value to attributes

```yaml
- type: add
field: $attributes.key2
field: attributes.key2
value: val2
```
Expand Down Expand Up @@ -193,7 +193,7 @@ Add a value to resource

```yaml
- type: add
field: $resource.key2
field: resource.key2
value: val2
```
Expand Down Expand Up @@ -235,8 +235,8 @@ Add a value to resource using an expression

```yaml
- type: add
field: $resource.key2
value: EXPR($.key1 + "_suffix")
field: resource.key2
value: EXPR(body.key1 + "_suffix")
```
<table>
Expand Down
6 changes: 3 additions & 3 deletions docs/operators/copy.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Copy a value from the body to resource
```yaml
- type: copy
from: key
to: $resource.newkey
to: resource.newkey
```
<table>
Expand Down Expand Up @@ -64,7 +64,7 @@ Copy a value from the body to attributes
```yaml
- type: copy
from: key2
to: $attributes.newkey
to: attributes.newkey
```
<table>
Expand Down Expand Up @@ -108,7 +108,7 @@ Copy a value from the body to attributes
Copy a value from attributes to the body
```yaml
- type: copy
from: $attributes.key
from: attributes.key
to: newkey
```
Expand Down
4 changes: 2 additions & 2 deletions docs/operators/csv_parser.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ The `csv_parser` operator parses the string-type field selected by `parse_from`
| `header_attribute` | required when `header` not set | An attribute name to read the header field from, to support dynamic field names |
| `delimiter` | `,` | A character that will be used as a delimiter. Values `\r` and `\n` cannot be used as a delimiter. |
| `lazy_quotes` | `false` | If true, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. |
| `parse_from` | $body | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | $body | The [field](/docs/types/field.md) to which the value will be parsed. |
| `parse_from` | `body` | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | `body` | The [field](/docs/types/field.md) to which the value will be parsed. |
| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). |
| `timestamp` | `nil` | An optional [timestamp](/docs/types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator. |
Expand Down
6 changes: 3 additions & 3 deletions docs/operators/filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,22 +17,22 @@ The `filter` operator filters incoming entries that match an expression.

```yaml
- type: filter
expr: '$body.message matches "^LOG: .* END$"'
expr: 'body.message matches "^LOG: .* END$"'
output: my_output
```
#### Filter entries based on a label value
```yaml
- type: filter
expr: '$attributes.env == "production"'
expr: 'attributes.env == "production"'
output: my_output
```
#### Filter entries based on an environment variable
```yaml
- type: filter
expr: '$body.message == env("MY_ENV_VARIABLE")'
expr: 'body.message == env("MY_ENV_VARIABLE")'
output: my_output
```
4 changes: 2 additions & 2 deletions docs/operators/json_parser.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ The `json_parser` operator parses the string-type field selected by `parse_from`
| --- | --- | --- |
| `id` | `json_parser` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `parse_from` | `$body` | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | `$body` | The [field](/docs/types/field.md) to which the value will be parsed. |
| `parse_from` | `body` | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | `body` | The [field](/docs/types/field.md) to which the value will be parsed. |
| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). |
| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. |
Expand Down
2 changes: 1 addition & 1 deletion docs/operators/metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Configuration:
- type: metadata
output: metadata_receiver
attributes:
environment: 'EXPR( $.environment == "production" ? "prod" : "dev" )'
environment: 'EXPR( body.environment == "production" ? "prod" : "dev" )'
```
<table>
Expand Down
8 changes: 4 additions & 4 deletions docs/operators/move.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Move a value from the body to attributes
```yaml
- type: move
from: ip
to: $attributes.ip
to: attributes.ip
```
<table>
Expand Down Expand Up @@ -145,7 +145,7 @@ Replace the body with an individual value nested within the body
```yaml
- type: move
from: log
to: $body
to: body
```
<table>
Expand Down Expand Up @@ -184,7 +184,7 @@ Remove a layer from the body
```yaml
- type: move
from: wrapper
to: $body
to: body
```
<table>
Expand Down Expand Up @@ -232,7 +232,7 @@ Merge a layer to the body
```yaml
- type: move
from: object
to: $body
to: body
```
<table>
Expand Down
4 changes: 2 additions & 2 deletions docs/operators/recombine.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Configuration:
- type: recombine
combine_field: message
combine_with: ""
is_last_entry: "$body.logtag == 'F'"
is_last_entry: "body.logtag == 'F'"
overwrite_with: "newest"
```
Expand Down Expand Up @@ -101,7 +101,7 @@ This can be expressed with the following configuration:
```yaml
- type: recombine
combine_field: message
is_first_entry: $body.message matches "^[^\s]"
is_first_entry: body.message matches "^[^\s]"
```
Given the following input file:
Expand Down
8 changes: 4 additions & 4 deletions docs/operators/regex_parser.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ This operator makes use of [Go regular expression](https://github.com/google/re2
| `id` | `regex_parser` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `regex` | required | A [Go regular expression](https://github.com/google/re2/wiki/Syntax). The named capture groups will be extracted as fields in the parsed body. |
| `parse_from` | `$body` | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | `$body` | The [field](/docs/types/field.md) to which the value will be parsed. |
| `parse_from` | `body` | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | `body` | The [field](/docs/types/field.md) to which the value will be parsed. |
| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). |
| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. |
Expand Down Expand Up @@ -162,7 +162,7 @@ Configuration:
- type: regex_parser
regex: '^Host=(?<host>)$'
parse_from: message
if: '$body.type == "hostname"'
if: 'body.type == "hostname"'
```
<table>
Expand Down Expand Up @@ -229,7 +229,7 @@ Configuration:
- type: regex_parser
regex: '^Host=(?<host>)$'
parse_from: message
if: '$body.type == "hostname"'
if: 'body.type == "hostname"'
```
<table>
Expand Down
10 changes: 5 additions & 5 deletions docs/operators/remove.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The `remove` operator removes a field from a record.
| --- | --- | --- |
| `id` | `remove` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `field` | required | The [field](/docs/types/field.md) to remove. if '$attributes' or '$resource' is specified, all fields of that type will be removed. |
| `field` | required | The [field](/docs/types/field.md) to remove. if 'attributes' or 'resource' is specified, all fields of that type will be removed. |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). |
| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. |

Expand Down Expand Up @@ -100,7 +100,7 @@ Remove an object from the body
Remove a value from attributes
```yaml
- type: remove
field: $attributes.otherkey
field: attributes.otherkey
```
<table>
Expand Down Expand Up @@ -142,7 +142,7 @@ Remove a value from attributes
Remove a value from resource
```yaml
- type: remove
field: $resource.otherkey
field: resource.otherkey
```
<table>
Expand Down Expand Up @@ -184,7 +184,7 @@ Remove a value from resource
Remove all resource fields
```yaml
- type: remove
field: $resource
field: resource
```
<table>
Expand Down Expand Up @@ -227,7 +227,7 @@ Remove all resource fields
Remove all attributes
```yaml
- type: remove
field: $attributes
field: attributes
```
<table>
Expand Down
12 changes: 6 additions & 6 deletions docs/operators/retain.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,8 @@ Retain fields from resource
```yaml
- type: retain
fields:
- $resource.key1
- $resource.key2
- resource.key1
- resource.key2
```
<table>
Expand Down Expand Up @@ -165,8 +165,8 @@ Retain fields from attributes
```yaml
- type: retain
fields:
- $attributes.key1
- $attributes.key2
- attributes.key1
- attributes.key2
```
<table>
Expand Down Expand Up @@ -214,8 +214,8 @@ Retain fields from all sources
```yaml
- type: retain
fields:
- $resource.key1
- $attributes.key3
- resource.key1
- attributes.key3
- key5
```
Expand Down
8 changes: 4 additions & 4 deletions docs/operators/router.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ An entry that does not match any of the routes is dropped and not processed furt
- type: router
routes:
- output: my_json_parser
expr: '$.format == "json"'
expr: 'body.format == "json"'
- output: my_syslog_parser
expr: '$.format == "syslog"'
expr: 'body.format == "syslog"'
```
#### Drop entries based on content
Expand All @@ -44,7 +44,7 @@ An entry that does not match any of the routes is dropped and not processed furt
- type: router
routes:
- output: my_output
expr: '$.message matches "^LOG: .* END$"'
expr: 'body.message matches "^LOG: .* END$"'
```
#### Route with a default
Expand All @@ -53,6 +53,6 @@ An entry that does not match any of the routes is dropped and not processed furt
- type: router
routes:
- output: my_json_parser
expr: '$.format == "json"'
expr: 'body.format == "json"'
default: catchall
```
4 changes: 2 additions & 2 deletions docs/operators/syslog_parser.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ The `syslog_parser` operator parses the string-type field selected by `parse_fro
| --- | --- | --- |
| `id` | `syslog_parser` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `parse_from` | `$body` | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | `$body` | The [field](/docs/types/field.md) to which the value will be parsed. |
| `parse_from` | `body` | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | `body` | The [field](/docs/types/field.md) to which the value will be parsed. |
| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). |
| `protocol` | required | The protocol to parse the syslog messages as. Options are `rfc3164` and `rfc5424`. |
Expand Down
4 changes: 2 additions & 2 deletions docs/operators/uri_parser.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ The `uri_parser` operator parses the string-type field selected by `parse_from`
| --- | --- | --- |
| `id` | `uri_parser` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `parse_from` | `$body` | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | `$body` | The [field](/docs/types/field.md) to which the value will be parsed. |
| `parse_from` | `body` | The [field](/docs/types/field.md) from which the value will be parsed. |
| `parse_to` | `body` | The [field](/docs/types/field.md) to which the value will be parsed. |
| `preserve_to` | | Preserves the unparsed value at the specified [field](/docs/types/field.md). |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](/docs/types/on_error.md). |
| `if` | | An [expression](/docs/types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. |
Expand Down
4 changes: 2 additions & 2 deletions docs/pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,9 @@ pipeline:
# Route based on log type
- type: router
routes:
- expr: '$body startsWith "ERROR"'
- expr: 'body startsWith "ERROR"'
output: error_parser
- expr: '$body startsWith "INFO"'
- expr: 'body startsWith "INFO"'
output: info_parser
# Parse logs with format one
Expand Down
8 changes: 4 additions & 4 deletions docs/types/expression.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ being processed.
For reference documentation of the expression language, see [here](https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md).

Available to the expressions are a few special variables:
- `$body` contains the entry's body
- `$attributes` contains the entry's attributes
- `$resource` contains the entry's resource
- `$timestamp` contains the entry's timestamp
- `body` contains the entry's body
- `attributes` contains the entry's attributes
- `resource` contains the entry's resource
- `timestamp` contains the entry's timestamp
- `env()` is a function that allows you to read environment variables

## Examples
Expand Down
Loading

0 comments on commit fd659de

Please sign in to comment.