Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: nest: bringing nest doc to basic style guidelines #1476

Merged
merged 3 commits into from
Oct 18, 2024
Merged

Docs: nest: bringing nest doc to basic style guidelines #1476

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c2ad933
Docs: nest: bringing nest doc to basic style guidelines
esmerel Oct 17, 2024
e7319f5
Docs: nest: fixing missing tick
esmerel Oct 17, 2024
9a0d31b
Apply suggestions from code review
esmerel Oct 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 74 additions & 50 deletions pipeline/filters/nest.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
# Nest

The _Nest Filter_ plugin allows you to operate on or with nested data. Its modes of operation are
The _Nest Filter_ plugin lets you operate on or with nested data. Its modes of operation are:

* `nest` - Take a set of records and place them in a map
* `lift` - Take a map by key and lift its records up
- `nest` - Take a set of records and place them in a map.
- `lift` - Take a map by key and lift its records up.

## Example usage \(nest\)
## Example usage for `nest`

As an example using JSON notation, to nest keys matching the `Wildcard` value `Key*` under a new key `NestKey` the transformation becomes,
As an example using JSON notation, to nest keys matching the `Wildcard` value `Key*`
under a new key `NestKey` the transformation becomes:

_Example \(input\)_
Input:

```text
{
Expand All @@ -19,7 +20,7 @@ _Example \(input\)_
}
```

_Example \(output\)_
Output:

```text
{
Expand All @@ -31,11 +32,12 @@ _Example \(output\)_
}
```

## Example usage \(lift\)
## Example usage for `lift`

As an example using JSON notation, to lift keys nested under the `Nested_under` value `NestKey*` the transformation becomes,
As an example using JSON notation, to lift keys nested under the `Nested_under` value
`NestKey*` the transformation becomes:

_Example \(input\)_
Input:

```text
{
Expand All @@ -47,7 +49,7 @@ _Example \(input\)_
}
```

_Example \(output\)_
Output:

```text
{
Expand All @@ -61,40 +63,47 @@ _Example \(output\)_

The plugin supports the following configuration parameters:

| Key | Value Format | Operation | Description |
| Key | Value format | Operation | Description |
| :--- | :--- | :--- | :--- |
| Operation | ENUM \[`nest` or `lift`\] | | Select the operation `nest` or `lift` |
| Wildcard | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard |
| Nest\_under | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key |
| Nested\_under | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key |
| Add\_prefix | FIELD STRING | ANY | Prefix affected keys with this string |
| Remove\_prefix | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string |
| `Operation` | ENUM [`nest` or `lift`] | | Select the operation `nest` or `lift` |
| `Wildcard` | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard |
| `Nest_under` | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key |
| `Nested_under` | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key |
| `Add_prefix` | FIELD STRING | ANY | Prefix affected keys with this string |
| `Remove_prefix` | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string |

## Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the [Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the following \(example\),
To start filtering records, run the filter from the command line or through the
configuration file. The following example invokes the
[Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the
following:

```text
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
```

## Example \#1 - nest
## Example 1 - nest

### Command Line

> Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
Using command line mode requires quotes to parse the wildcard properly. The use
of a configuration file is recommended.
Comment on lines +90 to +91
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not saying we should use a note here, but while we should be removing instances of "Please note" or "Note", we do also have a note alert box element that we can use wherever it can provide value. Again, this may or may not be a place for the note alert box, but I just wanted to mention its existence for potential use.


The following command will load the _mem_ plugin. Then the _nest_ filter will match the wildcard rule to the keys and nest the keys matching `Mem.*` under the new key `NEST`.
The following command loads the _mem_ plugin. Then the _nest_ filter matches the
wildcard rule to the keys and nests the keys matching `Mem.*` under the new key
`NEST`.

```text
$ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
```shell copy
bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
```

### Configuration File

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python

```python copy
[INPUT]
Name mem
Tag mem.local
Expand All @@ -111,10 +120,12 @@ $ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildc
Nest_under Memstats
Remove_prefix Mem.
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml

```yaml copy
pipeline:
inputs:
- name: mem
Expand All @@ -130,6 +141,7 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}

Expand All @@ -142,15 +154,17 @@ The output of both the command line and configuration invocations should be iden
[0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]
```

## Example \#2 - nest and lift undo
## Example 2 - nest and lift undo

This example nests all `Mem.*` and `Swap,*` items under the `Stats` key and then reverses these actions with a `lift` operation. The output appears unchanged.
This example nests all `Mem.*` and `Swap.*` items under the `Stats` key and then
reverses these actions with a `lift` operation. The output appears unchanged.

### Configuration File
### Example 2 Configuration File

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python

```python copy
[INPUT]
Name mem
Tag mem.local
Expand All @@ -175,10 +189,11 @@ This example nests all `Mem.*` and `Swap,*` items under the `Stats` key and then
Nested_under Stats
Remove_prefix NESTED
```
{% endtab %}

{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml

```yaml copy
pipeline:
inputs:
- name: mem
Expand All @@ -201,6 +216,7 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}

Expand All @@ -211,15 +227,17 @@ pipeline:
[0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]
```

## Example \#3 - nest 3 levels deep
## Example 3 - nest 3 levels deep

This example takes the keys starting with `Mem.*` and nests them under `LAYER1`, which itself is then nested under `LAYER2`, which is nested under `LAYER3`.
This example takes the keys starting with `Mem.*` and nests them under `LAYER1`,
which is then nested under `LAYER2`, which is nested under `LAYER3`.

### Configuration File
### Example 3 Configuration File

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python

```python copy
[INPUT]
Name mem
Tag mem.local
Expand Down Expand Up @@ -249,10 +267,11 @@ This example takes the keys starting with `Mem.*` and nests them under `LAYER1`,
Wildcard LAYER2*
Nest_under LAYER3
```
{% endtab %}

{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml

```yaml copy
pipeline:
inputs:
- name: mem
Expand All @@ -277,6 +296,7 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}

Expand All @@ -302,15 +322,19 @@ pipeline:
}
```

## Example \#4 - multiple nest and lift filters with prefix
## Example 4 - multiple nest and lift filters with prefix

This example starts with the 3-level deep nesting of _Example 2_ and applies the `lift` filter three times to reverse the operations. The end result is that all records are at the top level, without nesting, again. One prefix is added for each level that is lifted.
This example uses the 3-level deep nesting of _Example 2_ and applies the
`lift` filter three times to reverse the operations. The end result is that all
records are at the top level, without nesting, again. One prefix is added for each
level that's lifted.

### Configuration file

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python

```python copy
[INPUT]
Name mem
Tag mem.local
Expand Down Expand Up @@ -361,10 +385,12 @@ This example starts with the 3-level deep nesting of _Example 2_ and applies the
Nested_under Lifted3_Lifted2_LAYER1
Add_prefix Lifted3_Lifted2_Lifted1_
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml

```yaml copy
pipeline:
inputs:
- name: mem
Expand Down Expand Up @@ -404,23 +430,21 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}


### Result

```text
[0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}]


{
"Swap.total"=>1046524,
"Swap.used"=>0,
"Swap.free"=>1046524,
"Lifted3_Lifted2_Lifted1_Mem.total"=>4050908,
"Lifted3_Lifted2_Lifted1_Mem.used"=>1253912,
"Swap.total"=>1046524,
"Swap.used"=>0,
"Swap.free"=>1046524,
"Lifted3_Lifted2_Lifted1_Mem.total"=>4050908,
"Lifted3_Lifted2_Lifted1_Mem.used"=>1253912,
"Lifted3_Lifted2_Lifted1_Mem.free"=>2796996
}
```