Skip to content

Commit

Permalink
filter_nest: Add docs for multiple wildcards (fluent#83)
Browse files Browse the repository at this point in the history 
This adds documentation and examples for

 - Multiple wildcards
 - `Add_prefix` option
 - `Remove_prefix` option

Signed-off-by: Michiel Kalkman <[email protected]>
  • Loading branch information
@michiel @edsiper
michiel authored and edsiper committed Sep 19, 2018
1 parent 01cf817 commit 60fda9d
Showing 1 changed file with 54 additions and 20 deletions.
74 changes: 54 additions & 20 deletions filter/nest.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,23 +61,21 @@ _Example \(output\)_

The plugin supports the following configuration parameters:

| Key | Value Format | Operation | Description |
| :--- | :--- | :--- | :--- |
| Operation | ENUM \[`nest`\|`lift`\] | | Select the operation `nest` or `lift` |
| Wildcard | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard |
| Nest\_under | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key |
| Nested\_under | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key |
| Prefix\_with | FIELD STRING | `lift` | Prefix lifted keys with this string |
| Key | Value Format | Operation | Description |
|----------------|-----------------------|-------------|-------------------|
| Operation | ENUM [`nest` or `lift`] | &nbsp; | Select the operation `nest` or `lift` |
| Wildcard | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard |
| Nest\_under | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key |
| Nested\_under | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key |
| Add\_prefix | FIELD STRING | ANY | Prefix affected keys with this string |
| Remove\_prefix | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string |

## Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the [Memory Usage Input Plugin](https://github.com/fluent/fluent-bit-docs/tree/ad9d80e5490bd5d79c86955c5689db1cb4cf89db/input/mem.html), which outputs the following \(example\),

```text
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[1] memory: [1488543157, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[2] memory: [1488543158, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
[3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
```

## Example \#1 - nest
Expand All @@ -88,8 +86,8 @@ In order to start filtering records, you can run the filter from the command lin
The following command will load the _mem_ plugin. Then the _nest_ filter will match the wildcard rule to the keys and nest the keys matching `Mem.*` under the new key `NEST`.

```text
$ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=NEST' -m '*' -o stdout
```
$ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
```

### Configuration File
Expand All @@ -108,7 +106,8 @@ $ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildc
Match *
Operation nest
Wildcard Mem.*
Nest_under NEST
Nest_under Memstats
Remove_prefix Mem.
```

### Result
Expand All @@ -117,10 +116,45 @@ The output of both the command line and configuration invocations should be iden

```text
[2018/04/06 01:35:13] [ info] [engine] started
[0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "NEST"=>{"Mem.total"=>4050908, "Mem.used"=>714984, "Mem.free"=>3335924}}]
[1] mem.local: [1522978515.000715559, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "NEST"=>{"Mem.total"=>4050908, "Mem.used"=>714984, "Mem.free"=>3335924}}]
[2] mem.local: [1522978516.001057673, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "NEST"=>{"Mem.total"=>4050908, "Mem.used"=>714984, "Mem.free"=>3335924}}]
[3] mem.local: [1522978517.000795638, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "NEST"=>{"Mem.total"=>4050908, "Mem.used"=>714984, "Mem.free"=>3335924}}]
[0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]
```

## Example #1 - nest and lift undo

This example nests all `Mem.*` and `Swap,*` items under the `Stats` key and then reverses these actions with a `lift` operation. The output appears unchanged.

### Configuration File
```python
[INPUT]
Name mem
Tag mem.local

[OUTPUT]
Name stdout
Match *

[FILTER]
Name nest
Match *
Operation nest
Wildcard Mem.*
Wildcard Swap.*
Nest_under Stats
Add_prefix NESTED

[FILTER]
Name nest
Match *
Operation lift
Nested_under Stats
Remove_prefix NESTED
```

### Result

```
[2018/06/21 17:42:37] [ info] [engine] started (pid=17285)
[0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]
```

## Example \#2 - nest 3 levels deep
Expand Down Expand Up @@ -223,21 +257,21 @@ This example starts with the 3-level deep nesting of _Example 2_ and applies the
Match *
Operation lift
Nested_under LAYER3
Prefix_with Lifted3_
Add_prefix Lifted3_

[FILTER]
Name nest
Match *
Operation lift
Nested_under Lifted3_LAYER2
Prefix_with Lifted3_Lifted2_
Add_prefix Lifted3_Lifted2_

[FILTER]
Name nest
Match *
Operation lift
Nested_under Lifted3_Lifted2_LAYER1
Prefix_with Lifted3_Lifted2_Lifted1_
Add_prefix Lifted3_Lifted2_Lifted1_
```

### Result
Expand Down

0 comments on commit 60fda9d

Please sign in to comment.