Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Record accessor limitation (Backport of #1015). #1228

Merged
merged 1 commit into from
Oct 9, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,23 @@ Fluent Bit v1.x.x
{"date":1599862267.483692,"log":"message 4","labels":{"color":"blue"}}
```

### Limitations of record_accessor templating

The Fluent Bit record_accessor library has a limitation in the characters that can separate template variables- only dots and commas (`.` and `,`) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.

The following would be invalid templates because the two template variables are not separated by commas or dots:

- `$TaskID-$ECSContainerName`
- `$TaskID/$ECSContainerName`
- `$TaskID_$ECSContainerName`
- `$TaskIDfooo$ECSContainerName`

However, the following are valid:
- `$TaskID.$ECSContainerName`
- `$TaskID.ecs_resource.$ECSContainerName`
- `$TaskID.fooo.$ECSContainerName`

And the following are valid since they only contain one template variable with nothing after it:
- `fooo$TaskID`
- `fooo____$TaskID`
- `fooo/bar$TaskID`
23 changes: 2 additions & 21 deletions pipeline/filters/ecs-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,8 @@ The output log would be similar to:
}
```

Notice that the template variables in the value for the `resource` key are separated by dot characters. Please see the section below about limitations in which characters can be used to separate template variables.
Notice that the template variables in the value for the `resource` key are separated by dot characters, only dots and commas
(`.` and `,`) can come after a template variable. For more information, please check the [Record accessor limitation's section](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md#limitations-of-record_accessor-templating).

#### Example 3: Attach cluster metadata to non-container logs

Expand Down Expand Up @@ -148,23 +149,3 @@ This examples shows a use case for the `Cluster_Metadata_Only` option- attaching
Format json_lines
```

### Limitations of record_accessor templating

Notice in example 2, that the template values are separated by dot characters. This is important; the Fluent Bit record_accessor library has a limitation in the characters that can separate template variables- only dots and commas (`.` and `,`) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.

The following would be invalid templates because the two template variables are not separated by commas or dots:

- `$TaskID-$ECSContainerName`
- `$TaskID/$ECSContainerName`
- `$TaskID_$ECSContainerName`
- `$TaskIDfooo$ECSContainerName`

However, the following are valid:
- `$TaskID.$ECSContainerName`
- `$TaskID.ecs_resource.$ECSContainerName`
- `$TaskID.fooo.$ECSContainerName`

And the following are valid since they only contain one template variable with nothing after it:
- `fooo$TaskID`
- `fooo____$TaskID`
- `fooo/bar$TaskID`