Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[pkg/ottl] Add flatten function (open-telemetry#30455)
Browse files Browse the repository at this point in the history
**Description:**
Adds a `flatten` function that allows flattening maps.

I went with an editor instead of a converter, but I'm open to debate.
Using an editor means that a user can do `flatten(body) where
IsMap(body)` instead of `set(body, Flatten(body)) where IsMap(body).
When using ParseJson you have to do:

```
- merge_maps(cache, ParseJSON(body), "upsert")
- flatten(cache)
```

instead of `merge_maps(cache, Flatten(ParseJSON(body)), "upsert")`.
Ultimately I went with an editor for similar reasons that `merge_maps`
is an editor: chaining too many functions together is messy and updating
maps is very fast with pdata.

The function supports 2 optional parameters, `prefix` and `depth`. Use
`prefix` to add a "namespace" to the values that are being flattened.
Use `depth` to prevent trying to flatten maps that are too deep. See the
function doc for examples.

**Link to tracking Issue:** <Issue number if applicable>

Closes
open-telemetry#29283

**Testing:** <Describe what testing was performed and which tests were
added.>
Added new unit and e2e tests.  Please scrutinize.

**Documentation:** <Describe the documentation added.>
Added function doc.

---------

Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>
2 people authored and cparkins committed Feb 1, 2024
1 parent f2e9e33 commit 7f4f471
Showing 6 changed files with 469 additions and 18 deletions.
27 changes: 27 additions & 0 deletions .chloggen/ottl-flatten-function.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: pkg/ottl

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Add `flatten` function for flattening maps

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [30455]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: []
68 changes: 67 additions & 1 deletion pkg/ottl/e2e/e2e_test.go
Original file line number Diff line number Diff line change
@@ -48,6 +48,57 @@ func Test_e2e_editors(t *testing.T) {
tCtx.GetLogRecord().Attributes().Remove("http.url")
},
},
{
statement: `flatten(attributes)`,
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().Remove("foo")
tCtx.GetLogRecord().Attributes().PutStr("foo.bar", "pass")
tCtx.GetLogRecord().Attributes().PutStr("foo.flags", "pass")
tCtx.GetLogRecord().Attributes().PutStr("foo.slice.0", "val")
tCtx.GetLogRecord().Attributes().PutStr("foo.nested.test", "pass")
},
},
{
statement: `flatten(attributes, "test")`,
want: func(tCtx ottllog.TransformContext) {
m := pcommon.NewMap()
m.PutStr("test.http.method", "get")
m.PutStr("test.http.path", "/health")
m.PutStr("test.http.url", "http://localhost/health")
m.PutStr("test.flags", "A|B|C")
m.PutStr("test.total.string", "123456789")
m.PutStr("test.foo.bar", "pass")
m.PutStr("test.foo.flags", "pass")
m.PutStr("test.foo.bar", "pass")
m.PutStr("test.foo.flags", "pass")
m.PutStr("test.foo.slice.0", "val")
m.PutStr("test.foo.nested.test", "pass")
m.CopyTo(tCtx.GetLogRecord().Attributes())
},
},
{
statement: `flatten(attributes, depth=0)`,
want: func(tCtx ottllog.TransformContext) {},
},
{
statement: `flatten(attributes, depth=1)`,
want: func(tCtx ottllog.TransformContext) {
m := pcommon.NewMap()
m.PutStr("http.method", "get")
m.PutStr("http.path", "/health")
m.PutStr("http.url", "http://localhost/health")
m.PutStr("flags", "A|B|C")
m.PutStr("total.string", "123456789")
m.PutStr("foo.bar", "pass")
m.PutStr("foo.flags", "pass")
m.PutStr("foo.bar", "pass")
m.PutStr("foo.flags", "pass")
m.PutStr("foo.slice.0", "val")
m2 := m.PutEmptyMap("foo.nested")
m2.PutStr("test", "pass")
m.CopyTo(tCtx.GetLogRecord().Attributes())
},
},
{
statement: `keep_keys(attributes, ["flags", "total.string"])`,
want: func(tCtx ottllog.TransformContext) {
@@ -75,6 +126,11 @@ func Test_e2e_editors(t *testing.T) {
statement: `merge_maps(attributes, attributes["foo"], "insert")`,
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("bar", "pass")
s := tCtx.GetLogRecord().Attributes().PutEmptySlice("slice")
v := s.AppendEmpty()
v.SetStr("val")
m2 := tCtx.GetLogRecord().Attributes().PutEmptyMap("nested")
m2.PutStr("test", "pass")
},
},
{
@@ -88,6 +144,11 @@ func Test_e2e_editors(t *testing.T) {
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("bar", "pass")
tCtx.GetLogRecord().Attributes().PutStr("flags", "pass")
s := tCtx.GetLogRecord().Attributes().PutEmptySlice("slice")
v := s.AppendEmpty()
v.SetStr("val")
m2 := tCtx.GetLogRecord().Attributes().PutEmptyMap("nested")
m2.PutStr("test", "pass")
},
},
{
@@ -330,7 +391,7 @@ func Test_e2e_converters(t *testing.T) {
{
statement: `set(attributes["test"], Len(attributes["foo"]))`,
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutInt("test", 2)
tCtx.GetLogRecord().Attributes().PutInt("test", 4)
},
},
{
@@ -588,6 +649,11 @@ func constructLogTransformContext() ottllog.TransformContext {
m := logRecord.Attributes().PutEmptyMap("foo")
m.PutStr("bar", "pass")
m.PutStr("flags", "pass")
s := m.PutEmptySlice("slice")
v := s.AppendEmpty()
v.SetStr("val")
m2 := m.PutEmptyMap("nested")
m2.PutStr("test", "pass")

return ottllog.NewTransformContext(logRecord, scope, resource)
}
121 changes: 104 additions & 17 deletions pkg/ottl/ottlfuncs/README.md
Original file line number Diff line number Diff line change
@@ -47,6 +47,7 @@ Available Editors:

- [delete_key](#delete_key)
- [delete_matching_keys](#delete_matching_keys)
- [flatten](#flatten)
- [keep_keys](#keep_keys)
- [limit](#limit)
- [merge_maps](#merge_maps)
@@ -61,9 +62,9 @@ Available Editors:

`delete_key(target, key)`

The `delete_key` function removes a key from a `pdata.Map`
The `delete_key` function removes a key from a `pcommon.Map`

`target` is a path expression to a `pdata.Map` type field. `key` is a string that is a key in the map.
`target` is a path expression to a `pcommon.Map` type field. `key` is a string that is a key in the map.

The key will be deleted from the map.

@@ -78,9 +79,9 @@ Examples:

`delete_matching_keys(target, pattern)`

The `delete_matching_keys` function removes all keys from a `pdata.Map` that match a regex pattern.
The `delete_matching_keys` function removes all keys from a `pcommon.Map` that match a regex pattern.

`target` is a path expression to a `pdata.Map` type field. `pattern` is a regex string.
`target` is a path expression to a `pcommon.Map` type field. `pattern` is a regex string.

All keys that match the pattern will be deleted from the map.

@@ -91,13 +92,99 @@ Examples:

- `delete_matching_keys(resource.attributes, "(?i).*password.*")`

### flatten

`flatten(target, Optional[prefix], Optional[depth])`

The `flatten` function flattens a `pcommon.Map` by moving items from nested maps to the root.

`target` is a path expression to a `pcommon.Map` type field. `prefix` is an optional string. `depth` is an optional non-negative int.

For example, the following map

```json
{
"name": "test",
"address": {
"street": "first",
"house": 1234
},
"occupants": ["user 1", "user 2"]
}
```

is converted to

```json
{
"name": "test",
"address.street": "first",
"address.house": 1234,
"occupants.0": "user 1",
"occupants.1": "user 2"
}
```

If `prefix` is supplied, it will be appended to the start of the new keys. This can help you namespace the changes. For example, if in the above example a `prefix` of `app` was configured, the result would be

```json
{
"app.name": "test",
"app.address.street": "first",
"app.address.house": 1234,
"app.occupants.0": "user 1",
"app.occupants.1": "user 2"
}
```

If `depth` is supplied, the function will only flatten nested maps up to that depth. For example, if a `depth` of `2` was configured, the following map

```json
{
"0": {
"1": {
"2": {
"3": {
"4": "value"
}
}
}
}
}
```

the result would be

```json
{
"0.1.2": {
"3": {
"4": "value"
}
}
}
```

A `depth` of `0` means that no flattening will occur.

Examples:

- `flatten(attributes)`


- `flatten(cache, "k8s", 4)`


- `flatten(body, depth=2)`


### keep_keys

`keep_keys(target, keys[])`

The `keep_keys` function removes all keys from the `pdata.Map` that do not match one of the supplied keys.
The `keep_keys` function removes all keys from the `pcommon.Map` that do not match one of the supplied keys.

`target` is a path expression to a `pdata.Map` type field. `keys` is a slice of one or more strings.
`target` is a path expression to a `pcommon.Map` type field. `keys` is a slice of one or more strings.

The map will be changed to only contain the keys specified by the list of strings.

@@ -112,9 +199,9 @@ Examples:

`limit(target, limit, priority_keys[])`

The `limit` function reduces the number of elements in a `pdata.Map` to be no greater than the limit.
The `limit` function reduces the number of elements in a `pcommon.Map` to be no greater than the limit.

`target` is a path expression to a `pdata.Map` type field. `limit` is a non-negative integer.
`target` is a path expression to a `pcommon.Map` type field. `limit` is a non-negative integer.
`priority_keys` is a list of strings of attribute keys that won't be dropped during limiting.

The number of priority keys must be less than the supplied `limit`.
@@ -137,7 +224,7 @@ Examples:

The `merge_maps` function merges the source map into the target map using the supplied strategy to handle conflicts.

`target` is a `pdata.Map` type field. `source` is a `pdata.Map` type field. `strategy` is a string that must be one of `insert`, `update`, or `upsert`.
`target` is a `pcommon.Map` type field. `source` is a `pcommon.Map` type field. `strategy` is a string that must be one of `insert`, `update`, or `upsert`.

If strategy is:

@@ -159,11 +246,11 @@ Examples:

### replace_all_matches

`replace_all_matches(target, pattern, replacement, function)`
`replace_all_matches(target, pattern, replacement, Optional[function])`

The `replace_all_matches` function replaces any matching string value with the replacement string.

`target` is a path expression to a `pdata.Map` type field. `pattern` is a string following [filepath.Match syntax](https://pkg.go.dev/path/filepath#Match). `replacement` is either a path expression to a string telemetry field or a literal string. `function` is an optional argument that can take in any Converter that accepts a (`replacement`) string and returns a string. An example is a hash function that replaces any matching string with the hash value of `replacement`.
`target` is a path expression to a `pcommon.Map` type field. `pattern` is a string following [filepath.Match syntax](https://pkg.go.dev/path/filepath#Match). `replacement` is either a path expression to a string telemetry field or a literal string. `function` is an optional argument that can take in any Converter that accepts a (`replacement`) string and returns a string. An example is a hash function that replaces any matching string with the hash value of `replacement`.

Each string value in `target` that matches `pattern` will get replaced with `replacement`. Non-string values are ignored.

@@ -177,11 +264,11 @@ Examples:

### replace_all_patterns

`replace_all_patterns(target, mode, regex, replacement, function)`
`replace_all_patterns(target, mode, regex, replacement, Optional[function])`

The `replace_all_patterns` function replaces any segments in a string value or key that match the regex pattern with the replacement string.

`target` is a path expression to a `pdata.Map` type field. `regex` is a regex string indicating a segment to replace. `replacement` is either a path expression to a string telemetry field or a literal string.
`target` is a path expression to a `pcommon.Map` type field. `regex` is a regex string indicating a segment to replace. `replacement` is either a path expression to a string telemetry field or a literal string.

`mode` determines whether the match and replace will occur on the map's value or key. Valid values are `key` and `value`.

@@ -208,7 +295,7 @@ If using OTTL outside of collector configuration, `$` should not be escaped and

### replace_match

`replace_match(target, pattern, replacement, function)`
`replace_match(target, pattern, replacement, Optional[function])`

The `replace_match` function allows replacing entire strings if they match a glob pattern.

@@ -228,7 +315,7 @@ Examples:

### replace_pattern

`replace_pattern(target, regex, replacement, function)`
`replace_pattern(target, regex, replacement, Optional[function])`

The `replace_pattern` function allows replacing all string sections that match a regex pattern with a new value.

@@ -281,9 +368,9 @@ Examples:

`truncate_all(target, limit)`

The `truncate_all` function truncates all string values in a `pdata.Map` so that none are longer than the limit.
The `truncate_all` function truncates all string values in a `pcommon.Map` so that none are longer than the limit.

`target` is a path expression to a `pdata.Map` type field. `limit` is a non-negative integer.
`target` is a path expression to a `pcommon.Map` type field. `limit` is a non-negative integer.

The map will be mutated such that the number of characters in all string values is less than or equal to the limit. Non-string values are ignored.

Loading

0 comments on commit 7f4f471

Please sign in to comment.