Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

httpjson: Add toJSON helper function #32472

Merged
merged 4 commits into from
Aug 21, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff]
- Extend list of mapped record types in o365 Audit module. {pull}32217[32217]
- Add references for CRI-O configuration in input-container and in our kubernetes manifests {issue}32149[32149] {pull}32151[32151]
- httpjson input: Add `replaceAll` helper function to template context. {pull}32365[32365]
- httpjson input: Add `toJSON` helper function to template context. {pull}32472[32472]
- Optimize grok patterns in system.auth module pipeline. {pull}32360[32360]
- Checkpoint module: add authentication operation outcome enrichment. {issue}32230[32230] {pull}32431[32431]
- add documentation for decode_xml_wineventlog processor field mappings. {pull}32456[32456]
Expand Down
1 change: 1 addition & 0 deletions x-pack/filebeat/docs/inputs/input-httpjson.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,7 @@ Some built-in helper functions are provided to work with the input state inside
- `replaceAll(old, new, s)`: replaces all non-overlapping instances of `old` with `new` in `s`. Example: `[[ replaceAll "some" "my" "some value" ]]` returns `my value`.
- `sprintf`: formats according to a format specifier and returns the resulting string. Refer to https://pkg.go.dev/fmt#Sprintf[the Go docs] for usage. Example: `[[sprintf "%d:%q" 34 "quote this"]]`
- `toInt`: converts a value of any type to an integer when possible. Returns 0 if the conversion fails.
- `toJSON`: converts a value to a JSON string. This can be used with `value_type: json` to create an object from a template. Example: `[[ toJSON .last_response.body.pagingIdentifiers ]]`.
- `urlEncode`: URL encodes the supplied string. Example `[[urlEncode "string1"]]`. Example `[[urlEncode "<string1>"]]` will return `%3Cstring1%3E`.
- `userAgent`: generates the User Agent with optional additional values. If no arguments are provided, it will generate the default User Agent that is added to all requests by default. It is recommended to delete the existing User-Agent header before setting a new one. Example: `[[ userAgent "integration/1.2.3" ]]` would generate `Elastic-Filebeat/8.1.0 (darwin; amd64; 9b893e88cfe109e64638d65c58fd75c2ff695402; 2021-12-15 13:20:00 +0000 UTC; integration_name/1.2.3)`
- `uuid`: returns a random UUID such as `a11e8780-e3e7-46d0-8e76-f66e75acf019`. Example: `[[ uuid ]]`
Expand Down
11 changes: 11 additions & 0 deletions x-pack/filebeat/input/httpjson/value_tpl.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"hash"
Expand Down Expand Up @@ -64,6 +65,7 @@ func (t *valueTpl) Unpack(in string) error {
"hmac": hmacStringHex,
"hmacBase64": hmacStringBase64,
"join": join,
"toJSON": toJSON,
"mul": mul,
"now": now,
"parseDate": parseDate,
Expand Down Expand Up @@ -439,3 +441,12 @@ func urlEncode(value string) string {
func replaceAll(old, new, s string) string {
return strings.ReplaceAll(s, old, new)
}

// toJSON converts the given structure into a JSON string.
func toJSON(i interface{}) (string, error) {
result, err := json.Marshal(i)
if err != nil {
return "", fmt.Errorf("toJSON failed: %w", err)
}
return string(bytes.TrimSpace(result)), nil
}
11 changes: 11 additions & 0 deletions x-pack/filebeat/input/httpjson/value_tpl_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -574,6 +574,17 @@ func TestValueTpl(t *testing.T) {
paramTr: transformable{},
expectedVal: "my value",
},
{
name: "func toJSON",
value: "[[ toJSON .last_response.body.paginationParams ]]",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test passes without the toJSON call since the parameter is a mapstr.M whose String method renders to JSON.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I changed the test such that it will fail without toJSON. The code uses mapstr.M.Clone to convert inner objects to mapstr.M, but it does not look inside of []interface{} for maps. So without toJSON you get

        	Error Trace:	value_tpl_test.go:609
        	Error:      	Not equal: 
        	            	expected: "[{\"id\":1234}]"
        	            	actual  : "[map[id:1234]]"

paramCtx: &transformContext{
firstEvent: &mapstr.M{},
lastEvent: &mapstr.M{},
lastResponse: newTestResponse(mapstr.M{"paginationParams": mapstr.M{"id": 1234}}, nil, ""),
},
paramTr: transformable{},
expectedVal: `{"id":1234}`,
},
}

for _, tc := range cases {
Expand Down