httpjson: Add toJSON helper function

[andrewkroh]

What does this PR do?

toJSON converts the given structure into a JSON string.

Example usage:

  response.pagination:
    - set:
        target: body.pagingIdentifiers
        value: '[[ toJSON .last_response.body.pagingIdentifiers ]]'
        value_type: json
        fail_on_template_error: true

Why is it important?

I didn't find any way to use the set transform to add an existing object to the output. If you omit the toJSON and reference an object in a template then the output is a stringified version of the map like map[hashes:map[13b97806f9067edfc11359732d8c7d63:0] which is not what you want when constructing a JSON request body.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-08-19T15:05:39.638+0000

• Duration: 81 min 45 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2176
Skipped	166
Total	2342

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[efd6]

It seems to me that this should be made to work without needing this. In the case that top level maps are mapstr.M then this is true, so it would be worth finding out why the map that prompted the fix did not end up being that type.

I'm struggling to find where that could be happening or to make the test fail, e.g. this passes

		{
			name:  "func toJSON",
			value: "[[ .last_response.body.paginationParams.m.m ]]",
			paramCtx: &transformContext{
				firstEvent:   &mapstr.M{},
				lastEvent:    &mapstr.M{},
				lastResponse: newTestResponse(map[string]interface{}{"paginationParams": map[string]interface{}{"m": map[string]interface{}{"m": map[string]interface{}{"id": 1234}}}}, nil, ""),
			},
			paramTr:     transformable{},
			expectedVal: `{"id":1234}`,
		},

[efd6]

This test passes without the toJSON call since the parameter is a mapstr.M whose String method renders to JSON.

[andrewkroh]

Good point. I changed the test such that it will fail without toJSON. The code uses mapstr.M.Clone to convert inner objects to mapstr.M, but it does not look inside of []interface{} for maps. So without toJSON you get

        	Error Trace:	value_tpl_test.go:609
        	Error:      	Not equal: 
        	            	expected: "[{\"id\":1234}]"
        	            	actual  : "[map[id:1234]]"

[andrewkroh]

Interesting, one clue I have is that this was only needed when processing a response body that is an array of objects. If the response body is an object then it was working fine without running the toJSON on the value.

In the case that top level maps are mapstr.M then this is true

So perhaps when the response is an array then the inner objects are left as map[string]interface{} instead of being converted to mapstr.M that has a String() method that returns JSON?

[efd6]

Yes, that explains it. This would be fixed if mapstr.M did work to convert map[string]interface{} elements in a []map[string]interface{}. This can be done unsafely in mapstr.mapFind with zero cost, but ... unsafe. The alternative to fix it would be to fully implement json.Unmarshaler for mapstr.M so that the abstraction is complete.

Alternatively this, but with a failable test.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @andrewkroh? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)
  To fixup this pull request, you need to add the backport labels for the needed
  branches, such as:
• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[andrewkroh]

In decodeAsJSON, if we apply a conversion to the []interface{} that translates the map[string]interface{} values to mapstr.M then things start working. But this has a cascading effect since other parts of the code are not expecting to find mapstr.M.

If I knew all the places where data originates then I would make sure they it were all updated to only produce mapstr.M. I raised and issue to consider making some changes to elastic/elastic-agent-libs#75. But I think I would like to proceed with toJSON since it's simple and does not preclude other solutions.

Conversion hack: httpjson-deep-clone-mapstr.patch.txt