Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand fields in decode_json_fields if target is set #32010

Merged
merged 7 commits into from
Jun 21, 2022

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Jun 21, 2022

What does this PR do?

This PR applies the settings in expand_keys to the event even if the target is not empty.

Why is it important?

Previously, if target was set to anything besides "", expand_keys did not work. So you could not expand the keys of the parsed JSON if you wanted to put it under a custom field.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Config:

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  paths:
    - test.log
  processors:
    - decode_json_fields:
      target: "my_target"
      expand_fields: true
      fields: ["message"]
output.console:
  enabled: true

Example input file:

{"my.field": "interesting value"}

Check if the keys are expanded in the published event:

{
    "message": "{\"my.field\": \"interesting value\"}",
    "my_target": {
        "my": {
          "field": "interesting value"
        }
    }
}

Related issues

Closes #31712

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 21, 2022
@mergify mergify bot assigned kvch Jun 21, 2022
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 21, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-06-21T13:13:59.676+0000

  • Duration: 80 min 13 sec

Test stats 🧪

Test Results
Failed 0
Passed 22280
Skipped 1937
Total 24217

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@kvch kvch marked this pull request as ready for review June 21, 2022 12:16
@kvch kvch requested a review from a team as a code owner June 21, 2022 12:16
@kvch kvch requested review from belimawr and fearful-symmetry and removed request for a team June 21, 2022 12:16
@kvch kvch added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jun 21, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 21, 2022
@kvch kvch requested review from cmacknz and removed request for belimawr June 21, 2022 14:35
@kvch kvch added backport-v8.3.0 Automated backport with mergify backport-7.17 Automated backport to the 7.17 branch with mergify labels Jun 21, 2022
@kvch kvch merged commit 7b99a8e into elastic:main Jun 21, 2022
mergify bot pushed a commit that referenced this pull request Jun 21, 2022
## What does this PR do?

This PR applies the settings in `expand_keys` to the event even if the target is not empty.

## Why is it important?

Previously, if target was set to anything besides `""`, `expand_keys` did not work. So you could not expand the keys of the parsed JSON if you wanted to put it under a custom field.

(cherry picked from commit 7b99a8e)

# Conflicts:
#	libbeat/common/jsontransform/expand.go
#	libbeat/processors/actions/decode_json_fields.go
#	libbeat/processors/actions/decode_json_fields_test.go
mergify bot pushed a commit that referenced this pull request Jun 21, 2022
## What does this PR do?

This PR applies the settings in `expand_keys` to the event even if the target is not empty.

## Why is it important?

Previously, if target was set to anything besides `""`, `expand_keys` did not work. So you could not expand the keys of the parsed JSON if you wanted to put it under a custom field.

(cherry picked from commit 7b99a8e)
@kvch kvch removed the backport-7.17 Automated backport to the 7.17 branch with mergify label Jun 21, 2022
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
## What does this PR do?

This PR applies the settings in `expand_keys` to the event even if the target is not empty. 

## Why is it important?

Previously, if target was set to anything besides `""`, `expand_keys` did not work. So you could not expand the keys of the parsed JSON if you wanted to put it under a custom field.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v8.3.0 Automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Expand_keys doesn’t work if target option is used in decode_json_fields beat processor
3 participants