Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat (parser.json_v2): Support defining field/tag tables within an object table #9449

Merged
merged 14 commits into from
Oct 4, 2021

Conversation

sspaink
Copy link
Contributor

@sspaink sspaink commented Jun 29, 2021

resolves: #9381

To allow users to gather specific elements from an array, this pull request allows users to define field/tag tables within an object table. This will allow a user to gather a field/tag while still maintaining the original relationship within the JSON.

Examples of this new config feature have been added under the testdata folder in the json_v2 directory.

Here is one of the examples to give an idea of the new added feature:

[[inputs.file]]
    files = ["./testdata/subfieldtag_in_object/input.json"]
    data_format = "json_v2"
    [[inputs.file.json_v2]]
        [[inputs.file.json_v2.object]]
            path = "root.station"
            [[inputs.file.json_v2.object.field]]
                path = "#.etd.0.estimate.0.minutes"
                type = "int"
            [[inputs.file.json_v2.object.tag]]
                path = "#.abbr"
                rename = "from_station"
            [[inputs.file.json_v2.object.tag]]
                path = "#.etd.0.abbreviation"
                rename = "to_station"
            [[inputs.file.json_v2.object.tag]]
                path = "#.etd.0.estimate.0.direction"

@sspaink sspaink force-pushed the jsonv2subfieldtags branch from d835326 to 61e6bf4 Compare July 21, 2021 18:08
@sspaink sspaink changed the title Support defining field/tag tables within an object table feat: Support defining field/tag tables within an object table Aug 2, 2021
@sspaink sspaink changed the title feat: Support defining field/tag tables within an object table blah: Support defining field/tag tables within an object table Aug 2, 2021
@sspaink sspaink changed the title blah: Support defining field/tag tables within an object table feat: Support defining field/tag tables within an object table Aug 2, 2021
@sspaink sspaink marked this pull request as ready for review September 2, 2021 22:52
@sspaink sspaink requested a review from sjwang90 September 2, 2021 22:52
@sspaink sspaink changed the title feat: Support defining field/tag tables within an object table feat (parser.json_v2): Support defining field/tag tables within an object table Sep 2, 2021
@sspaink sspaink added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Sep 2, 2021
@sjwang90
Copy link
Contributor

sjwang90 commented Sep 9, 2021

Looks good! Ran this config.

[[inputs.http]]
    urls = ["http://api.bart.gov/api/etd.aspx?cmd=etd&orig=POWL&key=MW9S-E7SL-26DU-VV8V&json=y"
            ]
    data_format = "json_v2"
    tagexclude = ["url", "host"]
    [[inputs.http.json_v2]]
        [[inputs.http.json_v2.object]]
            path = "root.station"
            [[inputs.http.json_v2.object.tag]]
                path = "#.abbr"
                rename = "from_station"
            # destination = ANTC
            [[inputs.http.json_v2.object.tag]]
                path = "#.etd.0.abbreviation"
                rename = "to_station"
            [[inputs.http.json_v2.object.tag]]
                path = "#.etd.0.estimate.0.direction"
            [[inputs.http.json_v2.object.field]]
                path = "#.etd.0.estimate.0.minutes"
                type = "int"
            # destination = BERY
            [[inputs.http.json_v2.object.tag]]
                path = "#.etd.1.abbreviation"
                rename = "to_station"
            [[inputs.http.json_v2.object.tag]]
                path = "#.etd.1.estimate.0.direction"
            [[inputs.http.json_v2.object.field]]
                path = "#.etd.1.estimate.0.minutes"
                type = "int"

with this output

> http,etd_estimate_direction=North,from_station=POWL,to_station=ANTC etd_estimate_minutes=9i 1631214122000000000
> http,etd_estimate_direction=North,from_station=POWL,to_station=BERY etd_estimate_minutes=6i 1631214122000000000

Copy link
Member

@helenosheaa helenosheaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! It's the first time I've properly looked at the new parser, the documentation is awesome.

Few of nits / questions around certain comments. I'll test out the build as well and leave any feedback if I notice anything, but good job.

plugins/parsers/json_v2/parser.go Outdated Show resolved Hide resolved
plugins/parsers/json_v2/parser.go Outdated Show resolved Hide resolved
plugins/parsers/json_v2/parser.go Outdated Show resolved Hide resolved
plugins/parsers/json_v2/parser.go Outdated Show resolved Hide resolved
@sspaink
Copy link
Contributor Author

sspaink commented Sep 16, 2021

Thanks @helenosheaa for the review, I've addressed the changes and if you could try out the build artifacts that would be great.

Copy link
Member

@helenosheaa helenosheaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested, looks good!

@sspaink sspaink force-pushed the jsonv2subfieldtags branch from bea665c to 7b91a30 Compare October 4, 2021 17:58
@sspaink sspaink merged commit df5c19c into influxdata:master Oct 4, 2021
@sspaink sspaink deleted the jsonv2subfieldtags branch October 4, 2021 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update json_v2 parser to allow excluding/including specific array elements
3 participants