Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parsers.avro): Support multiple modes for union handling #13945

Merged
merged 9 commits into from
Sep 27, 2023

Conversation

athornton
Copy link
Contributor

@athornton athornton commented Sep 18, 2023

Required for all PRs

resolves #12970

Added a config item to allow user to specify "flatten" (current behavior), "nullable" (what the issue requestor wanted, and what we at Rubin Observatory need internally), and "any" (just take what comes in and put it in the named field) as modes by which Avro can handle unions.

The common use case of "nullable"--a field that can sometimes be null/missing, and if so, we get a measurement without that metric--is the major enhancement.

Either "nullable" with multiple non-null types or "any" would make InfluxDB unhappy with the received data (the entire measurement will be dropped when InfluxDB sees a metric of a type other than whatever it first saw (for a given shard)), but if getting data into InfluxDB isn't your goal, those modes could be useful.

@telegraf-tiger telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Sep 18, 2023
@athornton
Copy link
Contributor Author

Once #13939 is merged I'll rebase.

@athornton
Copy link
Contributor Author

I have a question that comes from @afausti -- what should the default behavior be if you explicitly mark an incoming field as both a tag and a field? The current code "does both" which ... I don't know what that does in practice. The right answer is "don't do that," of course...but should I catch it when building the parser and warn and leave it a tag but not add it as a field?

Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this @athornton! Just two small comments...

plugins/parsers/avro/README.md Outdated Show resolved Hide resolved
plugins/parsers/avro/parser.go Outdated Show resolved Hide resolved
@srebhan srebhan self-assigned this Sep 20, 2023
@srebhan srebhan changed the title feat(parsers/avro): Support multiple modes for union handling feat(parsers.avro): Support multiple modes for union handling Sep 20, 2023
@srebhan srebhan added the plugin/parser 1. Request for new parser plugins 2. Issues/PRs that are related to parser plugins label Sep 20, 2023
@athornton
Copy link
Contributor Author

Review suggestions applied, and rebased onto current master.

Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update @athornton!

@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Sep 26, 2023
@srebhan srebhan assigned powersj and unassigned srebhan Sep 26, 2023
@powersj powersj merged commit cb13577 into influxdata:master Sep 27, 2023
4 checks passed
@github-actions github-actions bot added this to the v1.29.0 milestone Sep 27, 2023
@athornton athornton deleted the features/avro-unions-redux branch September 27, 2023 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/parser 1. Request for new parser plugins 2. Issues/PRs that are related to parser plugins ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Avro processor doesnt support Unions
3 participants