-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(parsers.avro): Support multiple modes for union handling #13945
feat(parsers.avro): Support multiple modes for union handling #13945
Conversation
Once #13939 is merged I'll rebase. |
b6e4141
to
d5d8d59
Compare
I have a question that comes from @afausti -- what should the default behavior be if you explicitly mark an incoming field as both a tag and a field? The current code "does both" which ... I don't know what that does in practice. The right answer is "don't do that," of course...but should I catch it when building the parser and warn and leave it a tag but not add it as a field? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for tackling this @athornton! Just two small comments...
d5d8d59
to
a39d896
Compare
Review suggestions applied, and rebased onto current master. |
Co-authored-by: Sven Rebhan <[email protected]>
ddd0bf3
to
8c87ade
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the update @athornton!
Required for all PRs
resolves #12970
Added a config item to allow user to specify "flatten" (current behavior), "nullable" (what the issue requestor wanted, and what we at Rubin Observatory need internally), and "any" (just take what comes in and put it in the named field) as modes by which Avro can handle unions.
The common use case of "nullable"--a field that can sometimes be null/missing, and if so, we get a measurement without that metric--is the major enhancement.
Either "nullable" with multiple non-null types or "any" would make InfluxDB unhappy with the received data (the entire measurement will be dropped when InfluxDB sees a metric of a type other than whatever it first saw (for a given shard)), but if getting data into InfluxDB isn't your goal, those modes could be useful.