Fix schema_registry_decode processor #1198
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When dealing with logical types, we can't serialise the native struct emitted by goavro directly to JSON, since that will discard schema information that `codec.TextualFromNative` uses to produce the expected JSON. Also, the tip version of goavro is required because the fix for this regression linkedin/goavro#242 was merged, but there isn't a tagged release yet. Fixes redpanda-data#1161.
mihaitodor
force-pushed
the
fix-schema-registry-decode-processor
branch
from
587a07c
to
6e8022b
Compare
|
Yeah I think this is the best we can do for now without making some custom component to walk the structure for custom types. In bloblang we can technically deal with any types for basic value getting, but really we need proper basic values in order for basic methods and arithmetic to work as expected. It's not the end of the world if users make plugins that introduce odd types but our own processors definitely shouldn't as the user would have no way to work out what's happening. |
|
Sounds good, thanks for the quick merge! :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When dealing with AVRO logical types, we can't serialise the native struct emitted by goavro directly to JSON, since that will discard schema information that
codec.TextualFromNativeuses to produce the expected JSON.Also, the tip version of goavro is required because the fix for this regression linkedin/goavro#242 was merged, but there isn't a tagged release yet.
Fixes #1161.
For a bit of extra detail, when converting this JSON to binary AVRO using the schema from the test I added (see #1161 for steps on how to do that):
{"int_time_millis":{"int": 35245000},"long_time_micros":{"long":20192000000000},"long_timestamp_micros":{"long":62135596800000000},"pos_0_33333333":{"bytes":"!"}}and then we skip the
codec.TextualFromNativecall when passing it throughschema_registry_decode, we get the following serialised JSON from Benthos:{"int_time_millis":{"int.time-millis":35245000000000},"long_time_micros":{"long.time-micros":20192000000000000},"long_timestamp_micros":{"long.timestamp-micros":"3939-01-01T00:00:00Z"},"pos_0_33333333":{"bytes.decimal":"33/100"}}which doesn't look like something that users will expect, but not sure. I have very limited experience with AVRO.
On the other hand, calling
codec.TextualFromNativeproduces a JSON string which will then need to be unmarshalled back to a native struct if there are any subsequent processors. This is wasteful, but goavro doesn't seem to have any mechanism for crafting a native struct that is compatible with Benthos. I'm not sure right now what bloblang does under the hood if we callMessage.SetStructured()with a native struct that contains types such astime.Time,time.Duration,math/big.Ratetc. Is that supported?