Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kafka: 2.4.0 support - add support for new data types added in 2.4 #9813

Merged
merged 10 commits into from
Feb 10, 2020

Conversation

adamkotwasinski
Copy link
Contributor

@adamkotwasinski adamkotwasinski commented Jan 23, 2020

Description:
Provide codec support for data types added in 2.4.0.

In detail, we Kafka 2.4 added some new things in protocol:

  • variable-length uint32
  • new compact strings / byte-arrays / generic arrays that keep length using this varint, instead of fixed-len types
  • predefined "tagged_fields" structure (that's very similar to generic array, but doesn't do the +1 length)

This PR should not have any impact on existing codebase, just the volume of changes (if everything was posted at once) is IMHO too large to comfortably review.
Relates to #2852

These deserializers are working with 2.4 request types (they are not part of this PR as I'm still working on that and want to make another PR with them).

Risk Level: Low
Testing: Automated unit tests; embedded integration test (ensures compatibility with 2.3 client); manual tests with 2.3 client
Docs Changes: N/A
Release Notes: N/A

@adamkotwasinski adamkotwasinski changed the title WIP kafka: update client and server dependencies to 2.4.0 WIP: Upgrade Kafka dependencies to 2.4.0 Jan 23, 2020
…egers, compact strings, compact bytes, compact arrays

Signed-off-by: Adam Kotwasinski <[email protected]>
@adamkotwasinski adamkotwasinski changed the title WIP: Upgrade Kafka dependencies to 2.4.0 kafka: 2.4.0 support - add support for new data types added in 2.4 Feb 6, 2020
@junr03
Copy link
Member

junr03 commented Feb 7, 2020

@adamkotwasinski are you waiting on a review here, or still iterating?

@adamkotwasinski
Copy link
Contributor Author

@junr03 still iterating, didn't expect that build failure!
will assign the reviewer when I'm done

…variable-length uint deserializer

Signed-off-by: Adam Kotwasinski <[email protected]>
Signed-off-by: Adam Kotwasinski <[email protected]>
@adamkotwasinski
Copy link
Contributor Author

/wait-any

@adamkotwasinski adamkotwasinski changed the title kafka: 2.4.0 support - add support for new data types added in 2.4 WIP kafka: 2.4.0 support - add support for new data types added in 2.4 Feb 7, 2020
@adamkotwasinski adamkotwasinski changed the title WIP kafka: 2.4.0 support - add support for new data types added in 2.4 kafka: 2.4.0 support - add support for new data types added in 2.4 Feb 7, 2020
@adamkotwasinski
Copy link
Contributor Author

This PR should not have any impact on existing Kafka protocol coverage, as the new classes are not yet referenced from (generated) requests.h / responses.h protocol files.

It can be noticed that there's some duplication here, as we are handling very similar data types all the time (e.g. (nullable) (compact) byte array). I will want to revisit this place to refactor it later, to avoid PR noise.

/assign @mattklein123
/assign @junr03

Copy link
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from a skim, thanks. (Note that as we get more people using this in production we are going to need to find more people to do low level code reviews.)

@moderation
Copy link
Contributor

moderation commented Feb 12, 2020

@adamkotwasinski
Copy link
Contributor Author

@moderation yes - when I'm done with #10000

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants