Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue-494] FLIP-107: Expose metadata for the new table source #504

Conversation

thekingofcity
Copy link
Contributor

Change log description

FLIP-107 has added the support for metadata along with the event for both table source and sink.

For table source, we export the EventPointer in the table source. By doing this, we can cover the benifit similar with #180 for DataStream Source.

Purpose of the change

Fixes #494

What the code does

  • Use the new FlinkPravegaDynamicDeserializationSchema that expose the metadata from the EventRead. For now, only EventPointer is supported as the other two (scope adn stream) is not exposed from Pravega.
  • Implement the SupportsReadingMetadata for FlinkPravegaDynamicTableSource.
  • Test the corresponding code.

How to verify it

./gradlew clean build passes.

@codecov
Copy link

codecov bot commented Apr 30, 2021

Codecov Report

Merging #504 (75fa7b9) into master (35c5358) will increase coverage by 0.48%.
The diff coverage is 93.33%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #504      +/-   ##
============================================
+ Coverage     82.48%   82.97%   +0.48%     
- Complexity      487      498      +11     
============================================
  Files            44       45       +1     
  Lines          2284     2326      +42     
  Branches        224      231       +7     
============================================
+ Hits           1884     1930      +46     
+ Misses          236      230       -6     
- Partials        164      166       +2     
Impacted Files Coverage Δ Complexity Δ
...avega/connectors/flink/util/FlinkPravegaUtils.java 72.09% <ø> (ø) 7.00 <0.00> (ø)
.../dynamic/table/FlinkPravegaDynamicTableSource.java 71.95% <92.85%> (+8.79%) 11.00 <9.00> (+4.00)
...able/FlinkPravegaDynamicDeserializationSchema.java 94.11% <94.11%> (ø) 5.00 <5.00> (?)
...nk/dynamic/table/FlinkPravegaDynamicTableSink.java 63.82% <0.00%> (+2.12%) 5.00% <0.00%> (+1.00%)
...nk/serialization/PravegaDeserializationSchema.java 84.61% <0.00%> (+23.07%) 8.00% <0.00%> (+1.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 35c5358...75fa7b9. Read the comment docs.

@crazyzhou crazyzhou self-requested a review May 7, 2021 01:42
@thekingofcity thekingofcity force-pushed the issue-494-FLIP-107-Expose-metadata-for-new-Table-source branch from b9277e4 to 1734661 Compare May 14, 2021 07:16
Signed-off-by: zhongle.wang <[email protected]>
Signed-off-by: zhongle.wang <[email protected]>
Signed-off-by: zhongle.wang <[email protected]>
Signed-off-by: zhongle.wang <[email protected]>
Signed-off-by: zhongle.wang <[email protected]>
Signed-off-by: zhongle.wang <[email protected]>
Signed-off-by: zhongle.wang <[email protected]>
Signed-off-by: zhongle.wang <[email protected]>
Copy link
Contributor

@crazyzhou crazyzhou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@crazyzhou crazyzhou merged commit 63f174c into pravega:master May 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

FLIP-107: Expose metadata for new Table source
2 participants