feat(sink): add date, time, interval, bytea and jsonb data types for PG/MySQL sink of stream_chunk/json payload #9957

StrikeW · 2023-05-23T07:48:08Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Add date, time, interval, bytea and jsonb data types for postgres/mysql sink in stream_chunk/json payload
Since interval and jsonb are not standard jdbc types, we use PGInterval and PGObject from postgres driver to fulfill the prepare statement
An e2e sink test was added for both mysql and postgres, for types date, time, timestamp, jsonb and bytea

I tested it with the jdbc sink in the local environment.

part of #9918, #9919

Checklist For Contributors

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
I have demonstrated that backward compatibility is not broken by breaking changes and created issues to track deprecated features to be removed in the future. (Please refer to the issue)
All checks passed in ./risedev check (or alias, ./risedev c)

Checklist For Reviewers

I have requested macro/micro-benchmarks as this PR can affect performance substantially, and the results are shown.

Documentation

My PR DOES NOT contain user-facing changes.

Click here for Documentation

Types of user-facing changes

Release note

codecov · 2023-05-23T08:14:55Z

Codecov Report

Merging #9957 (2035ae8) into main (2b04ed5) will decrease coverage by 0.06%.
The diff coverage is 21.00%.

@@            Coverage Diff             @@
##             main    #9957      +/-   ##
==========================================
- Coverage   71.03%   70.98%   -0.06%     
==========================================
  Files        1231     1231              
  Lines      210684   210883     +199     
==========================================
+ Hits       149655   149690      +35     
- Misses      61029    61193     +164

Flag	Coverage Δ
rust	`70.98% <21.00%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/java_binding/src/lib.rs	`0.00% <0.00%> (ø)`
src/connector/src/sink/kafka.rs	`38.42% <14.28%> (-0.94%)`	⬇️
src/connector/src/sink/remote.rs	`57.07% <83.33%> (+0.43%)`	⬆️
src/connector/src/sink/mod.rs	`73.28% <85.71%> (+1.96%)`	⬆️

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

neverchanje · 2023-05-24T02:06:21Z

Good work @StrikeW ! But I want to know if pginterval and pgobject is only used for pgsink. What is the behavior of sinking intervals and jsonbs to a mysql?

StrikeW · 2023-05-24T02:35:45Z

Good work @StrikeW ! But I want to know if pginterval and pgobject is only used for pgsink. What is the behavior of sinking intervals and jsonbs to a mysql?

You are right. We should check whether the downstream is PG since those classes are specific to the PG driver.

wenym1

Rest LGTM

java/connector-node/risingwave-sink-jdbc/src/main/java/com/risingwave/connector/JDBCSink.java

wenym1 · 2023-05-24T04:53:23Z

src/java_binding/src/lib.rs

+        let string_value = env.new_string(value)?;
+        let (class_ref, constructor) =
+            pointer.as_ref().class_cache.date_ctor.get_or_try_init(|| {
+                let cls = env.find_class("java/sql/Date")?;


Is is possible to create a Data and Time object from a integer timestamp instead of from string?

I think you're worried about the performance issue. The string representation is non-ambiguous while the integer representation may encounter precision issues, for example ms vs us.

I think for Date and Time, their minimal required precision is millisecond, and therefore there is no need to care about the nano second level precision. And I think it should be safe to use the constructor from a timestamp of long type. It takes more time to convert to a string first and then parse the string.

cc @WillyKidd

https://docs.oracle.com/javase/8/docs/api/java/sql/Time.html#Time-long-
https://docs.oracle.com/javase/8/docs/api/java/sql/Date.html#Date-long-

maybe we can do this in the next pr :)

java/connector-node/risingwave-sink-jdbc/src/main/java/com/risingwave/connector/JDBCSink.java

src/java_binding/src/lib.rs

java/connector-node/risingwave-sink-jdbc/src/main/java/com/risingwave/connector/JDBCSink.java

StrikeW · 2023-05-30T13:37:45Z

src/connector/src/sink/mod.rs

+                let value = datum_to_json_object(
+                    &inner_field,
+                    sub_datum_ref,
+                    TimestampHandlingMode::String,


Why you don't use the passing argument timestamp_handling_mode for list and struct? @WillyKidd

…PG/MySQL sink of stream_chunk/json payload (#9957) Co-authored-by: weili <[email protected]> Co-authored-by: WillyKidd <[email protected]>

add date, time data types for jdbc sink(pg)

0b1ffcc

StrikeW requested review from neverchanje, fuyufjh, tabVersion and wenym1 May 23, 2023 07:48

github-actions bot added the type/feature label May 23, 2023

clean

aa2977b

StrikeW force-pushed the siyuan/jdbc-sink-data-types branch from 77dfa61 to aa2977b Compare May 23, 2023 07:49

add support for interval and jsonb

393b9fb

StrikeW changed the title ~~feat(sink): add date, time data types for streamchunk payload~~ feat(sink): add date, time, interval and jsonb data types for streamchunk payload May 23, 2023

StrikeW changed the title ~~feat(sink): add date, time, interval and jsonb data types for streamchunk payload~~ feat(sink): add date, time, interval and jsonb data types for PG sink of streamchunk payload May 23, 2023

StrikeW changed the title ~~feat(sink): add date, time, interval and jsonb data types for PG sink of streamchunk payload~~ feat(sink): add date, time, interval and jsonb data types for PG sink of stream_chunk payload May 23, 2023

StrikeW added the component/connector label May 23, 2023

wenym1 reviewed May 24, 2023

View reviewed changes

WillyKidd mentioned this pull request May 24, 2023

test: add data types testing for mysql sink #9919

Closed

3 tasks

check downstream database type

01b28fb

StrikeW marked this pull request as draft May 24, 2023 10:54

WillyKidd added 4 commits May 25, 2023 14:51

add date, time, jsonb for jdbc sink json format

4c44ab5

fix jsonb for mysql

5b7e046

fix fallthrough in switch

7ca3efb

bytea support for stream_chunk on connector side

7247041

WillyKidd force-pushed the siyuan/jdbc-sink-data-types branch from 0edb940 to 7247041 Compare May 26, 2023 03:56

bytea support for json format

10d2e4a

WillyKidd changed the title ~~feat(sink): add date, time, interval and jsonb data types for PG sink of stream_chunk payload~~ feat(sink): add date, time, interval, bytea and jsonb data types for PG sink of stream_chunk payload May 26, 2023

WillyKidd added 2 commits May 26, 2023 13:45

fix date

4b7b84f

"upgrade" sink e2e test

2725dea

WillyKidd marked this pull request as ready for review May 26, 2023 07:04

WillyKidd added 5 commits May 29, 2023 14:52

fix

6551611

try fix e2e

af21dc1

fix e2e

b6f4352

fix e2e

6fc4d8a

fix e2e

fb2018f

WillyKidd marked this pull request as draft May 29, 2023 09:08

StrikeW mentioned this pull request May 29, 2023

Remote Sink: switch to stream_chunk payload format #10047

Closed

2 tasks

fix

56120de

WillyKidd marked this pull request as ready for review May 29, 2023 09:54

WillyKidd added 2 commits May 29, 2023 18:15

fix

c60353a

revert upsert/append_only.result

b452f4e

fuyufjh approved these changes May 30, 2023

View reviewed changes

wenym1 reviewed May 30, 2023

View reviewed changes

WillyKidd and others added 6 commits May 30, 2023 11:45

fix misc

f84116e

Merge branch 'main' into siyuan/jdbc-sink-data-types

f97c11c

use base64 for bytea in json encoding

dd07a95

"upgrade" jdbc sink test

d176fde

Merge branch 'main' into siyuan/jdbc-sink-data-types

000a208

fix format

009c1cf

StrikeW commented May 30, 2023

View reviewed changes

minor

2035ae8

StrikeW added this pull request to the merge queue May 30, 2023

Merged via the queue into main with commit 5ea2204 May 30, 2023

StrikeW deleted the siyuan/jdbc-sink-data-types branch May 30, 2023 15:42

This was referenced May 31, 2023

Bug(sink): create tidb sink panic #9376

Closed

Tracking(sink datatype mapping): sink type mismatch for jdbc and Debezium #9604

Closed

lmatz pushed a commit that referenced this pull request Jun 1, 2023

feat(sink): add date, time, interval, bytea and jsonb data types for …

f23e99c

…PG/MySQL sink of stream_chunk/json payload (#9957) Co-authored-by: weili <[email protected]> Co-authored-by: WillyKidd <[email protected]>

xiangjinwu mentioned this pull request Nov 9, 2023

Kafka-cdc-sink bytea type dosen't work with JDBC connector #13327

Closed

lmatz mentioned this pull request Nov 24, 2023

for Date and Time, their minimal required precision is millisecond #13637

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sink): add date, time, interval, bytea and jsonb data types for PG/MySQL sink of stream_chunk/json payload #9957

feat(sink): add date, time, interval, bytea and jsonb data types for PG/MySQL sink of stream_chunk/json payload #9957

StrikeW commented May 23, 2023 •

edited by WillyKidd

Loading

codecov bot commented May 23, 2023 •

edited

Loading

neverchanje commented May 24, 2023

StrikeW commented May 24, 2023

wenym1 left a comment

wenym1 May 24, 2023

StrikeW May 24, 2023

wenym1 May 30, 2023

WillyKidd May 30, 2023

StrikeW May 30, 2023

feat(sink): add date, time, interval, bytea and jsonb data types for PG/MySQL sink of stream_chunk/json payload #9957

feat(sink): add date, time, interval, bytea and jsonb data types for PG/MySQL sink of stream_chunk/json payload #9957

Conversation

StrikeW commented May 23, 2023 • edited by WillyKidd Loading

What's changed and what's your intention?

Checklist For Contributors

Checklist For Reviewers

Documentation

Types of user-facing changes

Release note

codecov bot commented May 23, 2023 • edited Loading

Codecov Report

neverchanje commented May 24, 2023

StrikeW commented May 24, 2023

wenym1 left a comment

Choose a reason for hiding this comment

wenym1 May 24, 2023

Choose a reason for hiding this comment

StrikeW May 24, 2023

Choose a reason for hiding this comment

wenym1 May 30, 2023

Choose a reason for hiding this comment

WillyKidd May 30, 2023

Choose a reason for hiding this comment

StrikeW May 30, 2023

Choose a reason for hiding this comment

StrikeW commented May 23, 2023 •

edited by WillyKidd

Loading

codecov bot commented May 23, 2023 •

edited

Loading