Add authentication for max lag parameter and fix IT #763
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
sfc-gh-rcheng
commented
Dec 7, 2023
•
sfc-gh-rcheng
commented
- Validates that the given parameter for SNOWPIPE_STREAMING_MAX_CLIENT_LAG is a long
- Fixes a max lag IT
sfc-gh-rcheng
requested review from
sfc-gh-japatel,
sfc-gh-tzhang,
sfc-gh-tjones and
a team
as code owners
sfc-gh-rcheng
changed the title
src/main/java/com/snowflake/kafka/connector/internal/streaming/StreamingUtils.java
Show resolved
Hide resolved
sfc-gh-tjones
approved these changes
Dec 7, 2023
sfc-gh-rcheng
added a commit
that referenced
this pull request
Dec 7, 2023
sfc-gh-rcheng
added a commit
that referenced
this pull request
Dec 8, 2023
EduardHantig
pushed a commit
to streamkap-com/snowflake-kafka-connector
that referenced
this pull request
Feb 1, 2024
sudeshwasnik
added a commit
to confluentinc/snowflake-kafka-connector
that referenced
this pull request
Feb 16, 2024
* [external contributor] Support config providers in validation by @swehner (snowflakedb#753) Co-authored-by: Stefan Wehner <[email protected]> * v2.1.2 release change in master (snowflakedb#757) git cherry-pick d4d0e87 * NO-SNOW: Expose the Ingest SDK MAX_CLIENT_LAG configuration in KC (snowflakedb#758) * Add authentication for max lag parameter and fix IT (snowflakedb#763) * SNOW-989387: Fix for java.lang.NoClassDefFoundError (snowflakedb#774) java.lang.NoClassDefFoundError: Could not initialize class net.snowflake.client.jdbc.internal.apache.arrow.memory.RootAllocator occurs when the JDBC driver tries to process the result of the offset migration query. This is a known long-standing issue in the JDBC driver so a workaround is introduced with this fix. * [SNOW-1015644] Update ingest sdk to 2.0.5 (snowflakedb#778) * SNOW-943288, SNOW-938379: Fix two schema evolution cases that could cause non-exactly once delivery (snowflakedb#775) Fix two exactly once delivery behavior issues with Snowpipe Streaming: - When there're gaps between offsets due to records being put into the DLQ or NULL records being skipped, the current logic doesn't work after a channel reopening event (e.g. schema evolution) since it expects continuous offsets in order to guarantee exactly once delivery, causing ingestion to stop. - When a flush is triggered by size or rowcount threshold, it's possible that only partial rows in the buffer are flushed and with a channel reopening event (e.g. schema evolution), currently the leftover rows in the buffer are still being considered which is causing the offset to be advanced and skip some offsets. The fix in this PR makes sure we won't flush any leftover rows in the buffer, they will all be skipped and discarded. The next batch should start from the expected offset after the offset reset as part of the channel reopening event. * resolves issue with generating and building java libary (snowflakedb#780) * [SNOW-88848] Update readme with Snowflake documentation (snowflakedb#781) * Update version to 2.2.0 for release (snowflakedb#779) * Downgrade JDBC to 3.13.30 (snowflakedb#783) * SNOW-922795 Added third-party-licenses to .zip distribution (snowflakedb#784) * PROD-39429 Add PR template (snowflakedb#786) --------- Co-authored-by: revi cheng <[email protected]> Co-authored-by: Stefan Wehner <[email protected]> Co-authored-by: Toby Zhang <[email protected]> Co-authored-by: Waleed Fateem <[email protected]> Co-authored-by: Greg Jachimko <[email protected]> Co-authored-by: Michał Bobowski <[email protected]> Co-authored-by: Jay Patel <[email protected]>
sudeshwasnik
pushed a commit
to confluentinc/snowflake-kafka-connector
that referenced
this pull request
Feb 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.