Drop KPL / Sink a chunk rather than individual event #630
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
istreeter
reviewed
Jun 17, 2022
modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Enrich.scala
Outdated
Show resolved
Hide resolved
modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Enrich.scala
Outdated
Show resolved
Hide resolved
benjben
force-pushed
the
fix/kinesis_throughput
branch
from
ea75276 to
572adf1
Compare
benjben
force-pushed
the
fix/kinesis_throughput
branch
from
5a64e5e to
3bd4de8
Compare
benjben
changed the title
istreeter
reviewed
Jun 23, 2022
modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Enrich.scala
Outdated
Show resolved
Hide resolved
modules/kinesis/src/main/scala/com/snowplowanalytics/snowplow/enrich/kinesis/Sink.scala
Show resolved
Hide resolved
istreeter
approved these changes
Jun 24, 2022
benjben
force-pushed
the
fix/kinesis_throughput
branch
from
bc60ca4 to
36ef929
Compare
benjben
force-pushed
the
fix/kinesis_throughput
branch
from
36ef929 to
aa0d741
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
enrich-kinesis is the replacement for stream-enrich-kinesis.
common-fs2 (also used by
enrich-pubsub) contains all the logic, apart from the sourcing and sinking.Changes
This PR aims at changing the way we send records to Kinesis (more details on #633), to use the same mechanism as
stream-enrichand manually build PutRecords requests instead of having KPL doing it for us.The main consequence is that now the sinking functions (contained in the Environment) are defined for a chunk whereas before it was defined for individual records.
What's important to have in mind is that a
PutRecordsrequest can be successful but contain failures that need to be retried.So there are 2 types of retries:
PutRecordsrequest fails