-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High-level API for consuming messages with an effect #22
Changes from 3 commits
6b9595e
e2a3274
bbeff21
5262740
5c622ef
04454e8
b5de7b1
ff6b66b
0c77d68
ee14c80
8698ff0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -6,6 +6,7 @@ import zio.blocking.Blocking | |||||
import zio._ | ||||||
import zio.stream._ | ||||||
import zio.clock.Clock | ||||||
import zio.console._ | ||||||
|
||||||
import scala.collection.JavaConverters._ | ||||||
|
||||||
|
@@ -47,4 +48,54 @@ object Consumer { | |||||
) | ||||||
runloop <- Runloop(deps) | ||||||
} yield new Consumer(wrapper, settings, runloop) | ||||||
|
||||||
/** | ||||||
* Execute an effect for each record and commit the offset after processing | ||||||
* | ||||||
* This method is the easiest way of processing messages on a Kafka topic. | ||||||
* | ||||||
* Messages on a single partition are processed sequentially, while the processing of | ||||||
* multiple partitions happens in parallel. | ||||||
* | ||||||
* Messages are processed with 'at least once' consistency: it is not guaranteed that every message | ||||||
* that is processed by the effect has a corresponding offset commit before stream termination. | ||||||
* | ||||||
* Offsets are batched and committed after execution of the effect. | ||||||
* | ||||||
* The effect should must absorb any failures. Failures should be handled by retries or ignoring the | ||||||
* error, which will result in the Kafka message being skipped. | ||||||
* | ||||||
* Usage example: | ||||||
* | ||||||
* @param settings Settings for creating a [[Consumer]] | ||||||
* @param subscription Topic subscription parameters | ||||||
* @param f Function that returns the effect to execute for each message. It is passed the key and value | ||||||
* @tparam K Type of keys (an implicit [[Serde]] should be in scope) | ||||||
* @tparam V Type of values (an implicit [[Serde]] should be in scope) | ||||||
* @return Effect that completes with a unit value only when interrupted. May fail when the [[Consumer]] fails. | ||||||
*/ | ||||||
def consumeM[R, K: Serde, V: Serde]( | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We don't have a version of this that takes a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not happy about the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Well, the consuming function could require any |
||||||
subscription: Subscription, | ||||||
settings: ConsumerSettings | ||||||
)(f: (K, V) => ZIO[R, Nothing, Unit]): ZIO[R with Clock with Blocking with Console, Throwable, Unit] = | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
ZStream | ||||||
.managed(Consumer.make[K, V](settings)) | ||||||
.flatMap { consumer => | ||||||
ZStream | ||||||
.fromEffect(consumer.subscribe(subscription)) | ||||||
.flatMap { _ => | ||||||
consumer.partitioned | ||||||
.flatMapPar(Int.MaxValue, outputBuffer = settings.perPartitionChunkPrefetch) { | ||||||
case (partition @ _, partitionStream) => | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
partitionStream.mapM { | ||||||
case CommittableRecord(record, offset) => | ||||||
f(record.key(), record.value()).as(offset) | ||||||
}.flattenChunks | ||||||
} | ||||||
} | ||||||
} | ||||||
.aggregate(ZSink.foldLeft[Offset, OffsetBatch](OffsetBatch.empty)(_ merge _)) | ||||||
.mapM(_.commit) | ||||||
.tap(_ => putStrLn(s"Commit done")) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No printlns :-) |
||||||
.run(ZSink.drain) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
An example would be nice, yeah!