Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Producer sink #35

Merged
merged 5 commits into from
Sep 29, 2019
Merged

Producer sink #35

merged 5 commits into from
Sep 29, 2019

Conversation

svroonland
Copy link
Collaborator

@svroonland svroonland commented Sep 28, 2019

Implements #34

Heavily inspired by Alpakka Kafka of course.

@iravid
Copy link
Member

iravid commented Sep 28, 2019

There are two optimal ways to drive a producer from a stream:

  • Use aggregate/aggregateWithin, as you've hinted in the comment, and produce chunks to the producer. This amortizes the cost of waiting for the producer to flush. In code: .aggregate(groupingSink).mapM(produceChunk)
  • Use mapMPar, and produce individual records. The producer transparently buffers records so this is equivalent in a sense to the previous bullet. Possibly less efficient. In code: .mapMPar(parallelism)(produce)

Neither of these are possible with a pure Sink-based API. The problem is that we don't have any other way to offer users a stream "segment" from our library code.

For now, I suggest that we instead add additional documentation with recommendations on how to efficiently produce data.

@svroonland
Copy link
Collaborator Author

@iravid Thanks for your feedback.If I understand correctly, then all we can offer is the two sinks like in the updated PR. Is that according to your original ideas?

@iravid
Copy link
Member

iravid commented Sep 28, 2019

Yes, this is the based we can do right now. However, I would delete the one that produces individual records, because it's going to be very inefficient. Let's just offer the one that produces chunks for now.

@iravid iravid merged commit 2e3afdd into zio:master Sep 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants