-
Notifications
You must be signed in to change notification settings - Fork 151
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Fix leak in FlatMapPrefix operator. #1622
base: main
Are you sure you want to change the base?
Conversation
val prefix = accumulated.toVector | ||
accumulated.clear() | ||
val prefix = builder.result() | ||
builder = null // free for GC |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@queimadus would you like to give this patch a try?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll give it a try but I'm not sure this will work because prefixAndTail
uses a similar implementation and has the same problem described in #1566.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@queimadus There is a fix for prefixAndTail too #1623
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There do have leaks in the interpreter, eg the logic is still being refs.
I did a try in #1621, which needs changing more code to clean up the logics.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My original code before I simplified it with what looked to be the MRE, was more of a:
def myLogic(prefix: Seq[ByteString]): Flow[ByteString, ByteString, NotUsed] =
Flow[ByteString]
val s = Source
.repeat(())
.map(_ => ByteString('a' * 400000))
.take(1000000)
.prefixAndTail(50000)
.flatMapConcat { case (prefix, tail) =>
Source(prefix).concatLazy(tail).via(myLogic(prefix))
}
Source.empty
.concatAllLazy(List.tabulate(30000)(_ => s): _*)
.runWith(Sink.ignore).onComplete(println(_))
Where I would need to use the prefix to generate the remaining flow logic and then emit the prefix back again. It still runs out of heap in this case, but it may be a different problem?
Nonetheless, I think this change makes sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
prefixAndTail
fix is in another PR, could you verify that too?
@queimadus #1623 this one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For reference, this would be the flatMapPrefix
equivalent that wouldn't work with Concat
s but work with flatMapConcat
(the commented code):
val s = Source
.repeat(())
.map(_ => ByteString('a' * 400000))
.take(1000000)
.flatMapPrefix(50000) { prefix =>
Flow[ByteString].prepend(Source(prefix))
}
Source.empty
.concatAllLazy(List.tabulate(30000)(_ => s): _*)
.runWith(Sink.ignore).onComplete(println(_))
// Source
// .repeat(s)
// .take(30000)
// .flatMapConcat(x => x)
// .runWith(Sink.ignore)
// .onComplete(println(_))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the concat should be reimplemented to reduce memory usage somehow.
extracted from #1621
Motivation:
In #1566 , @queimadus points out a leak
Modification:
Result:
leak fixed.