-
Notifications
You must be signed in to change notification settings - Fork 207
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: memory leak when publishing messages #406
Conversation
If publish threads are marked as daemonic, the leak seemingly disappears.
Server 500 when running samples, re-starting. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Found a true cause of the issue, it's a bug in CPython.
The additional linked comment explains which `CPython` issue is the root cause of this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix typo
The fix is good, but I wonder if we'd run into issues by setting these threads as daemons.
So, this would mean that scripts that may have created a publisher client, published a bunch of messages, and had only exited when the commit threads (non-daemon) finished running - these scripts could now terminate earlier. However, this may not be a problem, because if users want to guarantee publishes happened, they need to wait for the publish futures to finish with success. Anyway, just wondering what your thoughts were here. |
Peter and I spoke offline: possible premature script termination b/c the threads are no daemons doesn't break a customer guarantee, because as long as the user hasn't received a completed future, they can't assume the publish succeeded. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Since it's already late into the workweek, I'll cut a release containing this on Monday. |
Fixes #395.
If publish threads are marked as daemonic, the leak seemingly disappears.
How to verify
Set up a topic and a subscription, install
memoryprofiler
. Add the@profile
decorator to themain()
function from the sample code and run it (with and without the fix):After each run, plot the memory usage graph:
The graphs should look similar to the ones posted here.
PR checklist