fix(recipe): No more memory leak once TreeCache was closed #524

tonyseek · 2018-10-01T22:32:51Z

The current implementation of TreeCache recipe could not be closed completely because some unexpected references:

The registered znode event watchers hold the reference of instancemethod which are bound with TreeNode instances
The background threads in connection handler keep last reference when they are idle

This pull request fixes those memory leak problems, and verifies that by using objgraph in test.

We could guarantee all closed TreeCache will be eventually released now. (not immediately since the cycle reference between TreeNode and TreeCache must be resolved by mark-swap GC of Python)

tonyseek · 2018-10-02T11:33:09Z

😸 It is ready for review now.

jeffwidman

Thanks for continuing to maintain this recipe. And I appreciate the detailed commit messages, very helpful.

When I see stuff like cb = None # GC: release last reference before idle, I wonder why not just do del cb?

Using del makes it explicit that you're clearing the memory, so that the annotation # GC: release last reference before idle becomes superfluous.

Given the attention to detail in your PR's, I suspect I'm missing something... Is there some reason that it's better to set it to None?

jeffwidman · 2018-10-06T07:46:55Z

kazoo/recipe/cache.py

@@ -57,6 +58,11 @@ def start(self):
        After a cache started, all changes of subtree will be synchronized
        from the ZooKeeper server. Events will be fired for those activity.

+        Don't forget to call :meth:`close` if it was started and you don't
+        need it anymore, or you will leak the memory of cached nodes, even you


typo: even you --> even if you (at least, that's what I think you mean)

jeffwidman · 2018-10-06T07:47:39Z

kazoo/recipe/cache.py

+        Don't forget to call :meth:`close` if it was started and you don't
+        need it anymore, or you will leak the memory of cached nodes, even you
+        give up all reference to the :class:`TreeCache` instance. (There are
+        so many callback has been registered)


callbacks should be plural: callback has been --> callbacks that have been

tonyseek · 2018-10-15T04:57:11Z

Hi @jeffwidman, Thanks for you review. The commits have been revised now.

tonyseek · 2018-10-15T05:29:07Z

kazoo/handlers/gevent.py

+                        log.warning("Exception in worker greenlet")
+                        log.exception(exc)
+                    finally:
+                        del func  # release before possible idle


Given the attention to detail in your PR's, I suspect I'm missing something... Is there some reason that it's better to set it to None?

I had set the func to None for preventing from NameError if an exception was raised earlier than func was assigned. But it is confusing and del func is more explicit that it, I agree with you.

I use the nested try-except like the implementation of SequentialThreadingHandler now. The NameError will never occur here.

tonyseek · 2018-10-26T12:49:52Z

I will keep the branch up-to-date for latest master. The new pushed commits are just for rebasing.

StephenSorriaux · 2018-11-18T12:33:37Z

Hi @tonyseek, thanks for your PR. I will try to give it a review during the week.

StephenSorriaux · 2018-11-20T09:22:54Z

kazoo/tests/test_cache.py

+            while not self.client.handler.completion_queue.empty():
+                self.client.handler.sleep_func(0.1)
+        else:
+            # Let hub of gevent/eventlet be in a short-time loop


Should we check here that the queue is empty?

It seems that switching to loop hub via sleep_func once is stable enough for waiting the async results to be finished when we are using eventlet and gevent. They will switch back only when IO event happened. But this conclusion doesn't not play well with threading handlers because the native threads have unpredictable schedule opportunity. That's why I wrote the special branch to wait completion_queue for the threading handler.

It is possible to check completion_queue for eventlet too, but not for gevent. The gevent uses native AsyncResult of itself instead of the completion_queue.

Modified in b9c7948 and d07a9b0

StephenSorriaux · 2018-11-20T09:35:22Z

kazoo/tests/test_cache.py

+                continue
+            except Exception:
+                logger.warning(
+                    'The handler module %s is imported but could not be used',


Did you notice cases where this can happen?

In my cases it only happen when the gevent installation is broken. I should delete it and make it fail fast.

Fixed in b9c7948 and d07a9b0

StephenSorriaux · 2018-11-20T12:57:27Z

kazoo/tests/test_cache.py

 from kazoo.exceptions import KazooException
 from kazoo.recipe.cache import TreeCache, TreeNode, TreeEvent


-class KazooTreeCacheTests(KazooTestCase):
+logger = logging.getLogger(__name__)


Seems like you can remove this object now since it is not used anymore

StephenSorriaux · 2018-11-20T12:57:42Z

kazoo/tests/test_cache.py

@@ -1,15 +1,48 @@
+import gc
+import importlib
+import logging


This import is now useless

StephenSorriaux · 2018-11-20T12:59:20Z

Thanks again for this great PR, it is impressive. I just added 2 small comments... I think we will be ready to merge this PR asap.

tonyseek · 2018-11-21T06:29:54Z

@StephenSorriaux My pleasure. And thank you for your review.

The unused logger has been removed in 4a26c99 now.

StephenSorriaux · 2018-11-21T10:35:40Z

@tonyseek merged to master. Thanks again for your hard work.

jeffwidman · 2018-12-12T12:47:41Z

Great job pushing this through @StephenSorriaux / @tonyseek

* python-zk#404 * python-zk#216 * python-zk#262 * python-zk#524

tonyseek force-pushed the tree-cache-memory-leak-closing branch 3 times, most recently from 57b3a22 to 4115dd4 Compare October 2, 2018 11:11

jeffwidman reviewed Oct 6, 2018

View reviewed changes

tonyseek force-pushed the tree-cache-memory-leak-closing branch 2 times, most recently from e73d157 to c2aadd9 Compare October 15, 2018 04:55

tonyseek commented Oct 15, 2018

View reviewed changes

tonyseek force-pushed the tree-cache-memory-leak-closing branch 2 times, most recently from ab3f49f to 3426dfb Compare October 26, 2018 12:45

tonyseek force-pushed the tree-cache-memory-leak-closing branch from 3426dfb to 1cdeafc Compare November 13, 2018 09:16

tonyseek closed this Nov 13, 2018

tonyseek reopened this Nov 13, 2018

tonyseek closed this Nov 13, 2018

tonyseek reopened this Nov 13, 2018

tonyseek added 6 commits November 17, 2018 02:23

fix(core): Fix memory leak on idle handler

b1068da

fix(recipe): Fix memory leak on closed TreeCache

9213e75

fix(recipe): Test TreeCache's memory with objgraph

b818be4

fix(recipe): Fix incorrect listen logger in TreeCache

d283aeb

fix(recipe): Let TreeCache start in safe way

dcb8550

docs: Suggest people close unused TreeCache

a601b55

tonyseek force-pushed the tree-cache-memory-leak-closing branch from 1cdeafc to a4c566c Compare November 16, 2018 18:23

StephenSorriaux reviewed Nov 20, 2018

View reviewed changes

tonyseek force-pushed the tree-cache-memory-leak-closing branch 3 times, most recently from b9c7948 to d07a9b0 Compare November 20, 2018 11:48

StephenSorriaux reviewed Nov 20, 2018

View reviewed changes

refactor: Test various handlers on TreeCache

4a26c99

tonyseek force-pushed the tree-cache-memory-leak-closing branch from d07a9b0 to 4a26c99 Compare November 21, 2018 06:19

StephenSorriaux approved these changes Nov 21, 2018

View reviewed changes

StephenSorriaux merged commit c48f273 into python-zk:master Nov 21, 2018

tonyseek deleted the tree-cache-memory-leak-closing branch November 21, 2018 11:04

mozillazg added a commit to huskar-org/kazoo that referenced this pull request Sep 29, 2019

Backport python-zk#404 python-zk#216 python-zk#262 python-zk#524

a1a5e9f

* python-zk#404 * python-zk#216 * python-zk#262 * python-zk#524

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(recipe): No more memory leak once TreeCache was closed #524

fix(recipe): No more memory leak once TreeCache was closed #524

tonyseek commented Oct 1, 2018 •

edited

Loading

tonyseek commented Oct 2, 2018

jeffwidman left a comment •

edited

Loading

jeffwidman Oct 6, 2018

jeffwidman Oct 6, 2018

tonyseek commented Oct 15, 2018

tonyseek Oct 15, 2018

tonyseek commented Oct 26, 2018

StephenSorriaux commented Nov 18, 2018

StephenSorriaux Nov 20, 2018

tonyseek Nov 20, 2018

tonyseek Nov 20, 2018 •

edited

Loading

StephenSorriaux Nov 20, 2018

tonyseek Nov 20, 2018

tonyseek Nov 20, 2018 •

edited

Loading

StephenSorriaux Nov 20, 2018

StephenSorriaux Nov 20, 2018

StephenSorriaux commented Nov 20, 2018

tonyseek commented Nov 21, 2018

StephenSorriaux commented Nov 21, 2018

jeffwidman commented Dec 12, 2018

fix(recipe): No more memory leak once TreeCache was closed #524

fix(recipe): No more memory leak once TreeCache was closed #524

Conversation

tonyseek commented Oct 1, 2018 • edited Loading

tonyseek commented Oct 2, 2018

jeffwidman left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonyseek commented Oct 15, 2018

Choose a reason for hiding this comment

tonyseek commented Oct 26, 2018

StephenSorriaux commented Nov 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonyseek Nov 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonyseek Nov 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StephenSorriaux commented Nov 20, 2018

tonyseek commented Nov 21, 2018

StephenSorriaux commented Nov 21, 2018

jeffwidman commented Dec 12, 2018

tonyseek commented Oct 1, 2018 •

edited

Loading

jeffwidman left a comment •

edited

Loading

tonyseek Nov 20, 2018 •

edited

Loading

tonyseek Nov 20, 2018 •

edited

Loading