-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Proposal for easier cache tuning. #9903
Comments
We could also use a counting bloom filter to count how many times a key was accessed in the last N minutes, this would allow things like "store in cache for 5 minutes, unless its been accessed more than once in the last 1hour", i.e. we could quickly drop items that only ever get accessed once, but retain nodes that have been accessed multiple times (though this likely means that we end up adding the value to the cache, dropping it, then re-adding it to the cache, at which point it stays there for a while). The reason to do this is to try and get a good balance between evicting stuff quickly that never gets used again, vs retaining items that get periodically queried at a period greater than the expiry. |
I think that some sort of cross-cache memory management is a good first step here. We can add fancier things like "how many times was it used in the last hour" later. (Likewise: I always think it would be good to take into account the amount of recalculation we save by having something in the cache, which depends on what it is that is cached. But again, I'd add that later.) |
Both of these are now done, so this issue is unblocked. My suggestion of how to implement this is to have three knobs that can be twiddled (but with better names):
Then it should just be a case of updating the looping call at: synapse/synapse/util/caches/lrucache.py Line 109 in d66d68f
8dc1836 refactors the current jemalloc stuff to expose the necessary APIs to get the currently allocated memory (by calling Note: we should only do this when using jemalloc. |
With #12701 having landed, can we consider this issue complete? |
Let's go with yes! |
We want caches to be more easily tunable (without requiring expert knowledge), and ideally for the defaults to provide a decent experience out of the box. The flip side is that we want to ensure that we don't use unnecessary amounts of memory, as a) we want to keep memory usage low in general, and b) more memory increases the GC resource usage.
What I'm currently thinking is being able to set thresholds based on the total memory usage of caches and/or last access time. That way we can have levels such as:
The idea being that we want to stay under the memory limits without evicting recently used stuff from the caches. It'll probably take some experimentation to come up with the right values.
This could efficiently be implemented by having the LruCache also insert nodes into a global linked list, with last access times tracked per node. To use memory limits we'd also need to either a) enable the memory tracking in #9881, or b) use jemalloc to get an accurate measure of how much allocated memory is actually being used.
We may also want to take into account how often a node has been accessed. E.g. we may wish to expire nodes that have only ever been accessed once more aggressively than for those that get accessed more periodically (the LRU nature of the cache takes this into account somewhat already).
The text was updated successfully, but these errors were encountered: