Cache memory management

[js-choi]

How should cache memory management work? Is there a way to get garbage collection by default? (Using WeakMaps for the caches would be ideal…except that WeakMaps do not support primitives as keys.)

Should we just use Maps and make the developer manage the cache memory themselves? (Related: I am planning to propose built-in LRUMaps and LFUMaps.)

There is also the compositeKeys proposal (#1).

[michaelficarra]

@js-choi You could map Object (except when == null) over the arguments to box them, no?

[js-choi]

@michaelficarra: If we used a WeakMap cache and Object(x), where x is a primitive, then how would we efficiently look up results in the cache, since Object(x) !== Object(x)?

[michaelficarra]

Oh yep, I forgot that the boxed primitives will be unique for the purposes of WeakMap keys.

[zloirock]

WeakMap with compositeKey as keys will allow properly store primitives and GC for objects and any arguments lists. However, sure, make sense to allow custom cach strategy for users.

[NickGard]

I think that managing cache is an implementation detail best left to the browser which would have the best information about how much memory is available and what the "hot paths" are for cache lookups (e.g. to switch between LFU and LRU).

The fact that users of memoization libraries have to deal with setting and managing cache is a leaky abstraction. If the cache could grow and shrink as the system determines the best use of memory, it would free up the developer to only worry about whether or not the function to be memoized is pure. The browser could determine that some memoized functions actually run fast without memoization or have mostly cache-misses (which greatly grows the cache) and opt them out of memoization. Or in a memory-constrained environment, like IOT devices, Function.memoize could be a simple identity function, returning the same function (unmemoized) as is passed in, rather than allowing the user to allocate memory in an unoptimized way. Regardless, the developer does not care if the function is truly memoized or not, as long as it is run as fast as possible in the given system.

// heavyFn has a sizeable cache, but it greatly improves repeated calls
// and has mostly cache hits, so it is truly memoized
const heavyFn = Function.memoize(heavy_fn);

// oneValueFn has a small cache, only one key-value, but the repeated
// calls are faster, so it is truly memoized
const oneValueFn = Function.memoize(only_called_with_same_arguments_fn);

// fastFn actually runs almost as fast as a cache lookup, so it is eventually
// opted out of true memoization and ran every call
const fastFn = Function.memoize(fast_fn);

// mostlyCacheMisses keeps growing its cache size with few repeated hits,
// so it is opted out of true memoization and ran every call
const mostlyCacheMisses = Function(lots_of_varied_calls_fn);

// rareCall is called a few times on page load, but then is never called again.
// After some time, the browser opts it out of true caching and frees up its memory
const rareCall = Function.memoize(on_page_load);

On a technical level, WeakRefs are widely supported now, so a Map could hold both primitives and WeakRef-wrapped objects as keys and values. This polyfill uses a trie of WeakRefs and primitives as a cache. The trie is pruned on idle cycles (with requestIdleCallback) so that empty WeakRef nodes and their descendants are removed from the cache.

[theScottyJam]

I think that managing cache is an implementation detail best left to the browser which would have the best information about how much memory is available and what the "hot paths" are for cache lookups (e.g. to switch between LFU and LRU).

While I think this could be a great default behavior, there is something to be said for managing your own cache to prevent entries from going stale. For example, if you're caching an async getUserGroups(userId) function, you might want a cache entry to never sit in cache for longer than 5 minutes, because you'd like it if you could make updates to a user's groups, and have the changes propagated around the system within 5 minutes. Perhaps, you'd even like to manually eject things from the cache when certain custom events happen, like, maybe you have a somewhat reliable way of getting notified when a user's groups gets updated, and you'd like to auto-eject the old content from the cache when this happens. And, you may also wish to simply auto-eject entries from the cache if their returned promise rejects.

Still, all of those examples are related to you manually choosing to eject something from the cache. It may be nice to have your custom ejection behaviors layered on top of the system behavior, where the system will likewise auto-eject items from the cache when memory starts to get full. So, perhaps we still don't need to give people the power to go so low-level as to actually choose between LRU and LFU, dunno.