[RFC] Proposal for Adding Cache in App Config

[tianleh]

Introduction

In open source 2.13.0 release, we add the support of dynamic application configuration. The feature stores the configuration in a new OpenSearch index and allows dynamic read and write. We onboard the first use case of CSP rules to the configuration. However, as the team evaluates it to support more use cases (e.g workspace #6300), there are concerns about its performance when large traffic routes to the underlying storage. The API under concern is getEntityConfig which is for getting the configuration value given an entity name. To address the concern, we are proposing to add cache in dynamic application configuration.

Challenges

We will evaluate the technical approaches from several challenges.

The first challenge is whether we will allow custom cache technique. OSS OSD could have a default cache technique and exposes registration of other cache techniques. This could give other OSD offerings the flexibility to choose their cache technique. As a contrast, we could fix a cache technique in OSS OSD and then all other offerings will inherit it.

The second challenge is choosing between distributed cache and local cache. In a multi-node cluster, a local cache in a node may be updated when the same node receives an update request while local cache in other nodes may keep returning the old value until the cache expiry. The distributed cache has a mechanism to sync data across the network but requires extra complexity to build.

The third challenge is the call pattern. Since application configurations are infrequent to change, the number of reads will be much larger than the number of writes.

The fourth challenge is between new dependencies and existing dependencies. New dependencies may introduce new vulnerabilities into our code base. If the existing dependency can meet the business requirement, we will prefer it.

Proposed Approach

We propose to use the existing cache library lru-cache to add a cache layer before querying the configuration index. It is a in-memory local cache that deletes the least-recently-used items. The key is the entity name of string type and the value is the configuration value of string type.

We plan to initialize the cache with the following parameters.

max: 100
maxAge: 5 * 60 * 1000

The parameter max indicates the maximum size of the cache. The parameter maxAge indicates the time to live in ms.

For getEntityConfig function, we will first attempt to get its cached value. If the value exists, we will use it. Otherwise, we will query the index to get the value. When updateEntityConfig function is called, we will update the value in the cache. When deleteEntityConfig function is called, we will remove the value from the cache.

Below is the list of functions provided by the cache which we plan to call.

   /**
     * Will update the "recently used"-ness of the key. They do what you think.
     * `maxAge` is optional and overrides the cache `maxAge` option if provided.
     *
     * If the key is not found, will return `undefined`.
     */
    get(key: K): V | undefined;



   /**
     * Will update the "recently used"-ness of the key. They do what you think.
     * `maxAge` is optional and overrides the cache `maxAge` option if provided.
     */
    set(key: K, value: V, maxAge?: number): boolean;



   /**
     * Deletes a key out of the cache.
     */
    del(key: K): void;

The cache will be initialized inside the constructor of the plugin applicationConfig. Whenever a OpenSearchConfigurationClient object is created, the cache will be passed into it. It will stay in memory until the OSD server stops and will not persist during a server restart.

We plan to directly use this cache library and do not expose registration of other cache clients into OSD.

Alternative Approaches

Approach 1 In-memory Cache Library “node-cache”

node-cache (https://www.npmjs.com/package/node-cache) is a popular cache library in NodeJS community. It is also an in-memory cache which supports TTL. Functionality wise, it could fit into our use case. However, this is a new dependency to OSD. When attempting to install it to OSD 3.0.0, it introduces 38 vulnerabilities (12 moderate, 26 high). Fixing the vulnerabilities would be extra development cost which doesn’t bring significant benefits compared to lru-cache .

Approach 2 Distributed Cache Library “memcached”

memcached (https://www.npmjs.com/package/memcached) as a distributed cache stores frequently accessed data across multiple nodes. It allows cache data to sync over the network which increases the scalability and robustness. Considering our call pattern where updates are very rare, the implementation and maintenance bring extra complexity to the system.

Approach 3 External Cache Registration

Similar to how we allow other plugins to register a different storage besides the default OpenSearch as storage, we could expose registration function to take a cache client which meets our defined cache interface. By default, a lru-cache cache will be provided. We haven’t seen any new use cases which the proposed cache could not meet. Thus it is a bit over engineering for such flexibility. If there are emerging requirements for a different cache technology, it is not a breaking change to introduce it with an existing implementation based on the proposed cache.

[tianleh]

@SuZhou-Joe @kavilla @bandinib-amzn @ZilongX @xinruiba @Flyingliuhub

cc @seraphjiang @zengyan-amazon