storage: investigate interaction between SST sideloading and raft entry cache

[tbg]

Currently we use the Raft entry cache to hold on to the SSTable payloads when they are inlined as this saves roundtrips to disk. However, since the entry cache is shared across the store, workloads running concurrently with a RESTORE operation could see their performance severely degraded.

[nvanbenschoten]

I suspect that this will result in comically bad behavior. The Raft entry with the sideloaded SST will come along and flush the entire cache, evicting everything so that they all need to be read from disk during application. Then a single entry from a different Range will come along and flush the sideloaded SST entry before it applies, rendering the entire process of caching the inlined entry completely useless.

I think the best fix would be to cache only thin SST entries. This will result in an extra disk read to pull the SST into memory when it is applied, but that's a fixed cost and it will avoid the current behavior of needlessly hurting other concurrent work.

[nvanbenschoten]

This was fixed by the new raftentry.Cache, which refuses to cache Raft entries above 16 MB. These large entries no longer cause all other entries to be evicted.