CBG-4323: fix for per shard memory based eviction

[gregns1]

CBG-4323

• Hit a panic in 3.2.1 testing that was due to using the overall memory stat for memory based eviction when using sharded rev cache, this panic was from trying to evict from empty rev cache shard. As I was using the overall stat for rev cache memory usage when I hit a point where memory based eviction was needed (as overall memory exceeded the parameter), per chance I hit this point when trying to add a new doc that was going to empty shard. When adding this doc eviction as hit and then we tried to evict from the current shard but to was empty.
• Fix is to keep separate count for memory usage per shard and evict per shard aligning with what we do for num items eviction
• Had to do this through another atomic as we don't always hold rev cache lock when incrementing/decrementing memory and reacquiring lock will probably be more expensive
• Have minimum of 50MB for cache config
• Removed 10% buffer per shard

Pre-review checklist

• Removed debug logging (fmt.Print, log.Print, ...)
• Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
• Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Integration Tests

• GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/2764/

[github-actions]

Redocly previews

• Admin API
• Public API
• Metric API
• Diagnostic API

[adamcfraser]

Generally looks good with some naming/comment suggestions for clarity and a suggestion for a future enhancement ticket.

[adamcfraser]

It doesn't look like this needs to be a public property (can be maxMemoryCapacity). Actually - I'm not clear why this needed to change from memoryCapacity.

[gregns1]

Making public was mistake, the renaming was just to make it more clear in terms of distinction from currMemoryCapacity but after the rename of that to currMemoryUsage I can revert it.

[adamcfraser]

In other scenarios like this (for the channel cache in particular), we compute aggregate stats at stat collection time (see UpdateCalculatedStats). The above approach may be fine for 3.2.1 (based on performance results), but filing a tracking ticket to use this approach going forward would be a good idea.

[gregns1]

This was the approach I wanted to take, but it came with its complexities with fetching the current memory usage. Would require a new interface method on the rev cache which I thought at this time may not be wanted (given the time constraints). I can file a ticket to explore this in a future release.

stale review