[Feature] Enhanced snapshot compaction based on events size

[shreyas-s-rao]

Feature (What you would like to be added):
Smarter snapshot compaction based on events size.

Motivation (Why is this needed?):
Snapshot compaction today is based purely on the number of etcd events recorded in the delta snapshots, based on which druid takes the decision to trigger a compaction job for compacting the latest set of delta snapshots into the latest full snapshot. This approach does not consider the size of the events recorded by the delta snapshots. Events can be quite large, upto 1.5 MB which is the default threshold for event size defined by etcd.

To put things in perspective, the default etcd events threshold for druid is 1 million events. In the worst case, if each event is 1.5 MB large, then the total size of accumulated delta snapshots can go up to 1.5 million MB, ie, 1.5 TB. Compacting this kind of backup is practically impossible due to the following reasons:

1. During restoration as part of compaction, the etcd does not support this scale of data since the DB is not defragmented during restoration
2. Even if the DB size is kept in check during restoration with the use of multiple defragmentations while applying the events to the etcd, the amount of time it takes to compact such a DB is impractically large
3. Letting the backup size blow up means a very long restoration time in case of an actual restoration upon a PV corruption

We cannot rely on the number of etcd events alone to trigger snapshot compactions.

Approach/Hint to the implement solution (optional):
Druid should also consider the total size of the accumulated events and trigger compaction after a certain (configurable) threshold is reached. For druid to get this information, it must depend on the EtcdMember[State] resource using which the backup-restore sidecar publishes snapshot information for druid to consume and act upon.

[shreyas-s-rao]

/assign @abdasgupta