Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
rangefeed: improve memory accounting for event queue
To prevent OOMs, we previously introduced memory budget to limit event buffering to processor.eventC channel. These events could point to large underlying data structure and keeping them in the channel prevent them from being garbage collected. However, the current memory accounting did not have a good way to predict the memory footprint of events. It uses calculateDateSize which 1. does not account for memory of the new RangeFeedEvents that could be spawned (such as checkpoint events) 2. inaccurately uses protobuf-encoded wire size for data 3. does not account for memory of the base structs. This patch improves the memory estimation by resolving the three points above. It tries its best effort to account for base struct's memory, predict generation of new events, and use actual data size rather than compressed protobuf size. Since it is challenging to predict whether there would be a new checkpoint event, our strategy in this PR is over-accounting >> under-accounting. GitHub issue (todo) tracks the remaining work to improve accuracy. Resolves: #114190 Release note: None
- Loading branch information