ESQL: Add memory Accountancy to Blocks and Vectors #99235
Conversation
ChrisHegarty
added
>refactoring
Team:QL (Deprecated)
|
Pinging @elastic/es-ql (Team:QL) |
|
Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL) |
...k/plugin/esql/compute/src/test/java/org/elasticsearch/compute/data/BlockAccountingTests.java
Outdated
Show resolved
Hide resolved
| @@ -57,6 +61,16 @@ public DoubleBlock expand() { | |||
| return new DoubleArrayBlock(values, end, firstValues, shiftNullsToExpandedPositions(), MvOrdering.UNORDERED); | |||
| } | |||
|
|
|||
| public static long ramBytesEstimated(double[] values, int[] firstValueIndexes, BitSet nullsMask) { | |||
There was a problem hiding this comment.
Is it useful that this is public static? I feel like it might be useful to pre-bump a breaker, but you'd want to do that before allocating the arrays.
There was a problem hiding this comment.
Yeah, it's still a little unclear if these need to be public or not. I felt them public for now. The idea (for a subsequent PR), since the various arrays can grow independently from each other, is to account for / circuit-break the array element sizes in the builder, then compute and add the actual difference when building the block. That way we can work off reasonable estimates until we build the actual block.
|
@elasticmachine update branch |
This commit adds memory accountancy to Blocks and Vectors.
This change is just the first step along the way to eventually add circuit breaking - we first need to the ability to determine ram usage, both before and after construction. Generally, we can track primitive array allocation and growth in builders separately (either before or after allocation), then use the block
ramBytesUsedwhen releasing (giving back to the breaker). Care will be needed when calculating the actual ram bytes used before allocation (if needed), since that will need to take object reference and array header sizes into account, but seems reasonable.relates #99173