Prefetching a configurable amount of data. #394
Conversation
There was a problem hiding this comment.
Thanks, @AnatolyPopov .
This looks promising. I just have some comments about config definition, and some suggestions to refactor the prefetching logic.
Overall, the approach looks sound to me: Pre-fetching once per fetch request.
For this to be effective, we should consider the prefetched size to be at least higher than the chunk size (else no chunk will be cached) and, better, to be equal or larger than the max fetch bytes per partition, so at least all the get requests to e.g. S3 will be done at once. If double, we can expect the next fetch request to be sourced directly from local cache.
Also, cache size will be affected by this. If 30MB is prefetched per partition, then a broker to prefetch from 10 partitions concurrently will require at least 300MB to cache all required chunks.
core/src/main/java/io/aiven/kafka/tieredstorage/RemoteStorageManager.java
Outdated
Show resolved
Hide resolved
core/src/main/java/io/aiven/kafka/tieredstorage/chunkmanager/cache/ChunkCacheConfig.java
Show resolved
Hide resolved
core/src/main/java/io/aiven/kafka/tieredstorage/chunkmanager/cache/ChunkCacheConfig.java
Outdated
Show resolved
Hide resolved
core/src/main/java/io/aiven/kafka/tieredstorage/chunkmanager/cache/ChunkCache.java
Outdated
Show resolved
Hide resolved
AnatolyPopov
force-pushed
the
anatolii/prefetching
branch
3 times, most recently
from
d7cce31 to
41f01af
Compare
| if (chunkManager instanceof ChunkCache) { | ||
| ((ChunkCache<?>) chunkManager).startPrefetching(segmentKey, segmentManifest, range.from); | ||
| } |
There was a problem hiding this comment.
Any reason this is happening here and not in ChunkCache#getChunk? Looks like a leaky abstraction.
There was a problem hiding this comment.
You are right, fixed.
AnatolyPopov
force-pushed
the
anatolii/prefetching
branch
from
41f01af to
4808dcd
Compare
core/src/test/java/io/aiven/kafka/tieredstorage/chunkmanager/cache/ChunkCacheConfigTest.java
Outdated
Show resolved
Hide resolved
core/src/main/java/io/aiven/kafka/tieredstorage/manifest/index/ChunkIndex.java
Outdated
Show resolved
Hide resolved
No description provided.