-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
storage: avoid unmarshaling snapshot data in Store.canApplySnapshot #7850
Comments
We could also unmarshal into a partial message type:
By using the same tag numbers (and disabling unrecognized-field support), we can parse just the parts we need and skip over the rest (although just skipping over all the KV pairs takes time, so we'd be saving memory more than CPU). |
cc #7830 since a solution proposed there (range reservations) would play along nicely as a container for the unmarshaled snapshot. |
In Store.canApplySnapshot, the entire snapshot is unmarshaled just to access the range descriptor (and then unmarshaled again later). Introduce a new proto, PartialRaftSnapshot, which only unmarshals the range descriptor. This is not a complete fix, but saves some memory. Close cockroachdb#7850, for now.
This has been fixed with streaming snapshots; the range descriptor is now transmitted separately from the bulk of the snapshot data. |
Store.canApplySnapshot
unmarshals the snapshot data (roachpb.RaftSnapshotData
) in order to access theroachpb.RaftSnapshotData.RangeDescriptor
which is needed for the range start and end keys. TheRangeDescriptor
is small in comparison toroachpb.RaftSnapshotData
. We could either replicate it inRaftMessageRequest
or include the start and end keys (i.e. aroachpb.Span
) inRaftMessageRequest
whenever a snapshot is present.The text was updated successfully, but these errors were encountered: