Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv: Add server option for maximum number of intents per WriteIntentError for scans #64783

Closed
aliher1911 opened this issue May 6, 2021 · 2 comments · Fixed by #65465
Closed

kv: Add server option for maximum number of intents per WriteIntentError for scans #64783

aliher1911 opened this issue May 6, 2021 · 2 comments · Fixed by #65465
Assignees
@aliher1911
Labels
A-kv-transactions Relating to MVCC and the transactional model. A-storage Relating to our storage engine (Pebble) on-disk storage. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) N-followup Needs followup. O-postmortem Originated from a Postmortem action item. T-kv KV Team

Comments

@aliher1911
Copy link
Contributor

aliher1911 commented May 6, 2021 •
edited by lunevalex
Loading

We added batching of discovered intents for subsequent resolutions during sst exports similar to what scan does (Issue: #59704, PR: #64131). On top of that, the change provides an option to limit number of collected intents to keep batch size predictable.

We may consider using the same approach to MVCCScan to make number of collected intents capped.

Epic: CRDB-2554
@aliher1911 aliher1911 added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label May 6, 2021
@aliher1911
Copy link
Contributor Author

@nvanbenschoten is it used for range scans mostly? Can you provide some context on when this size would be relevant?

@aliher1911 aliher1911 mentioned this issue May 6, 2021
Merged
@nvanbenschoten nvanbenschoten changed the title Add server option for maximum number of intents per WriteIntentError for scans kv: Add server option for maximum number of intents per WriteIntentError for scans May 6, 2021
@nvanbenschoten nvanbenschoten added A-kv-transactions Relating to MVCC and the transactional model. A-storage Relating to our storage engine (Pebble) on-disk storage. labels May 6, 2021
@nvanbenschoten
Copy link
Member

Can you provide some context on when this size would be relevant?

This size limit would be relevant for any large scans. Currently, there is no limit on the number of intents that a scan (coordinated by the pebbleMVCCScanner) can pull into memory and hand up to the lockTable in a WriteIntentError. So the only real limit is the size of a range, which can only hold about 512MB worth of intents before splitting.

@aliher1911 aliher1911 self-assigned this May 18, 2021
@tbg tbg added the O-postmortem Originated from a Postmortem action item. label May 19, 2021
@aliher1911 aliher1911 linked a pull request May 19, 2021 that will close this issue
storage: max number of intents in WriteIntentError during scan #65465
Merged
@craig craig bot closed this as completed in #65465 Jun 1, 2021
@lunevalex lunevalex added the N-followup Needs followup. label Jun 4, 2021
@exalate-issue-sync exalate-issue-sync bot added the T-kv KV Team label Jun 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aliher1911 aliher1911

Labels
A-kv-transactions Relating to MVCC and the transactional model. A-storage Relating to our storage engine (Pebble) on-disk storage. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) N-followup Needs followup. O-postmortem Originated from a Postmortem action item. T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

storage: max number of intents in WriteIntentError during scan aliher1911/cockroach
4 participants
@tbg @aliher1911 @nvanbenschoten @lunevalex