Skip to content

Commit

Permalink
[#24783] docdb: Skip loading transactions with start_time < min_repla…
Browse files Browse the repository at this point in the history
…y_txn_start_time

Summary:
Transactions that have start_time < min_replay_txn_start_time are by definition
transactions that are not used at bootstrap time, so there is no need to load their metadata
into transaction participant.

When CDC is in use (before image and lagging stream cases), there may be very large
intent SST files. While we filter out entire intent SST files during transaction loading in
tablet bootstrap when the entire SST file is below min_replay_txn_start_time, this still
leaves us with up to several million unnecessary transactions loaded, costing multiple
GB of memory and causing further issues (slow/no progress in bootstrap)
 if there is insufficient memory as a result.

This diff avoids loading those transactions into transaction participant.
Jira: DB-13882

Test Plan: Jenkins: urgent

Reviewers: sergei

Reviewed By: sergei

Subscribers: rthallam, ybase

Differential Revision: https://phorge.dev.yugabyte.com/D39730
  • Loading branch information
es1024 committed Nov 8, 2024
1 parent 5255fd0 commit fa6ff5b
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions src/yb/tablet/transaction_participant.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1894,6 +1894,12 @@ class TransactionParticipant::Impl
TransactionalBatchData&& last_batch_data,
OneWayBitmap&& replicated_batches,
const ApplyStateWithCommitHt* pending_apply) override {
auto start_time = metadata.start_time;
auto replay_start_time = MinReplayTxnStartTime();
if (start_time && replay_start_time && start_time < replay_start_time) {
return;
}

MinRunningNotifier min_running_notifier(&applier_);
std::lock_guard lock(mutex_);
auto txn = std::make_shared<RunningTransaction>(
Expand Down

0 comments on commit fa6ff5b

Please sign in to comment.