Skip to content

Commit

Permalink
Merge #46108
Browse files Browse the repository at this point in the history
46108: backup: re-enable fast incremental BACKUP via TBI r=dt a=dt

Within the current BACKUP/RESTORE feature offering, the way to reduce
RPO is to backup more often. For this to work for more frequent
durations, the cost of backing up what changed must scale with the size
what changed, as opposed to the total data size. This is exactly what
the time-bound iterator optimization is supposed to deliver -- by
recording the span of timestamps that appear in any given SSTable and
then installing a filter to only open sstables that contain relevant
times, we can ignore irrelevant data very cheaply.

However over 2017 and 2018, we encountered a few tricky correctness
issues in the interaction of the time-bound iterator and our MVCC logic
and handling intents, and generally lost confidence in this code,
ultimately disabling its use in the scans used by BACKUP. However #45785
resolved those concerns by ensuring that every key emitted by the
IncrementalIterator was actually read by a normal, non-TBI iterator even
when TBI is used. Thus it is now believed to be safe to re-enable TBI
for incremental backups.

Closes #43799.

Release note (enterprise change): Incremental BACKUP can quickly skip of unchanged data making frequent incremental BACKUPs 10-100x faster depending on data-size and frequency.

Release justification: low-risk and high-impact.

Co-authored-by: David Taylor <[email protected]>
  • Loading branch information
craig[bot] and dt committed Mar 16, 2020
2 parents a06be62 + 347243e commit 2fac30c
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion pkg/ccl/backupccl/backup_planning.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ var fullClusterSystemTables = []string{
var useTBI = settings.RegisterBoolSetting(
"kv.bulk_io_write.experimental_incremental_export_enabled",
"use experimental time-bound file filter when exporting in BACKUP",
false,
true,
)

var backupOptionExpectValues = map[string]sql.KVStringOptValidate{
Expand Down

0 comments on commit 2fac30c

Please sign in to comment.