Skip to content

Commit

Permalink
backupccl: add setting to write files in SQL
Browse files Browse the repository at this point in the history
This adds a setting -- default off -- to force BACKUP to always ask
KV to return files to SQL to write instead of writing them directly.
This is currently what it does for tenants but not for the system
tenant due to the extra network hop, data copy, cpu and mem overhead.

Release note: none.
  • Loading branch information
dt committed Jun 16, 2021
1 parent 8308b93 commit 8aae3c9
Showing 1 changed file with 7 additions and 1 deletion.
8 changes: 7 additions & 1 deletion pkg/ccl/backupccl/backup_processor.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,12 @@ var (
time.Minute*5,
settings.NonNegativeDuration,
)
alwaysWriteInProc = settings.RegisterBoolSetting(
"bulkio.backup.proxy_file_writes.enabled",
"return files to the backup coordination processes to write to "+
"external storage instead of writing them directly from the storage layer",
false,
)
)

const backupProcessorName = "backupDataProcessor"
Expand Down Expand Up @@ -201,7 +207,7 @@ func runBackupProcessor(
exportRequestStoreByLocalityKV := storageConfByLocalityKV

// If this is a tenant backup, we need to write the file from the SQL layer.
writeSSTsInProcessor := !flowCtx.Cfg.Codec.ForSystemTenant()
writeSSTsInProcessor := !flowCtx.Cfg.Codec.ForSystemTenant() || alwaysWriteInProc.Get(&clusterSettings.SV)

var defaultStore cloud.ExternalStorage
if writeSSTsInProcessor {
Expand Down

0 comments on commit 8aae3c9

Please sign in to comment.