casync decompresses x1.5 faster than borg on same config (and other benchmarks) #7674
Comments
|
Thanks for the information and doing so much testing. I see you used misc. VPSes - how can you know they provide consistent CPU and I/O performance, so your benchmark timings are comparable? Even on bare hardware, timings might fluctuate quite a bit due to other stuff the machine is doing in parallel. Also: did these VPSes offer sha2 hw acceleration? This is quite importing when using sha256, which is otherwise quite slow. Without sha2 hw accel, blake2 is better (in case of borg). You need to use a strong cryptographic hash - there is no point in using a fast weak hash if you have a higher risk of hash collisions. At "extraction" time, one needs to assert Maybe for comparisons of misc tools, it would be better to just focus on one compression algorithm and level and have less result data to review. Of course that should be a fast one as we are not testing for compression performance, but rather for tool performance, so I'ld suggest lz4 or zstd,1. |
|
I did some more testing. And now I see that AWS's performance is completely unstable. Just now I added "00" to empty borg repo with settings ( @ThomasWaldmann, I see your questions, I will answer them later) |
|
I did better benchmark. Here is list of changes (see the first post #7674 (comment) for context):
My dedupper: Cargo.toml[package]
name = "my-chunker"
version = "0.1.0"
edition = "2021"
[dependencies]
blake2b_simd = "1.0.1"
blake3 = "1.4.0"
fastmurmur3 = "0.2.0"
hex = "0.4.3"
libc = "0.2.146"
openssl = { version = "0.10.55", features = ["vendored"] }
zstd = "0.12.3"src/main.rs#![feature(fs_try_exists)]
#![feature(file_create_new)]
use std::io::Read;
use std::io::Write;
struct Killed;
// "kill" just means "completely drop this value". I. e. we drop it and then shadow it to make sure we will not be able to access it even if it is Copy. I. e. when you see "kill!", assume it is "drop"
macro_rules! kill {
($x:ident) => {
#[allow(dropping_references)]
#[allow(dropping_copy_types)]
std::mem::drop($x);
#[allow(unused_variables)]
let $x = Killed;
}
}
#[derive(Clone, Copy)]
enum Hash {
MurMur3,
Sha256,
Blake3,
Blake2b,
}
fn calc_hash(hash_type: Hash, data: &[u8]) -> Vec<u8> {
return match hash_type {
Hash::MurMur3 => fastmurmur3::hash(data).to_le_bytes().to_vec(),
Hash::Sha256 => {
openssl::sha::sha256(data).to_vec()
/*use sha2::Digest;
let mut hasher = sha2::Sha256::new();
hasher.update(data);
hasher.finalize().to_vec()*/
},
Hash::Blake3 => blake3::hash(data).as_bytes().to_vec(),
Hash::Blake2b => blake2b_simd::blake2b(data).as_bytes().to_vec(),
};
}
fn hash_name_to_enum(name: &str) -> Hash {
return match name {
"--strong-hash=murmur3" => Hash::MurMur3,
"--strong-hash=sha256" => Hash::Sha256,
"--strong-hash=blake3" => Hash::Blake3,
"--strong-hash=blake2b" => Hash::Blake2b,
_ => panic!(),
};
}
// "[()] = [...]" is needed because of clippy bug: https://github.com/rust-lang/rust-clippy/issues/9048 . So when you see "[()] = [f()]", just assume "() = f()"
fn main() {
if std::env::args().collect::<Vec<_>>()[1] == "add" {
let [_, _, ref o_storage, ref o_block_bytes, ref o_zstd, ref o_hash] = *std::env::args().collect::<Vec<_>>() else { panic!("Usage"); };
let block_bytes: usize = o_block_bytes.strip_prefix("--block-bytes=").unwrap().parse().unwrap();
kill!(o_block_bytes);
if !(4096 <= block_bytes && block_bytes <= 10_000_000_000) {
panic!();
}
let zstd_level: i32 = o_zstd.strip_prefix("--zstd=").unwrap().parse().unwrap();
kill!(o_zstd);
let hash_type = hash_name_to_enum(o_hash);
kill!(o_hash);
let storage = o_storage.strip_prefix("--chunks=").unwrap();
kill!(o_storage);
let mut input = std::io::stdin().lock();
let mut stdout = std::io::stdout().lock();
let mut buf = vec![0u8; block_bytes];
kill!(block_bytes);
let mut hashs = vec![];
loop {
let buf_size = input.read(&mut buf).unwrap();
// buf_size can be lesser than block_bytes, even if we are not in EOF. We consider this normal (in other words, it is NOT guaranted that we will actually split file as equal-sized blocks)
if buf_size == 0 {
break;
}
let buf = &buf[..buf_size];
kill!(buf_size);
let hash = calc_hash(hash_type, &buf);
hashs.extend_from_slice(&hash);
let hash_str = hex::encode(&hash);
kill!(hash);
if !std::fs::try_exists(format!("{storage}/{hash_str}")).unwrap() {
let compressed = zstd::bulk::compress(buf, zstd_level).unwrap();
let mut file = std::fs::File::create_new(format!("{storage}/{hash_str}")).unwrap();
[()] = [file.write_all(&u64::try_from(buf.len()).unwrap().to_le_bytes()).unwrap()];
kill!(buf);
[()] = [file.write_all(&compressed).unwrap()];
}
}
[()] = [stdout.write_all(&hashs).unwrap()];
} else if std::env::args().collect::<Vec<_>>()[1] == "extract" {
let [_, _, ref o_storage, ref o_hash, ref o_check] = *std::env::args().collect::<Vec<_>>() else { panic!("Usage"); };
let hash_type = hash_name_to_enum(o_hash);
kill!(o_hash);
let storage = o_storage.strip_prefix("--chunks=").unwrap();
kill!(o_storage);
let check = match &**o_check {
"--check=true" => true,
"--check=false" => false,
_ => panic!(),
};
let mut input = std::io::stdin().lock();
let mut output = std::io::stdout().lock();
loop {
let mut hash_buf = vec![0u8; match hash_type {
Hash::MurMur3 => 128 / 8,
Hash::Sha256 => 256 / 8,
Hash::Blake3 => 256 / 8,
Hash::Blake2b => 64,
}];
{
let have_read = input.read(&mut hash_buf).unwrap();
if have_read == 0 {
break;
} else if have_read == hash_buf.len() {
// OK
} else {
panic!();
}
}
let hash_str = hex::encode(&hash_buf);
let mut file = std::fs::File::open(format!("{storage}/{hash_str}")).unwrap();
let mut size = [0u8; 8];
[()] = [file.read_exact(&mut size).unwrap()];
let size = usize::try_from(u64::from_le_bytes(size)).unwrap();
let mut data = vec![];
let _: usize = file.read_to_end(&mut data).unwrap();
kill!(file);
{
let decompressed = zstd::bulk::decompress(&data, size).unwrap();
kill!(data);
assert_eq!(decompressed.len(), size);
kill!(size);
// I think we don't need to check hash here. So the check is optional (but we check decompressed size to make sure chunk is not truncated)
if check {
assert_eq!(calc_hash(hash_type, &decompressed), hash_buf);
}
[()] = [output.write_all(&decompressed).unwrap()];
}
}
} else {
panic!("Usage");
}
}Program for executing benchmarks: Cargo.toml[package]
name = "rust"
version = "0.1.0"
edition = "2021"
[dependencies]
regex = "1.8.4"
serde_json = { version = "1.0.79", features = ["preserve_order"] }
serde = { version = "1.0.136", features = ["derive"] }src/main.rs#![feature(exit_status_error)]
// "[()] = [...]" is needed because of clippy bug: https://github.com/rust-lang/rust-clippy/issues/9048 . So when you see "[()] = [f()]", just assume "() = f()"
fn run(s: &str) {
[()] = [std::process::Command::new("bash").arg("-ec").arg(s).status().unwrap().exit_ok().unwrap()];
}
fn main() {
const REPO: &str = "/home/user/dedup-bench/fs/repo";
const CMP_ME: &str = "/home/user/dedup-bench/fs/cmp-me";
#[derive(Debug)]
enum Method {
ZStd,
Borg { chunker_params: String, compression: String, encryption: String, },
Casync { chunker_params: String, },
Zpaq,
MyChunker { block_bytes: usize, zstd: i32, hash: String, check: bool },
}
let range = 0..=11;
// let range = 0..=2;
// let range = 0..=5;
run(&format!("rm -rf {REPO}"));
run(&format!("rm -rf {CMP_ME}"));
#[derive(Debug)]
struct Run {
method: Method,
size: usize,
compression: Vec<std::time::Duration>,
decompression: Vec<std::time::Duration>,
total_compression: std::time::Duration,
total_decompression: std::time::Duration,
}
let mut methods = vec![];
{
methods.push(Method::ZStd);
for chunker_params in [
"buzhash,19,23,21,4095", // borg default
"buzhash,10,23,16,4095", // borg alternative
"buzhash,14,18,16,4095", // casync default
"buzhash,19,23,21,48", // borg default (casync's window size)
"buzhash,10,23,16,48", // borg alternative
"buzhash,14,18,16,48", // casync default
"fixed,4194304",
] {
for compression in [
"zstd,3", // casync default
] {
for encryption in [
"none",
"authenticated",
"authenticated-blake2",
] {
methods.push(Method::Borg { chunker_params: chunker_params.to_owned(), compression: compression.to_owned(), encryption: encryption.to_owned() });
}
}
}
for chunker_params in [
format!("{}:{}:{}", 1 << 19, 1 << 21, 1 << 23), // borg default
format!("{}:{}:{}", 1 << 10, 1 << 16, 1 << 23), // borg alternative
format!("{}:{}:{}", 1 << 14, 1 << 16, 1 << 18), // casync default
] {
methods.push(Method::Casync { chunker_params: chunker_params.to_owned(), });
}
methods.push(Method::Zpaq);
for block_bytes in [4194304] {
for zstd in [3] {
for hash in ["murmur3", "sha256", "blake3", "blake2b"] {
for check in [false, true] {
methods.push(Method::MyChunker { block_bytes, zstd, hash: hash.to_owned(), check });
}
}
}
}
}
for method in methods {
run(&format!("mkdir {REPO}"));
run(&format!("mkdir {CMP_ME}"));
// Initializing
match &method {
Method::ZStd => {},
Method::Borg { encryption, .. } => run(&format!("BORG_PASSPHRASE=password borg2 rcreate --encryption={encryption} --repo {REPO}")),
Method::Casync { .. } => run(&format!("mkdir {REPO}/index {REPO}/storage.castr")),
Method::Zpaq => {},
Method::MyChunker { .. } => run(&format!("mkdir {REPO}/chunks {REPO}/index")),
}
let mut compression = vec![];
for i in range.clone() {
run(&format!("sync -f {REPO}/; sync -f {CMP_ME}/"));
println!("**** {method:?} {i} compression...");
let now = std::time::Instant::now();
match &method {
Method::ZStd => run(&format!("zstd -3 -T1 < /home/user/dedup-bench/sto/{i:02} > {REPO}/{i:02}.zst")),
Method::Borg { chunker_params, compression, encryption: _ } => run(&format!("cd /home/user/dedup-bench/sto; BORG_PASSPHRASE=password borg2 create --repo {REPO} --chunker-params {chunker_params} --compression {compression} {i:02} {i:02}")),
Method::Casync { chunker_params } => run(&format!("casync make --compression=zstd --chunk-size={chunker_params} --store={REPO}/storage.castr {REPO}/index/{i:02}.caibx /home/user/dedup-bench/sto/{i:02} > /dev/null")),
Method::Zpaq => run(&format!("cd /home/user/dedup-bench/sto; zpaq add {REPO}/repo.zpaq {i:02} -threads 1")),
Method::MyChunker { block_bytes, zstd, hash, check: _ } => run(&format!("/home/user/dedup-bench/my-chunker/target/release/my-chunker add --chunks={REPO}/chunks --block-bytes={block_bytes} --zstd={zstd} --strong-hash={hash} < /home/user/dedup-bench/sto/{i:02} > {REPO}/index/{i:02}")),
}
run(&format!("sync -f {REPO}/; sync -f {CMP_ME}/"));
compression.push(now.elapsed());
println!("**** {method:?} {i} compression: {:?}", now.elapsed());
}
let output = std::process::Command::new("bash").arg("-ec").arg(format!("du --bytes --apparent-size -s {REPO}")).output().unwrap();
[()] = [output.status.exit_ok().unwrap()];
assert_eq!(output.stderr.len(), 0);
let size: usize;
{
let x = String::from_utf8(output.stdout).unwrap();
let caps = regex::Regex::new(r##"^(?<s>[0-9]+)\t/[-/a-z]+\n$"##).unwrap().captures_iter(&x).collect::<Vec<_>>();
assert!(caps.len() == 1);
size = caps[0]["s"].parse().unwrap();
}
{
let len = size
.to_string()
.chars()
.collect::<Vec<_>>()
.rchunks(3)
.map(|x|x.into_iter().collect::<String>())
.rev()
.collect::<Vec<_>>()
.join("_");
println!("size: {len}");
}
let mut decompression = vec![];
for i in range.clone() {
run(&format!("sync -f {REPO}/; sync -f {CMP_ME}/"));
println!("**** {method:?} {i} decompression...");
let now = std::time::Instant::now();
match method {
Method::ZStd => run(&format!("zstd -d -T0 < {REPO}/{i:02}.zst > {CMP_ME}/data")),
Method::Borg { .. } => run(&format!("cd {CMP_ME}; BORG_PASSPHRASE=password borg2 extract --repo {REPO} {i:02} {i:02}; mv -i {i:02} data")),
Method::Casync { .. } => run(&format!("casync extract --store={REPO}/storage.castr {REPO}/index/{i:02}.caibx {CMP_ME}/data")),
Method::Zpaq => run(&format!("cd {CMP_ME}; zpaq extract {REPO}/repo.zpaq {i:02}; mv -i {i:02} data")),
Method::MyChunker { block_bytes: _, zstd: _, ref hash, check } => run(&format!("/home/user/dedup-bench/my-chunker/target/release/my-chunker extract --chunks={REPO}/chunks --strong-hash={hash} --check={check:?} < {REPO}/index/{i:02} > {CMP_ME}/data")),
}
run(&format!("sync -f {REPO}/; sync -f {CMP_ME}/"));
decompression.push(now.elapsed());
println!("**** {method:?} {i} decompression: {:?}", now.elapsed());
run(&format!("cmp /home/user/dedup-bench/sto/{i:02} {CMP_ME}/data"));
run(&format!("rm {CMP_ME}/data"));
}
run(&format!("rm -r {REPO}"));
run(&format!("rm -r {CMP_ME}"));
let total_compression = compression.iter().sum();
let total_decompression = decompression.iter().sum();
let run = Run { method, size, compression, decompression, total_compression, total_decompression };
println!("{:?}", run);
#[derive(Debug, serde::Serialize)]
struct JsonRun {
method: String,
size: usize,
compression: Vec<u128>,
decompression: Vec<u128>,
total_compression: u128,
total_decompression: u128,
}
let Run { method, size, compression, decompression, total_compression, total_decompression } = run;
println!("{}", serde_json::to_string(&JsonRun {
method: format!("{:?}", method),
size,
compression: compression.into_iter().map(|x|x.as_micros()).collect(),
decompression: decompression.into_iter().map(|x|x.as_micros()).collect(),
total_compression: total_compression.as_micros(),
total_decompression: total_decompression.as_micros(),
}).unwrap());
}
}Programs under test:
Results in json (this time 34 lines only here, so there is nothing scary here): {"method":"ZStd","size":11657765795,"compression":[9283690,9678287,11367581,26593358,28771616,28750747,31540400,31680346,32376822,32378808,33297324,33229071],"decompression":[2962382,2323811,2608823,5841187,6354606,5967899,6779104,6508008,6673395,7319743,6901865,7207101],"total_compression":308948055,"total_decompression":67447929}
{"method":"Borg { chunker_params: \"buzhash,19,23,21,4095\", compression: \"zstd,3\", encryption: \"none\" }","size":2851383597,"compression":[18318222,11317438,21237556,45904208,44393790,40996476,48499568,45409089,46290323,44191118,46500109,34804936],"decompression":[8965775,8972195,9445975,35787014,36940309,36681542,37723561,37497282,38120893,37698965,37893902,38152120],"total_compression":447862839,"total_decompression":363879538}
{"method":"Borg { chunker_params: \"buzhash,19,23,21,4095\", compression: \"zstd,3\", encryption: \"authenticated\" }","size":2818873535,"compression":[20102364,12104014,21250364,46696052,45152668,41188824,48250885,44991233,46159774,44186195,36323582,27978731],"decompression":[9221565,9235633,9506502,36005242,36901609,36936570,37744812,37906248,38391585,38012484,38246767,38289875],"total_compression":434384693,"total_decompression":366398899}
{"method":"Borg { chunker_params: \"buzhash,19,23,21,4095\", compression: \"zstd,3\", encryption: \"authenticated-blake2\" }","size":2849714452,"compression":[20135335,11978013,21000557,46756333,45916109,41730267,48565669,45327100,46166187,43952637,37075990,28128240],"decompression":[8950584,9058208,9232180,35887827,36183489,36365780,38041709,38177238,37987861,37854767,35663794,36203717],"total_compression":436732442,"total_decompression":359607160}
{"method":"Borg { chunker_params: \"buzhash,10,23,16,4095\", compression: \"zstd,3\", encryption: \"none\" }","size":2149064038,"compression":[21433546,12854993,19036014,45568056,46895977,42824579,48388277,46630976,48315401,46168988,37319377,29560584],"decompression":[9630351,9612465,10176755,36682904,37336673,37462619,38755277,38619795,38835548,39275470,39528064,39358626],"total_compression":444996774,"total_decompression":375274551}
{"method":"Borg { chunker_params: \"buzhash,10,23,16,4095\", compression: \"zstd,3\", encryption: \"authenticated\" }","size":2146105711,"compression":[21699606,12760410,19070372,44908917,46769279,43102345,48906422,47002961,48680681,46378196,36566607,29968325],"decompression":[9835854,9947821,10439907,36840113,38043775,38003903,39142987,39261288,39676490,39826286,39719886,39615441],"total_compression":445814126,"total_decompression":380353757}
{"method":"Borg { chunker_params: \"buzhash,10,23,16,4095\", compression: \"zstd,3\", encryption: \"authenticated-blake2\" }","size":2157499367,"compression":[20231258,11349910,17575160,43394101,45211290,41337098,46657102,44495116,45848495,43480255,34457257,27440901],"decompression":[8551623,8478314,8925691,33317706,35799953,35982652,36745073,34477784,34747132,37280720,37107451,37128391],"total_compression":421477949,"total_decompression":348542494}
{"method":"Borg { chunker_params: \"buzhash,14,18,16,4095\", compression: \"zstd,3\", encryption: \"none\" }","size":2162838917,"compression":[19109363,10336038,16429374,35520360,36950678,33812374,38626489,37122860,38390452,36431809,29693996,25317951],"decompression":[9474411,9369419,9906256,36352626,37122162,36789206,38670100,38596851,38257199,38321414,38882990,38705175],"total_compression":357741749,"total_decompression":370447816}
{"method":"Borg { chunker_params: \"buzhash,14,18,16,4095\", compression: \"zstd,3\", encryption: \"authenticated\" }","size":2164563083,"compression":[19867389,10839858,17143115,37769750,38042463,33716280,39030970,37433763,38800999,36812705,30371597,25625937],"decompression":[9869781,9855416,10347908,36474136,37551748,37721483,39537396,39539771,38971688,39011972,39560917,39063998],"total_compression":365454832,"total_decompression":377506221}
{"method":"Borg { chunker_params: \"buzhash,14,18,16,4095\", compression: \"zstd,3\", encryption: \"authenticated-blake2\" }","size":2164097589,"compression":[18323471,9420528,15444298,36607423,36075649,32132941,36828551,34915397,36142264,34018785,27635133,22985850],"decompression":[8284186,8156134,8694047,29891890,30563610,30278642,32062420,31944804,31583080,32087650,32181069,32029320],"total_compression":340530295,"total_decompression":307756859}
{"method":"Borg { chunker_params: \"buzhash,19,23,21,48\", compression: \"zstd,3\", encryption: \"none\" }","size":2835630939,"compression":[19633874,11845746,20817763,45678328,44171313,41569956,48457003,45433164,46500342,44508459,35825546,28015594],"decompression":[9040288,9055733,9340413,35888074,37150820,36750609,37612061,37497925,38169093,37852680,38028829,38221644],"total_compression":432457094,"total_decompression":364608174}
{"method":"Borg { chunker_params: \"buzhash,19,23,21,48\", compression: \"zstd,3\", encryption: \"authenticated\" }","size":2837102441,"compression":[20211541,12130002,21500578,47107778,45693547,41644459,48569615,45460227,46557889,44690148,36637580,28108682],"decompression":[9161378,9090975,9472423,36316948,36667485,36725586,37645070,37964265,38433782,38147539,38428219,38334092],"total_compression":438312053,"total_decompression":366387767}
{"method":"Borg { chunker_params: \"buzhash,19,23,21,48\", compression: \"zstd,3\", encryption: \"authenticated-blake2\" }","size":2841300228,"compression":[19839563,11948675,20904316,46158648,44926572,40998189,48310763,45180332,46265710,44223851,36329235,27875015],"decompression":[8834926,8981821,9498129,32888619,36480689,36418669,37700783,34601169,35593303,37906159,37761933,38201286],"total_compression":432960874,"total_decompression":354867489}
{"method":"Borg { chunker_params: \"buzhash,10,23,16,48\", compression: \"zstd,3\", encryption: \"none\" }","size":2135031845,"compression":[21824758,12569046,18681384,45450353,47126995,43234937,48642344,46956004,48549333,45958714,37056332,29454305],"decompression":[9653232,9603802,10341111,36923956,37754482,37579775,38597648,39008547,39077172,38761438,39343815,39212642],"total_compression":445504510,"total_decompression":375857624}
{"method":"Borg { chunker_params: \"buzhash,10,23,16,48\", compression: \"zstd,3\", encryption: \"authenticated\" }","size":2145692735,"compression":[21699815,12685825,19091380,45176414,46856537,43071094,48642945,46888967,48576570,46042103,36645446,29798289],"decompression":[9829861,9798758,10423285,36929763,37717415,37929072,38826619,39035325,39339638,39266448,39800269,39882055],"total_compression":445175390,"total_decompression":378778514}
{"method":"Borg { chunker_params: \"buzhash,10,23,16,48\", compression: \"zstd,3\", encryption: \"authenticated-blake2\" }","size":2144992581,"compression":[20044677,11372242,17551540,43968373,45297914,41559775,46745420,44649502,46056635,43873550,33579295,27196563],"decompression":[8439352,8370019,8874321,33292381,35650680,35798891,34367147,36168242,36831071,36745149,34808103,36881564],"total_compression":421895491,"total_decompression":346226925}
{"method":"Borg { chunker_params: \"buzhash,14,18,16,48\", compression: \"zstd,3\", encryption: \"none\" }","size":2149367658,"compression":[19202188,10282185,16158967,35386376,36863967,33524678,38459514,36883128,38468975,36310201,30181380,25607996],"decompression":[9560774,9720572,10081141,36231706,37176500,36832882,38732369,38359583,38258917,38308732,38774052,38777435],"total_compression":357329559,"total_decompression":370814669}
{"method":"Borg { chunker_params: \"buzhash,14,18,16,48\", compression: \"zstd,3\", encryption: \"authenticated\" }","size":2165212174,"compression":[19994496,11001909,16816310,37486176,38088082,33874466,38906818,37310671,38929168,36742808,30276421,25853547],"decompression":[9848387,9860636,10360622,36518689,37782960,37368730,39356883,39091052,39301352,38903208,39747421,39198518],"total_compression":365280878,"total_decompression":377338462}
{"method":"Borg { chunker_params: \"buzhash,14,18,16,48\", compression: \"zstd,3\", encryption: \"authenticated-blake2\" }","size":2170608909,"compression":[18257714,9587410,15555097,35455938,36490702,32159211,36668103,34849661,36387537,34196607,27808237,22916115],"decompression":[8204530,8187822,8610032,28935022,30186082,29866673,32000599,31765863,32466908,31627415,32487425,32223529],"total_compression":340332337,"total_decompression":306561906}
{"method":"Borg { chunker_params: \"fixed,4194304\", compression: \"zstd,3\", encryption: \"none\" }","size":2837954615,"compression":[16641977,8981518,17861236,31209683,29905411,27442906,34127943,31146410,32103404,30310530,21990501,14716631],"decompression":[9042302,9005331,9466467,35760136,36634388,36565886,37465949,37453933,37762713,37556223,37919234,37787852],"total_compression":296438154,"total_decompression":362420421}
{"method":"Borg { chunker_params: \"fixed,4194304\", compression: \"zstd,3\", encryption: \"authenticated\" }","size":2837954708,"compression":[17080083,9349320,18338772,32196291,30998950,27604881,34254323,31223193,32094800,30438246,22705156,14743870],"decompression":[9182614,9202718,9594391,35669209,36698953,36712690,37423578,37843085,37820586,38092966,37820292,38017167],"total_compression":301027891,"total_decompression":364078253}
{"method":"Borg { chunker_params: \"fixed,4194304\", compression: \"zstd,3\", encryption: \"authenticated-blake2\" }","size":2837955293,"compression":[16681822,9286359,18376840,31673226,30222772,27483311,33793628,30569652,31373937,29621764,21741968,14200312],"decompression":[9017839,9029086,9237240,35717027,34026683,32061108,37518438,37291120,37486746,37251674,35155094,33467949],"total_compression":295025598,"total_decompression":347260010}
{"method":"Casync { chunker_params: \"524288:2097152:8388608\" }","size":2879294306,"compression":[37502132,29828953,38388829,162425453,158344403,155316081,159016379,155002634,155527517,153506076,154936234,153428320],"decompression":[6906559,6962960,7419022,25905660,26947735,26924282,27862707,27946487,28365741,28437593,28403558,28441681],"total_compression":1513223016,"total_decompression":270523990}
{"method":"Casync { chunker_params: \"1024:65536:8388608\" }","size":2095335450,"compression":[40231441,32560349,37437359,166501923,166456700,163119571,165479303,164293470,164053685,163130507,163433177,163316443],"decompression":[6807262,6824586,7248889,26100854,26633976,26576270,27336775,27486418,27915298,27683154,27854510,28035610],"total_compression":1590013933,"total_decompression":266503608}
{"method":"Casync { chunker_params: \"16384:65536:262144\" }","size":2123125835,"compression":[36880804,28885367,33321629,156750983,155885464,153104536,155009332,152763670,153169678,151563761,151688622,151495075],"decompression":[6573250,6564462,7090823,23860988,24478632,24588526,25476227,25581478,26007763,25979937,26140739,26167777],"total_compression":1480518927,"total_decompression":248510607}
{"method":"Zpaq","size":2020546862,"compression":[64846208,15999299,45113975,84095781,88217304,71561268,87819893,74730235,79004485,72064634,74074986,73135629],"decompression":[11504019,11442280,17221606,22817373,27206092,27206680,31463812,32912227,34842936,34780759,35448384,35439551],"total_compression":830663702,"total_decompression":322285723}
{"method":"MyChunker { block_bytes: 4194304, zstd: 3, hash: \"murmur3\", check: false }","size":2837727787,"compression":[11972998,4501924,12993601,28032655,25078869,21966876,26911725,23199234,23649817,21983818,23332558,22013069],"decompression":[2742755,2685176,3043580,6158789,6734138,6970033,7542192,7608766,8009475,7905699,8143381,8095599],"total_compression":245637149,"total_decompression":75639589}
{"method":"MyChunker { block_bytes: 4194304, zstd: 3, hash: \"murmur3\", check: true }","size":2837727787,"compression":[11975526,4508979,12905268,28090643,25081155,21973767,26848370,23173431,23627225,22000042,23324900,22000017],"decompression":[3306916,3255777,3611647,8924207,10175322,9619307,10386219,10362477,11191207,10910435,10920920,10985661],"total_compression":245509328,"total_decompression":103650100}
{"method":"MyChunker { block_bytes: 4194304, zstd: 3, hash: \"sha256\", check: false }","size":2838121003,"compression":[16749795,9173678,17869084,51539683,48496978,45374019,50195634,46567898,47029669,45336113,46776453,45328806],"decompression":[2753205,2697572,3062993,6109653,6723439,6694980,7456240,7805207,7903225,7941173,8034388,8094563],"total_compression":470437817,"total_decompression":75276643}
{"method":"MyChunker { block_bytes: 4194304, zstd: 3, hash: \"sha256\", check: true }","size":2838121003,"compression":[16691595,9177770,17885076,51471117,48542486,45351514,50248956,46587920,47013303,45374848,46801416,45384168],"decompression":[8165492,8177129,8513070,33387636,34289479,33884103,34666111,34804762,35095678,35119255,35282643,35206638],"total_compression":470530175,"total_decompression":336592002}
{"method":"MyChunker { block_bytes: 4194304, zstd: 3, hash: \"blake3\", check: false }","size":2838121003,"compression":[12079168,4576985,13078441,28469710,25368211,22287125,27185902,23682801,23922831,22378973,23702148,22375989],"decompression":[2719902,2694905,3049538,6100986,6721047,6668222,7668869,7601103,7961794,7940910,8065481,8110993],"total_compression":249108289,"total_decompression":75303755}
{"method":"MyChunker { block_bytes: 4194304, zstd: 3, hash: \"blake3\", check: true }","size":2838121003,"compression":[12025524,4493332,13012576,28009321,24979423,21895509,26819883,23100268,23629206,21931283,23282489,21969911],"decompression":[3458388,3461221,3816301,9637175,10457420,10680664,11447276,11428309,11520339,11409510,11951739,11662459],"total_compression":245148729,"total_decompression":110930807}
{"method":"MyChunker { block_bytes: 4194304, zstd: 3, hash: \"blake2b\", check: false }","size":2838907435,"compression":[13230124,5637490,14433041,33935850,30851062,27675085,32690185,30380651,29399646,27728548,29126626,27772168],"decompression":[2714048,2688687,3056966,6073851,6747581,6908087,7494888,7550462,7893937,7930224,8101463,8143793],"total_compression":302860481,"total_decompression":75303993}
{"method":"MyChunker { block_bytes: 4194304, zstd: 3, hash: \"blake2b\", check: true }","size":2838907435,"compression":[13097185,5644976,14237922,33926943,30907781,27754915,32844449,29006030,30476310,27681230,30693571,27749998],"decompression":[4825484,4872713,4938557,15678113,17426670,16434735,17054877,17291963,17479366,18607035,17708124,17825607],"total_compression":304021316,"total_decompression":170143248}The same as a nice table:
Output of Now conclusions.
Now let me list actionable items (in my opinion) for borg:
Rust's sha256 performance problems are strange. I will try to investigate more and possibly will report bug to sha256 implementations. It seems that my host supports sha256 hardware acceleration, but Rust's implementations are unable to detect this (but python's can) |
|
IIRC, we already have some notes in the docs that speed advantages of misc. algorithms depend on the CPU capabilities. But, as you have experienced, this is sometimes a bit tricky or even counter-intuitive. Because of that, I have added a simple I know that
Speed: in general, if some other code than borg is faster, that does not imply that there is something wrong in borg. It could be just that borg does something that that code does not do. But if that is something useful, that's good. So, such a finding needs more research. It might well be though that |
|
Had a look and also re-grouped the lines, removed some less relevant ones, maybe it is useful if someone else looks at this: |
|
Why would borg be faster using |
|
@jdchristensen it has to compute the id-hash in any case and if blake2b is faster than sha256 (which is rather slow if computed purely in sw), that might be the reason. otoh, whether a secret key is involved (authenticated-blake2) or not (none) doesn't make a significant difference. |
|
Right, that makes sense. With borg2, one can use |
|
I did some more research using crate https://docs.rs/cpufeatures and now I see that my host has no hardware sha support. Rust's various sha256 implementations are slow on my host, I reported this here: RustCrypto/hashes#327 (comment) . So result " |
|
One idea to make |
|
I'm attempting to understand why Line 1247 in f43fcd3 But even if I remove LRU cache, borg is still sometimes faster than my code, so I'm continuing to investigate why it is faster |
|
@safinaskar sure, some sort of files, like disk image files can contain huge amounts of all-zero chunks (in the optimal case as sparse areas, but also as on-disk zeros) and the chunkers will often cut chunks of a few specific sizes, so not re-computing the same hash again and again saves a lot of time. IIRC, borg create has this optimisation but borg extract does not have any special optimisation for repeating (all-zero) chunks yet. another optimisation is that borg's |
|
I did more tests and found that POSIX_FADV_DONTNEED actually causes very big speedup in
Here is how I did my tests:
And this algorithm causes POSIX_FADV_DONTNEED to give x2.5 speedup. You may say that this is very artificial set of conditions. Well, no. Speedup appeared naturally. Here is what happened: at first I did very unscientific benchmarks and found that borg2 is faster than my dedupper. Then I tried to understand why it is so and found that one of the reasons is POSIX_FADV_DONTNEED, I reported this here: #7674 (comment) . Then I discovered that POSIX_FADV_DONTNEED not always gives speedup, so I tried to find exact set of conditions that cause it. And I found them. Note that if some of conditions don't hold, then we can get great slow down instead of speedup. For example, if available memory is bigger than all input data, then we get x1.3 slow down instead of x2.5 speedup |
Guess the 2nd will mostly neutralise the effect of the first. Although there might be a small difference:
About why borg is using POSIX_FADV_DONTNEED:
About the slowdown:
|
|
Thanks for answer!
There is some misunderstanding here. I first drop caches and then pre-cache input file. My tests look similar to this: #!/bin/bash
sync
echo 3 > /proc/sys/vm/drop_caches
for I in {00..11}; do
cat "$I" > /dev/null
time -p path-to/my-dedupper make "$I" ......
done |
|
Not sure why you do the Also, it is not a very realistic scenario. |
|
Again: I found that POSIX_FADV_DONTNEED is fast, and then I tried to write down exact scenario when it is fast. And it turned out that I need that |
|
I added more optimizations to my dedupper (in particular, it is parallel now). And now I did truly scientific benchmark. Now my dedupper is parallel thanks to great Rust library "rayon". (My dedupper is called azwyon now, this name was generated using Still, rayon lacked particular feature I need, so I implemented it: rayon-rs/rayon#1071 . Also, this new version of dedupper manages not only chunks, but also index (as opposed to previous version). Now let's proceed to experiment description. Now I do truly scientific benchmark using these advises: https://easyperf.net/blog/2019/08/02/Perf-measurement-environment-on-Linux , https://llvm.org/docs/Benchmarking.html . I use AWS bare metal server for benchmarks. Yes, my previous AWS experience was bad. I don't know why. Possibly this is because I did read data from networked disk. This time everything is on AWS bare metal server, in tmpfs. AWS has many threads, but I restricted all tools to 4 CPUs using I did run every tool in its best settings. For example, all single-threaded tools run in single-threaded mode, but all parallel-capable tools run in parallel mode. Yes, this is probably "unfair", but goal for my benchmark is pragmatic: choosing tool for my use case. So I simply run every tool in its "best" mode. Tools under test are:
All parallel capable tools run in parallel mode. These are: zstd, zpaq, azwyon, desync. Single-threaded: borg, casync. All tools (except for zpaq) use zstd level 3 compression. Comparison between borg and azwyon is "completely unfair", because:
Yes, I know this. I simply want to find what suits my case best. If you disagree, you may ask to re-run benchmark with different settings. Okay, so here is code for azwyon: https://paste.gg/p/anonymous/7842d4fc787a4b42bcc4ae02a6e0ecfa And here is code for benchmark: https://paste.gg/p/anonymous/2bac07a20d5f48dbabd4d9e3ad2c4ab1 And here are results: Exact command lines: // Initializing
match &method {
Method::ZStd => {},
Method::Borg { encryption, .. } => cmd!("BORG_PASSPHRASE=password borg2 rcreate --encryption={encryption} --repo {REPO}"),
Method::Casync { .. } => cmd!("mkdir {REPO}/index {REPO}/storage.castr"),
Method::Zpaq => {},
Method::Azwyon { .. } => cmd!("mkdir {REPO}/chunks {REPO}/index"),
Method::Desync { .. } => cmd!("mkdir {REPO}/index {REPO}/storage.castr"),
}
// //
match &method {
Method::ZStd => cmd!("zstd -3 -T0 < /home/user/dedup-bench/input-fs/{i:02} > {REPO}/{i:02}.zst"),
Method::Borg { chunker_params, compression, encryption: _ } => cmd!("cd /home/user/dedup-bench/input-fs; BORG_PASSPHRASE=password borg2 create --repo {REPO} --chunker-params {chunker_params} --compression {compression} {i:02} {i:02}"),
Method::Casync { chunker_params } => cmd!("casync make --compression=zstd --chunk-size={chunker_params} --store={REPO}/storage.castr {REPO}/index/{i:02}.caibx /home/user/dedup-bench/input-fs/{i:02} > /dev/null"),
Method::Zpaq => cmd!("cd /home/user/dedup-bench/input-fs; zpaq add {REPO}/repo.zpaq {i:02} > /dev/null"),
Method::Azwyon { chunk_size, zstd, digest, check_extracted: _ } => cmd!("/home/user/dedup-bench/azwyon-dedup/target/release/azwyon-dedup make --chunk-size={chunk_size} --store={REPO} --zstd={zstd} --id={i:02} --digest={digest} < /home/user/dedup-bench/input-fs/{i:02}"),
Method::Desync { chunker_params } => cmd!("/desync make -m {chunker_params} -s {REPO}/storage.castr {REPO}/index/{i:02}.caibx /home/user/dedup-bench/input-fs/{i:02}"),
}
// //
match method {
Method::ZStd => cmd!("zstd -d -T0 < {REPO}/{i:02}.zst > /home/user/dedup-bench/input-fs/data"),
Method::Borg { .. } => cmd!("cd /home/user/dedup-bench/input-fs; BORG_PASSPHRASE=password borg2 extract --repo {REPO} {i:02} {i:02}; mv -i {i:02} data"),
Method::Casync { .. } => cmd!("casync extract --store={REPO}/storage.castr {REPO}/index/{i:02}.caibx /home/user/dedup-bench/input-fs/data"),
Method::Zpaq => cmd!("cd /home/user/dedup-bench/input-fs; zpaq extract {REPO}/repo.zpaq {i:02} > /dev/null; mv -i {i:02} data"),
Method::Azwyon { chunk_size: _, zstd: _, ref digest, check_extracted } => cmd!("/home/user/dedup-bench/azwyon-dedup/target/release/azwyon-dedup extract --store={REPO} --id={i:02} --to=/home/user/dedup-bench/input-fs/data --digest={digest} --check-extracted={check_extracted:?}"),
Method::Desync { .. } => cmd!("/desync extract -s {REPO}/storage.castr {REPO}/index/{i:02}.caibx /home/user/dedup-bench/input-fs/data"),
}Output of Result: azwyon is fastest. :) On compression. azwyon compresses x2 faster than second place, i. e. zstd. Yes, azwyon compresses data using zstd level 3 faster than zstd level 3 itself. :) This is because azwyon doesn't compress same blocks twice. 3rd place is borg's. And azwyon is x4 faster than borg. On decompression. Well, azwyon is fastest again. Why azwyon is so fast? Well, this is not because of POSIX_FADV_DONTNEED. POSIX_FADV_DONTNEED matters for disks only, and this benchmark was on tmpfs. So, what are reasons? First of all, I do very few things. Second, I do compression/decompression in parallel. Third, I don't have any kind of database or locking. Repo maintains its consistency thanks to |
|
I added rdedup ( https://github.com/dpc/rdedup ). It is written in Rust, too. It is parallel. It uses content-defined chunking. I tested blake2b hash. Command lines: Method::RDedup { chunker_params, chunk_size, } => cmd!("rdedup --dir {REPO} init --chunking {chunker_params} --chunk-size {chunk_size} --compression-level 3 --encryption none --nesting 0"),
Method::RDedup { .. } => cmd!("rdedup --dir {REPO} store {i:02} < /home/user/dedup-bench/input-fs/{i:02}"),
Method::RDedup { .. } => cmd!("rdedup --dir {REPO} load {i:02} > /home/user/dedup-bench/input-fs/data"),My dedupper is still the best |
|
OK. Maybe we should get back to the general topic of borgbackup. :-) So, after all the research you did, did you find anything in borg missing or wrong (besides parallelism, for which we already have a ticket)? |
|
I have nothing to add to actionable items listed here: #7674 (comment) You said (on adding new hash algorithms): "the code structure in borg dealing with this does not scale well to a lot of algorithms". Unfortunately (if you really care about hash collisions) you should regularly migrate to new hash algorithms as described here: https://valerieaurora.org/hash.html . So, such infrastructure should be eventually added |
|
Also, as well as I understand, borg requires locking, so you cannot run two But in azwyon I was able to implement consistent parallel But azwyon doesn't support deleting of blobs. But duplicacy (don't confuse with duplicity and duplicati) supports lock-free parallel adding and deleting of blobs (as well as I understand). Here is how they achieved this: https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication . This can be very useful addition to borg. Also, rdedup supports asymmetric cryptography: https://github.com/dpc/rdedup/wiki/My-original-use-case-%28old-README.md%29 |
|
Last benchmark compares fixed sized chunking to CDC, this is not good. Also, CDC-based tools with different average chunk sizes compared to each other, this is not good, too. So here are new rules:
Here are results: I leaved zstd, because it has no concept of chunking. I leaved zpaq, because, well, I don't want to play with it. Zpaq has many settings, it is even possible it can beat other tools with right settings, but I don't want to spend my time on it |
|
Note for most CDC implementations the "average" avg-size setting is not really the average blocksize, it's the average extra length after the min-size. So the real average blocksize (ignoring truncation effects from max-size) is min-size+avg-size. In general you want to set max-size large enough to avoid truncation effects. For a given real target average blocksize the best de-duplication and speed settings are |
|
Okay, so here is list of Github issues I |
|
I again summarized my list of problems with existing deduppers: https://lobste.rs/s/0itosu/look_at_rapidcdc_quickcdc#c_ygqxsl |
|
@safinaskar That post has some issues and is not true as it is now. https://borgbackup.readthedocs.io/en/stable/usage/init.html?highlight=blake2 there it talks quite a bit about speed of hash algos, so it is documented and discoverable. Also "uses slow sha256" is not true on modern machines - with sha2 hw acceleration, sha256 is faster than blake2, maybe even faster than blake3 would be? The hash verification at extraction time is done for security reasons, you omitted that in your post. IIRC I also told you already why we did not add blake3 yet, one main reason being the dependency on rust and that the C stuff is still beta/experimental there and not on pypi IIRC. So, please fix that post. |
|
You gave link to borg1. Yes, I see phrase "which makes authenticated-blake2 faster than none" there. But docs for borg2 do not have such phrase ( https://borgbackup.readthedocs.io/en/2.0.0b6/usage/rcreate.html ). (I updated my post.)
I updated, too.
I think this is obvious.
Yes, but blake3 support is still absent. I don't say that borg is wrong in not supporting blake3. I just told in that post that borg doesn't support blake3 Also, I can give you invite on lobste.rs if you want |
|
@safinaskar borg2 has That made a lot of related docs superfluous and is easier for the users (e.g. sha2-support can be tricky: you might have a modern cpu with sha2, but run borg in a VM that does not pass through sha2). |
|
FTR, I did some analysis/tests of CDC and rollsums, and in particular FastCDC here; https://github.com/dbaarda/rollsum-chunking/blob/master/RESULTS.rst Important to note is that FastCDC has some problems, and is actually worse than a simple chunker. Specifically they claim 2 improvements; a better/faster match criteria, and "normalized chunking" giving a better size distribution that allows for more "cut point skipping" speedups without hurting deduplication. Neither of these are that great. First, they realised that the Gear rollsum only has entropy from the most recent bytes in the least-significant-bits, but they didn't realise that it shifts and accumulates the entropy towards the most significant bits. So their complicated zero-padded mask used for their Second, their complicated "normalized chunking" actually gives worse deduplication than a simple chunker with the same amount of cut-point-skipping speedups. Any sort of normalization makes chunk cut-points more dependent on where the previous cut-point was, which messes with re-synchronising the cut-points after a delta, which adversely affects deduplication. All their measured "improvements" were an artefact of the settings they chose producing a smaller average chunk size. A simple chunker with settings to give the same min and average chunk size is simpler/faster with better deduplication. It also turns out that a simple chunker performs really well with a large min-size (lots of cut-point-skipping speed improvement), and most people using simple chunkers are probably using settings that are sub-optimal for speed and deduplication. |
|
And there is not only big fat hardware. Here is the result with So yes, very different result. Chaha20 is very fast. Blake2b is much faster than hmac-sh256, because I don't have HW acceleration. I would love seeing any performance increase in computation time in any of these categories :) |
UPD 2023-06-26 14:30 UTC: this is useless benchmark, see #7674 (comment) . I did proper benchmark here: #7674 (comment) .
Hi. I did many speed comparisons. Here are results:
borg --chunker-params fixedhas x2.5 slower compression speed than my own trivial single-threaded Rust program for data deduplication on same settings/algorithms (compression ratio is same). Decompression for borg is x4.5 slower"Compression" above means "all steps needed for putting given file (VM image) to deduplicator's storage". "Decompression" means opposite action, i. e. extracting from such storage. And everywhere I say borg I mean borg2.
Now let me give details. I'm writing "something like docker, but for VMs" for Linux. And I need deduplicating storage for this, i. e. something like what borg and casync do. So I did benchmark of various deduplicating solutions (and wrote my own trivial single-threaded Rust program). Test data is sequence of 12 snapshots of same VM. The zeroth snapshot (numbered 00) is VM created after fresh Debian install. Snapshot 01 is the same machine after execution of command
echo deb http://deb.debian.org/debian sid main > /etc/apt/sources.list. Snapshot 02 is snapshot 01 after commandapt-get update && apt-get install -y debootstrap && debootstrap sid /debian http://deb.debian.org/debian && : > /OLD && : > /debian/NEW && echo systemctl switch-root /debian /bin/bash > /root/.bash_history && apt-get install -y systemd, etc.Of course, all this snapshots are very similar, and there are a lot of redundancy here.
Test was performed so: I created fresh deduplicating storage, for example, by
borg2 rcreate .... Then I stored snapshot 00 to it using command likeborg2 create .... Then 01, etc. Then I extracted them back one by one. I recorded time of each storing and extracting. And size of resulting storage.The programs under test are these:
zstd -4 -T0(this command uses all available cores)--chunker-params fixedand--chunker-params buzhashwas tested. borg is single-threaded, according to docs.caibxas part of its storage. I. e. raw chunks only counted from casync's point of view. But (for fair comparison with borg) I counted them as part of storage. casync uses same chunking algorithm as borg2, i. e. buzhash. As it seems from docs, casync is single-threadedborg2 --chunker-params fixed), then deduplicates them and compresses using zstd. Similarly to casync the program manages chunks storage, but doesn't manage index files (i. e. user should manage index files somehow on its own, similarly to casync). But for fair comparison with other solutions index file sizes was counted as part of storage. The program is absolutely trivial and does bare minimum. Also it cheats: it compares chunks using weak murmur3 hash instead of some cryptographic hash (this is possible reason for good speed). But this should not affect decompression speed, because there is no reason to compute hashes when decompressing. The program is single-threaded. Unfortunately, there is no analog of casync'scasync gc(garbage collector) command, so proper deletion of data is not supported, i. e. chunk storage is effectively append-only (of course, this feature can be added if needed)The systems for testing was these:
The same versions of software used for both tests (expect for zstd, i. e. "control group", I'm sorry about that). All software was installed from debian repos (except for my solution, of course).
Okay, so here are some particular facts from this experiment.
Let's compare casync's default chunker settings with exact same borg's chunker settings. Here is result (DO):
{"method":"Casync { chunker_params: \"16384:65536:262144\" }","size":2288245859,"compression":[25018278,14123602,20728147,75336765,74989789,71274134,75145956,73516900,73657489,71545543,72030643,71385677],"decompression":[12319323,9242255,11615280,24205058,27393626,27160242,30992227,32594951,32978333,33110100,32684700,32696767],"total_compression":718752927,"total_decompression":306992867} {"method":"Borg { chunker_params: \"buzhash,14,18,16,48\", compression: \"zstd,3\" }","size":2149378514,"compression":[17679646,8448045,14671946,21504361,23543654,20046573,25479210,24576887,25604774,23566982,24462718,22118519],"decompression":[12550832,12346506,13107262,49148793,48386676,45512108,49184202,50710626,53943566,54338980,56295807,54046982],"total_compression":251703322,"total_decompression":499572346}As well as I understand from casync's sources, casync uses 48 as its window size, so I set same window size for borg. So, everything is same, chunker is same, compression is same (level 3 is default for
casync make --compression=zstd). So, we (predictably) get nearly same repo size. But compression speed is way bigger for borg. And decompression speed is way bigger for casync.Now let's compare
borg2 --chunker-params=fixedwith my dedupper with same chunk size and same compression (AWS):{"method":"Borg { chunker_params: \"fixed,4194304\", compression: \"zstd,1\" }","size":3045630161,"compression":[10498009,5765099,11638767,11808097,11682813,9746850,14360024,12748675,13860854,12773828,13976675,13142882],"decompression":[8436011,8436552,8773607,34001769,34560878,34688943,35455363,35270733,35649059,35656812,35832813,35752490],"total_compression":142002579,"total_decompression":342515034} {"method":"MyChunker { block_bytes: 4194304, zstd: 1, hash: \"murmur3\" }","size":3045403863,"compression":[5198602,836222,5851959,7503258,5633901,3456217,6540237,4397998,4985617,3890258,4718666,3864603],"decompression":[2717125,2661899,3047367,6441092,6868591,6854122,7567294,7639675,7984931,7982271,8272383,8156871],"total_compression":56877544,"total_decompression":76193627} {"method":"MyChunker { block_bytes: 4194304, zstd: 1, hash: \"sha256\" }","size":3045797079,"compression":[10820321,6458880,11505570,35665701,33806750,31593533,34644788,32626207,33124389,32051083,32847354,32049456],"decompression":[2677887,2655985,3093219,6251224,6857964,6813882,7581230,7604525,7907221,7911569,8100640,8103418],"total_compression":327194037,"total_decompression":75558768}So, everything is same. And predictably we got nearly same compressed size. But my dedupper is way faster than borg both in compression and decompression (if we cheat using murmur3 for chunk comparision). Compression is x2.5 faster. And decompression is x4.5 faster. I don't compute hashes during decompression (I think there is no reason to do so), so even in sha256 mode decompression is still x4.5 faster than borg. But with sha256 compression becomes x2.3 slower than borg (it seems I chose slow sha256 implementation from crates.io or maybe the reason is
overflow-checks = true).Both casync and borg can achieve compression level similar to zpaq's default settings (if properly configured). But in such case casync and borg become better in compression speed or in decompression speed (DO):
{"method":"Zpaq","size":2020550898,"compression":[66510942,13447238,47685405,80578869,80038181,59823872,80654039,65066410,71060346,61404565,64172466,61875377],"decompression":[18750110,20467299,29366756,36280852,41599404,41990291,48276667,51680639,54273853,54363762,55445757,55403322],"total_compression":752317716,"total_decompression":507898716} {"method":"Borg { chunker_params: \"buzhash,14,18,16,4095\", compression: \"zstd,4\" }","size":2145319615,"compression":[20378384,8886346,16710992,23303757,24969724,20268377,26531417,25136239,26505187,24139859,25912861,22098249],"decompression":[12205145,12290434,12885718,43323609,44173501,43728450,44725463,46298867,46943885,46165601,48665056,48574784],"total_compression":264841397,"total_decompression":449980518} {"method":"Casync { chunker_params: \"16384:65536:262144\" }","size":2288245859,"compression":[25018278,14123602,20728147,75336765,74989789,71274134,75145956,73516900,73657489,71545543,72030643,71385677],"decompression":[12319323,9242255,11615280,24205058,27393626,27160242,30992227,32594951,32978333,33110100,32684700,32696767],"total_compression":718752927,"total_decompression":306992867}Conclusions:
If I had blog, I would post everything there. But I don't have a blog, so I post everything here, to borg bug tracker. I hope this report will be useful to borg devs. If you want to write me, but don't want to pollute borg bug tracker, just write me directly to [email protected] .
Okay, now let me show you code and raw data.
Code for my dedupper:
Cargo.toml
src/main.rs
Now program for executing benchmark
Cargo.toml
src/main.rs
Now snapshots. All snapshots are raw GPT disk images with single ext4 partition. Snapshot 00 is 2 GiB fresh install of debian sid from https://snapshot.debian.org/archive/debian/20220917T224346Z . Next snapshots are produced by executing the following sequence of commands:
I. e. snapshot 04 is snapshot 03 plus
apt-get install -y git build-essential. These snapshots naturally appeared when I attempted to reproduce particular systemd bug, so data is 100% realistic.The snapshot contain nothing confidential, so if you want, I can send them to you, just write me.
Now full benchmark results. (JSON is in the end.)
DO: https://paste.gg/p/anonymous/063883027e9848d8b7f53a046c4bd3cf
AWS: https://paste.gg/p/anonymous/bd6d25d5906f4c0fb94b78af605fb475
(Unfortunately, I used
dufor calculating storage size, but, as well as I remember,duutils from DO and AWS seem to produce slightly different results, i. e. these twodu's use different algorithms)Assume all this code as public domain. If you want me to release any of my mentioned software as proper free software project (say, to crates.io), just write me.
Exact commands for invoking borg and other programs under test:
Is this a BUG / ISSUE report or a QUESTION?
I think borg is slower than needed. Possible bug
The text was updated successfully, but these errors were encountered: