backhand: Remove duplicate data

[wcampbell0x2a]

• Add cache to DataWriter, that if enabled will remember the file length and a hash of a previous file and use that instead of writing the matching file to disk.
• Add set_no_duplicate_files to FilesystemWriter to make this configurable.
• Make superblock Flags public

TODO

• Maybe hash the first 32 bytes to do a quick comparison, before hashing the entire file
• cleanup code quality
• Add test that adds a file that already exists, should be the same size? (might want to check that just_copy_it, works)
• Use https://github.com/paritytech/nohash-hasher (some other hashes I use could use this!)

[github-actions]

Benchmark for d58f903

Click to view benchmark

Test	Base	PR	%
only_read/netgear_ax6100v2	2.4±0.04ms	2.4±0.00ms	0.00%
only_read/tplink_ax1800	6.2±0.01ms	6.2±0.01ms	0.00%
unsquashfs/full	11.1±0.16ms	11.0±0.19ms	-0.90%
unsquashfs/full-path-filter	7.7±0.10ms	7.7±0.10ms	0.00%
unsquashfs/list	8.1±0.13ms	8.1±0.15ms	0.00%
unsquashfs/list-path-filter	7.2±0.13ms	7.2±0.10ms	0.00%
write_read/netgear_ax6100v2	1287.8±7.45ms	1291.3±10.50ms	+0.27%
write_read/tplink_ax1800	7.1±0.04s	7.2±0.05s	+1.41%

[github-actions]

Benchmark for 866e857

Click to view benchmark

Test	Base	PR	%
only_read/netgear_ax6100v2	2.4±0.00ms	2.3±0.00ms	-4.17%
only_read/tplink_ax1800	6.3±0.01ms	6.2±0.01ms	-1.59%
unsquashfs/full	10.9±0.13ms	10.9±0.50ms	0.00%
unsquashfs/full-path-filter	7.6±0.06ms	7.6±0.05ms	0.00%
unsquashfs/list	7.9±0.07ms	7.9±0.09ms	0.00%
unsquashfs/list-path-filter	7.1±0.04ms	7.0±0.04ms	-1.41%
write_read/netgear_ax6100v2	1277.2±3.77ms	1282.6±5.90ms	+0.42%
write_read/tplink_ax1800	7.0±0.01s	7.0±0.01s	0.00%

[github-actions]

Benchmark for f9cb352

Click to view benchmark

Test	Base	PR	%
only_read/netgear_ax6100v2	2.4±0.00ms	2.3±0.00ms	-4.17%
only_read/tplink_ax1800	6.2±0.00ms	6.2±0.01ms	0.00%
unsquashfs/full	10.9±0.13ms	10.9±0.16ms	0.00%
unsquashfs/full-path-filter	7.6±0.06ms	7.6±0.06ms	0.00%
unsquashfs/list	8.2±0.37ms	7.9±0.07ms	-3.66%
unsquashfs/list-path-filter	7.1±0.10ms	7.0±0.08ms	-1.41%
write_read/netgear_ax6100v2	1276.1±5.81ms	1275.0±2.06ms	-0.09%
write_read/tplink_ax1800	7.2±0.01s	7.1±0.04s	-1.39%

[github-actions]

Benchmark for d1c6030

Click to view benchmark

Test	Base	PR	%
only_read/netgear_ax6100v2	2.3±0.00ms	2.3±0.00ms	0.00%
only_read/tplink_ax1800	6.2±0.05ms	6.2±0.01ms	0.00%
unsquashfs/full	11.0±0.25ms	11.0±0.17ms	0.00%
unsquashfs/full-path-filter	7.7±0.06ms	7.6±0.06ms	-1.30%
unsquashfs/list	8.0±0.07ms	8.0±0.13ms	0.00%
unsquashfs/list-path-filter	7.1±0.07ms	7.2±0.13ms	+1.41%
write_read/netgear_ax6100v2	1271.4±1.29ms	1275.0±5.79ms	+0.28%
write_read/tplink_ax1800	7.0±0.01s	7.1±0.03s	+1.43%