Implement Erasure Coding

[gilbertchen]

A new option -erasure-coding is added to the init and add commands to provide the ability to recover from corrupted chunks.

Example use:

duplicacy init -erasure-coding 5:2 repository_id storage_url

When restoring from a storage with erasure coding enabled, the following log messages show corrupted chunks are being repaired:

  Restoring /private/tmp/duplicacy_test/repository to revision 1
  Recovering a 1824550 byte chunk from 364910 byte shards: ***--**
  Downloaded chunk 1 size 1817347, 1.73MB/s 00:00:11 9.0%
  Recovering a 6617382 byte chunk from 1323477 byte shards: **--***
  Downloaded chunk 2 size 6591322, 8.02MB/s 00:00:02 42.0%
  Recovering a 5136934 byte chunk from 1027387 byte shards: --*****
  Downloaded chunk 3 size 5116593, 12.90MB/s 00:00:01 67.6%
  Recovering a 2515494 byte chunk from 503099 byte shards: -*****-
  Downloaded chunk 4 size 2505558, 15.29MB/s 00:00:01 80.1%
  Recovering a 3984934 byte chunk from 796987 byte shards: --*****
  Downloaded chunk 5 size 3969180, 19.07MB/s 00:00:01 100.0%
  Downloaded file1 (20000000)

[gilbertchen]

This pull request has been mentioned on Duplicacy Forum. There might be relevant details there:

https://forum.duplicacy.com/t/new-feature-erasure-coding/4168/1

[TheBestPessimist]

About naming:

From https://forum.duplicacy.com/t/new-feature-erasure-coding/4168/2?u=thebestpessimist i understand that this only protects the same chunk from corruption, and not the whole storage (ie. like raid5/6 does).

I think the Usage should be reworded to something like enable erasure coding to protect against intra-chunk corruption, as this is not "storage level protection", just "chunk-level protection".

What would happen if in the future you would also add something like storage-level protection via something similar to raid5/6, which would do parity for any chunk?

[TheBestPessimist]

I could be missunderstanding stuff but
RSA encrypted chunks start with "duplicacy\002"
ERASURE_CODING_BANNER = "duplicacy\003"
does this mean that we cannot have rsa and erasure at the same time?

[TheBestPessimist]

If we can have all of those, then the flag ENCRYPTION_BANNER could be better renamed to FEATURE_LEVEL.