diff --git a/docs/faq.rst b/docs/faq.rst index d982aec2c9..b3c3d73171 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -89,7 +89,7 @@ Also, you must not run borg against multiple instances of the same repo (which is an issue if they happen to be not the same). See :issue:`4272` for an example. - Encryption security issues if you would update repo and copy-of-repo - independently, due to AES counter reuse. + independently, due to AES counter reuse (when using legacy encryption modes). See also: :ref:`faq_corrupt_repo` @@ -246,6 +246,8 @@ then use ``tar`` to perform the comparison: My repository is corrupt, how can I restore from an older copy of it? --------------------------------------------------------------------- +Note: this is only required for repos using legacy encryption modes. + If your repositories are encrypted and have the same ID, the recommended method is to delete the corrupted repository, but keep its security info, and then copy the working repository to the same location: @@ -473,8 +475,11 @@ Security .. _borg_security_critique: -Isn't BorgBackup's AES-CTR crypto broken? ------------------------------------------ +Isn't BorgBackup's legacy AES-CTR-based crypto broken? +------------------------------------------------------ + +Note: in borg 1.3 new AEAD cipher based modes with session keys were added, +solving the issues of the legacy modes. If a nonce (counter) value is reused, AES-CTR mode crypto is broken. @@ -713,6 +718,8 @@ Please disclose security issues responsibly. How important are the nonce files? ------------------------------------ +This only applies to repositories using legacy encryption modes. + Borg uses :ref:`AES-CTR encryption `. An essential part of AES-CTR is a sequential counter that must **never** repeat. If the same value of the counter is used twice in the same repository, @@ -881,14 +888,14 @@ What's the expected backup performance? --------------------------------------- Compared to simply copying files (e.g. with ``rsync``), Borg has more work to do. -This can make creation of the first archive slower, but saves time +This can make creation of the first archive slower, but saves time and disk space on subsequent runs. Here what Borg does when you run ``borg create``: - Borg chunks the file (using the relatively expensive buzhash algorithm) -- It then computes the "id" of the chunk (hmac-sha256 (often slow, except +- It then computes the "id" of the chunk (hmac-sha256 (often slow, except if your CPU has sha256 acceleration) or blake2b (fast, in software)) -- Then it checks whether this chunk is already in the repo (local hashtable lookup, - fast). If so, the processing of the chunk is completed here. Otherwise it needs to +- Then it checks whether this chunk is already in the repo (local hashtable lookup, + fast). If so, the processing of the chunk is completed here. Otherwise it needs to process the chunk: - Compresses (the default lz4 is super fast) - Encrypts (AES, usually fast if your CPU has AES acceleration as usual @@ -896,9 +903,9 @@ and disk space on subsequent runs. Here what Borg does when you run ``borg creat - Authenticates ("signs") using hmac-sha256 or blake2b (see above), - Transmits to repo. If the repo is remote, this usually involves an SSH connection (does its own encryption / authentication). -- Stores the chunk into a key/value store (the key is the chunk id, the value +- Stores the chunk into a key/value store (the key is the chunk id, the value is the data). While doing that, it computes a CRC32 of the data (repo low-level - checksum, used by borg check --repository) and also updates the repo index + checksum, used by borg check --repository) and also updates the repo index (another hashtable). Subsequent backups are usually very fast if most files are unchanged and only @@ -928,14 +935,14 @@ If you feel your Borg backup is too slow somehow, here is what you can do: - Make sure Borg has enough RAM (depends on how big your repo is / how many files you have) -- Use one of the blake2 modes for --encryption except if you positively know +- Use one of the blake2 modes for --encryption except if you positively know your CPU (and openssl) accelerates sha256 (then stay with hmac-sha256). - Don't use any expensive compression. The default is lz4 and super fast. Uncompressed is often slower than lz4. - Just wait. You can also interrupt it and start it again as often as you like, it will converge against a valid "completed" state (see ``--checkpoint-interval``, maybe use the default, but in any case don't make it too short). It is starting - from the beginning each time, but it is still faster then as it does not store + from the beginning each time, but it is still faster then as it does not store data into the repo which it already has there from last checkpoint. - If you don’t need additional file attributes, you can disable them with ``--noflags``, ``--noacls``, ``--noxattrs``. This can lead to noticable performance improvements @@ -945,12 +952,12 @@ If you feel that Borg "freezes" on a file, it could be in the middle of processi large file (like ISOs or VM images). Borg < 1.2 announces file names *after* finishing with the file. This can lead to displaying the name of a small file, while processing the next (larger) file. For very big files this can lead to the progress display show some -previous short file for a long time while it processes the big one. With Borg 1.2 this +previous short file for a long time while it processes the big one. With Borg 1.2 this was changed to announcing the filename before starting to process it. To see what files have changed and take more time processing, you can also add -``--list --filter=AME --stats`` to your ``borg create`` call to produce more log output, -including a file list (with file status characters) and also some statistics at +``--list --filter=AME --stats`` to your ``borg create`` call to produce more log output, +including a file list (with file status characters) and also some statistics at the end of the backup. Then you do the backup and look at the log output: diff --git a/docs/internals/data-structures.rst b/docs/internals/data-structures.rst index 9e1c067f24..3d70e7807d 100644 --- a/docs/internals/data-structures.rst +++ b/docs/internals/data-structures.rst @@ -865,6 +865,31 @@ Encryption .. seealso:: The :ref:`borgcrypto` section for an in-depth review. +AEAD modes +~~~~~~~~~~ + +Uses modern AEAD ciphers: AES-OCB or CHACHA20-POLY1305. +For each borg invocation, a new sessionkey is derived from the borg key material +and the 48bit IV starts from 0 again (both ciphers internally add a 32bit counter +to our IV, so we'll just count up by 1 per chunk). + +The chunk layout is best seen at the bottom of this diagram: + +.. figure:: encryption-aead.png + :figwidth: 100% + :width: 100% + +No special IV/counter management is needed here due to the use of session keys. + +A 48 bit IV is way more than needed: If you only backed up 4kiB chunks (2^12B), +the IV would "limit" the data encrypted in one session to 2^(12+48)B == 2.3 exabytes, +meaning you would run against other limitations (RAM, storage, time) way before that. +In practice, chunks are usually bigger, for big files even much bigger, giving an +even higher limit. + +Legacy modes +~~~~~~~~~~~~ + AES_-256 is used in CTR mode (so no need for padding). A 64 bit initialization vector is used, a MAC is computed on the encrypted chunk and both are stored in the chunk. Encryption and MAC use two different keys. @@ -884,6 +909,9 @@ To reduce payload size, only 8 bytes of the 16 bytes nonce is saved in the payload, the first 8 bytes are always zeros. This does not affect security but limits the maximum repository capacity to only 295 exabytes (2**64 * 16 bytes). +Both modes +~~~~~~~~~~ + Encryption keys (and other secrets) are kept either in a key file on the client ('keyfile' mode) or in the repository config on the server ('repokey' mode). In both cases, the secrets are generated from random and then encrypted by a diff --git a/docs/internals/encryption-aead.odg b/docs/internals/encryption-aead.odg new file mode 100644 index 0000000000..0f74fb428d Binary files /dev/null and b/docs/internals/encryption-aead.odg differ diff --git a/docs/internals/encryption-aead.png b/docs/internals/encryption-aead.png new file mode 100644 index 0000000000..b9eb2339b2 Binary files /dev/null and b/docs/internals/encryption-aead.png differ diff --git a/docs/internals/security.rst b/docs/internals/security.rst index 1b9764e1b9..84f443f320 100644 --- a/docs/internals/security.rst +++ b/docs/internals/security.rst @@ -124,7 +124,88 @@ prompt is a set BORG_PASSPHRASE. See issue :issue:`2169` for details. Encryption ---------- -Encryption is currently based on the Encrypt-then-MAC construction, +AEAD modes +~~~~~~~~~~ + +Modes: --encryption (repokey|keyfile)-[blake2-](aes-ocb|chacha20-poly1305) + +Supported: borg 1.3+ + +Encryption with these modes is based on AEAD ciphers (authenticated encryption +with associated data) and session keys. + +Depending on the chosen mode (see :ref:`borg_init`) different AEAD ciphers are used: + +- AES-256-OCB - super fast, single-pass algorithm IF you have hw accelerated AES. +- chacha20-poly1305 - very fast, purely software based AEAD cipher. + +The chunk ID is derived via a MAC over the plaintext (mac key taken from borg key): + +- HMAC-SHA256 - super fast IF you have hw accelerated SHA256. +- Blake2b - very fast, purely software based algorithm. + +For each borg invocation, a new session id is generated by `os.urandom`_. + +From that session id, the initial key material (ikm, taken from the borg key) +and an application and cipher specific salt, borg derives a session key via HKDF. + +For each session key, IVs (nonces) are generated by a counter which increments for +each encrypted message. + +Session:: + + sessionid = os.urandom(24) + ikm = enc_key || enc_hmac_key + salt = "borg-session-key-CIPHERNAME" + sessionkey = HKDF(ikm, sessionid, salt) + message_iv = 0 + +Encryption:: + + id = MAC(id_key, data) + compressed = compress(data) + + header = type-byte || 00h || message_iv || sessionid + aad = id || header + message_iv++ + encrypted, auth_tag = AEAD_encrypt(session_key, message_iv, compressed, aad) + authenticated = header || auth_tag || encrypted + +Decryption:: + + # Given: input *authenticated* data and a *chunk-id* to assert + type-byte, past_message_iv, past_sessionid, auth_tag, encrypted = SPLIT(authenticated) + + ASSERT(type-byte is correct) + + past_key = HKDF(ikm, past_sessionid, salt) + decrypted = AEAD_decrypt(past_key, past_message_iv, authenticated) + + decompressed = decompress(decrypted) + + ASSERT( CONSTANT-TIME-COMPARISON( chunk-id, MAC(id_key, decompressed) ) ) + +Notable: + +- More modern and often faster AEAD ciphers instead of self-assembled stuff. +- Due to the usage of session keys, IVs (nonces) do not need special care here as + they did for the legacy encryption modes. +- The id is now also input into the authentication tag computation. + This strongly associates the id with the written data (== associates the key with + the value). When later reading the data for some id, authentication will only + succeed if what we get was really written by us for that id. + + +Legacy modes +~~~~~~~~~~~~ + +Modes: --encryption (repokey|keyfile)-[blake2] + +Supported: all borg versions, blake2 since 1.1 + +DEPRECATED. We strongly suggest you use the safer AEAD modes, see above. + +Encryption with these modes is based on the Encrypt-then-MAC construction, which is generally seen as the most robust way to create an authenticated encryption scheme from encryption and message authentication primitives. @@ -137,7 +218,7 @@ in the future. Depending on the chosen mode (see :ref:`borg_init`) different primitives are used: -- The actual encryption is currently always AES-256 in CTR mode. The +- Legacy encryption modes use AES-256 in CTR mode. The counter is added in plaintext, since it is needed for decryption, and is also tracked locally on the client to avoid counter reuse. @@ -253,7 +334,7 @@ Implementations used We do not implement cryptographic primitives ourselves, but rely on widely used libraries providing them: -- AES-CTR and HMAC-SHA-256 from OpenSSL 1.0 / 1.1 are used, +- AES-CTR, AES-OCB, CHACHA20-POLY1305 and HMAC-SHA-256 from OpenSSL 1.1 are used, which is also linked into the static binaries we provide. We think this is not an additional risk, since we don't ever use OpenSSL's networking, TLS or X.509 code, but only their @@ -268,7 +349,8 @@ on widely used libraries providing them: Implemented cryptographic constructions are: -- Encrypt-then-MAC based on AES-256-CTR and either HMAC-SHA-256 +- AEAD modes: AES-OCB and CHACHA20-POLY1305 are straight from OpenSSL. +- Legacy modes: Encrypt-then-MAC based on AES-256-CTR and either HMAC-SHA-256 or keyed BLAKE2b256 as described above under Encryption_. - Encrypt-and-MAC based on AES-256-CTR and HMAC-SHA-256 as described above under `Offline key security`_. diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 52fad66392..dd9d45eaae 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -387,6 +387,7 @@ For automated backups the passphrase can be specified using the A backup inside of the backup that is encrypted with that key/passphrase won't help you with that, of course. + Only applies to repos using legacy encryption modes: In case you lose your repository and the security information, but have an older copy of it to restore from, don't use that later for creating new backups – you would run into security issues (reuse of nonce counter diff --git a/docs/usage/init.rst b/docs/usage/init.rst index ae0fd8b423..73e62bb79f 100644 --- a/docs/usage/init.rst +++ b/docs/usage/init.rst @@ -4,16 +4,19 @@ Examples ~~~~~~~~ :: - # Local repository, repokey encryption, BLAKE2b (often faster, since Borg 1.1) - $ borg init --encryption=repokey-blake2 /path/to/repo + # Local repository, recommended repokey AEAD crypto modes + $ borg init --encryption=repokey-aes-ocb /path/to/repo + $ borg init --encryption=repokey-chacha20-poly1305 /path/to/repo + $ borg init --encryption=repokey-blake2-aes-ocb /path/to/repo + $ borg init --encryption=repokey-blake2-chacha20-poly1305 /path/to/repo - # Local repository (no encryption) + # Local repository (no encryption), not recommended $ borg init --encryption=none /path/to/repo # Remote repository (accesses a remote borg via ssh) # repokey: stores the (encrypted) key into /config - $ borg init --encryption=repokey-blake2 user@hostname:backup + $ borg init --encryption=repokey-aes-ocb user@hostname:backup # Remote repository (accesses a remote borg via ssh) # keyfile: stores the (encrypted) key into ~/.config/borg/keys/ - $ borg init --encryption=keyfile user@hostname:backup + $ borg init --encryption=keyfile-aes-ocb user@hostname:backup diff --git a/src/borg/archive.py b/src/borg/archive.py index 728c76bb10..8336166615 100644 --- a/src/borg/archive.py +++ b/src/borg/archive.py @@ -1789,7 +1789,7 @@ def mark_as_possibly_superseded(id_): def add_callback(chunk): id_ = self.key.id_hash(chunk) - cdata = self.key.encrypt(chunk) + cdata = self.key.encrypt(id_, chunk) add_reference(id_, len(chunk), len(cdata), cdata) return id_ @@ -1811,7 +1811,7 @@ def verify_file_chunks(archive_name, item): def replacement_chunk(size): chunk = Chunk(None, allocation=CH_ALLOC, size=size) chunk_id, data = cached_hash(chunk, self.key.id_hash) - cdata = self.key.encrypt(data) + cdata = self.key.encrypt(chunk_id, data) csize = len(cdata) return chunk_id, size, csize, cdata @@ -1998,7 +1998,7 @@ def valid_item(obj): archive.items = items_buffer.chunks data = msgpack.packb(archive.as_dict()) new_archive_id = self.key.id_hash(data) - cdata = self.key.encrypt(data) + cdata = self.key.encrypt(new_archive_id, data) add_reference(new_archive_id, len(data), len(cdata), cdata) self.manifest.archives[info.name] = (new_archive_id, info.ts) pi.finish() diff --git a/src/borg/archiver.py b/src/borg/archiver.py index b6923495ef..9be09948ee 100644 --- a/src/borg/archiver.py +++ b/src/borg/archiver.py @@ -604,9 +604,9 @@ def chunkit(chunker_name, *args, **kwargs): if not is_libressl: tests.extend([ ("aes-256-ocb", lambda: AES256_OCB( - None, key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')), + key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')), ("chacha20-poly1305", lambda: CHACHA20_POLY1305( - None, key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')), + key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')), ]) for spec, func in tests: print(f"{spec:<24} {size:<10} {timeit(func, number=100):.3f}s") @@ -4169,22 +4169,19 @@ def define_borg_mount(parser): Encryption mode TLDR ++++++++++++++++++++ - The encryption mode can only be configured when creating a new repository - - you can neither configure it on a per-archive basis nor change the - encryption mode of an existing repository. + The encryption mode can only be configured when creating a new repository - you can + neither configure it on a per-archive basis nor change the mode of an existing repository. + This example will likely NOT give optimum performance on your machine (performance + tips will come below): - Use ``repokey``:: + :: borg init --encryption repokey /path/to/repo - Or ``repokey-blake2`` depending on which is faster on your client machines (see below):: - - borg init --encryption repokey-blake2 /path/to/repo - Borg will: 1. Ask you to come up with a passphrase. - 2. Create a borg key (which contains 3 random secrets. See :ref:`key_files`). + 2. Create a borg key (which contains some random secrets. See :ref:`key_files`). 3. Encrypt the key with your passphrase. 4. Store the encrypted borg key inside the repository directory (in the repo config). This is why it is essential to use a secure passphrase. @@ -4220,79 +4217,53 @@ def define_borg_mount(parser): You can change your passphrase for existing repos at any time, it won't affect the encryption/decryption key or other secrets. - More encryption modes - +++++++++++++++++++++ + Choosing an encryption mode + +++++++++++++++++++++++++++ - Only use ``--encryption none`` if you are OK with anyone who has access to - your repository being able to read your backups and tamper with their - contents without you noticing. + Depending on your hardware, hashing and crypto performance may vary widely. + The easiest way to find out about what's fastest is to run ``borg benchmark cpu``. - If you want "passphrase and having-the-key" security, use ``--encryption keyfile``. - The key will be stored in your home directory (in ``~/.config/borg/keys``). + `repokey` modes: if you want ease-of-use and "passphrase" security is good enough - + the key will be stored in the repository (in ``repo_dir/config``). - If you do **not** want to encrypt the contents of your backups, but still - want to detect malicious tampering use ``--encryption authenticated``. + `keyfile` modes: if you rather want "passphrase and having-the-key" security - + the key will be stored in your home directory (in ``~/.config/borg/keys``). - If ``BLAKE2b`` is faster than ``SHA-256`` on your hardware, use ``--encryption authenticated-blake2``, - ``--encryption repokey-blake2`` or ``--encryption keyfile-blake2``. Note: for remote backups - the hashing is done on your local machine. + The following table is roughly sorted in order of preference, the better ones are + in the upper part of the table, in the lower part is the old and/or unsafe(r) stuff: .. nanorst: inline-fill - +----------+---------------+------------------------+--------------------------+ - | Hash/MAC | Not encrypted | Not encrypted, | Encrypted (AEAD w/ AES) | - | | no auth | but authenticated | and authenticated | - +----------+---------------+------------------------+--------------------------+ - | SHA-256 | none | `authenticated` | repokey | - | | | | keyfile | - +----------+---------------+------------------------+--------------------------+ - | BLAKE2b | n/a | `authenticated-blake2` | `repokey-blake2` | - | | | | `keyfile-blake2` | - +----------+---------------+------------------------+--------------------------+ + +---------------------------------+---------------+---------------+------------------+-------+ + |**mode (* = keyfile or repokey)**|**ID-Hash** |**Encryption** |**Authentication**|**V>=**| + +---------------------------------+---------------+---------------+------------------+-------+ + | ``*-blake2-chacha20-poly1305`` | BLAKE2b | CHACHA20 | POLY1305 | 1.3 | + +---------------------------------+---------------+---------------+------------------+-------+ + | ``*-chacha20-poly1305`` | HMAC-SHA-256 | CHACHA20 | POLY1305 | 1.3 | + +---------------------------------+---------------+---------------+------------------+-------+ + | ``*-blake2-aes-ocb`` | BLAKE2b | AES256-OCB | AES256-OCB | 1.3 | + +---------------------------------+---------------+---------------+------------------+-------+ + | ``*-aes-ocb`` | HMAC-SHA-256 | AES256-OCB | AES256-OCB | 1.3 | + +---------------------------------+---------------+---------------+------------------+-------+ + | ``*-blake2`` | BLAKE2b | AES256-CTR | BLAKE2b | 1.1 | + +---------------------------------+---------------+---------------+------------------+-------+ + | ``*`` | HMAC-SHA-256 | AES256-CTR | HMAC-SHA256 | any | + +---------------------------------+---------------+---------------+------------------+-------+ + | authenticated-blake2 | BLAKE2b | none | BLAKE2b | 1.1 | + +---------------------------------+---------------+---------------+------------------+-------+ + | authenticated | HMAC-SHA-256 | none | HMAC-SHA256 | 1.1 | + +---------------------------------+---------------+---------------+------------------+-------+ + | none | SHA-256 | none | none | any | + +---------------------------------+---------------+---------------+------------------+-------+ .. nanorst: inline-replace - Modes `marked like this` in the above table are new in Borg 1.1 and are not - backwards-compatible with Borg 1.0.x. - - On modern Intel/AMD CPUs (except very cheap ones), AES is usually - hardware-accelerated. - BLAKE2b is faster than SHA256 on Intel/AMD 64-bit CPUs - (except AMD Ryzen and future CPUs with SHA extensions), - which makes `authenticated-blake2` faster than `none` and `authenticated`. - - On modern ARM CPUs, NEON provides hardware acceleration for SHA256 making it faster - than BLAKE2b-256 there. NEON accelerates AES as well. - - Hardware acceleration is always used automatically when available. - - `repokey` and `keyfile` use AES-CTR-256 for encryption and HMAC-SHA256 for - authentication in an encrypt-then-MAC (EtM) construction. The chunk ID hash - is HMAC-SHA256 as well (with a separate key). - These modes are compatible with Borg 1.0.x. - - `repokey-blake2` and `keyfile-blake2` are also authenticated encryption modes, - but use BLAKE2b-256 instead of HMAC-SHA256 for authentication. The chunk ID - hash is a keyed BLAKE2b-256 hash. - These modes are new and *not* compatible with Borg 1.0.x. - - `authenticated` mode uses no encryption, but authenticates repository contents - through the same HMAC-SHA256 hash as the `repokey` and `keyfile` modes (it uses it - as the chunk ID hash). The key is stored like `repokey`. - This mode is new and *not* compatible with Borg 1.0.x. - - `authenticated-blake2` is like `authenticated`, but uses the keyed BLAKE2b-256 hash - from the other blake2 modes. - This mode is new and *not* compatible with Borg 1.0.x. - - `none` mode uses no encryption and no authentication. It uses SHA256 as chunk - ID hash. This mode is not recommended, you should rather consider using an authenticated - or authenticated/encrypted mode. This mode has possible denial-of-service issues - when running ``borg create`` on contents controlled by an attacker. - Use it only for new repositories where no encryption is wanted **and** when compatibility - with 1.0.x is important. If compatibility with 1.0.x is not important, use - `authenticated-blake2` or `authenticated` instead. - This mode is compatible with Borg 1.0.x. + `none` mode uses no encryption and no authentication. You're advised to NOT use this mode + as it would expose you to all sorts of issues (DoS, confidentiality, tampering, ...) in + case of malicious activity in the repository. + + If you do **not** want to encrypt the contents of your backups, but still want to detect + malicious tampering use an `authenticated` mode. It's like `repokey` minus encryption. """) subparser = subparsers.add_parser('init', parents=[common_parser], add_help=False, description=self.do_init.__doc__, epilog=init_epilog, diff --git a/src/borg/cache.py b/src/borg/cache.py index 92df82e118..4e6de052aa 100644 --- a/src/borg/cache.py +++ b/src/borg/cache.py @@ -942,7 +942,7 @@ def add_chunk(self, id, chunk, stats, overwrite=False, wait=True): refcount = self.seen_chunk(id, size) if refcount and not overwrite: return self.chunk_incref(id, stats) - data = self.key.encrypt(chunk) + data = self.key.encrypt(id, chunk) csize = len(data) self.repository.put(id, data, wait=wait) self.chunks.add(id, 1, size, csize) @@ -1107,7 +1107,7 @@ def add_chunk(self, id, chunk, stats, overwrite=False, wait=True): refcount = self.seen_chunk(id, size) if refcount: return self.chunk_incref(id, stats, size=size) - data = self.key.encrypt(chunk) + data = self.key.encrypt(id, chunk) csize = len(data) self.repository.put(id, data, wait=wait) self.chunks.add(id, 1, size, csize) diff --git a/src/borg/constants.py b/src/borg/constants.py index 6b9544a311..5fb6b9a65a 100644 --- a/src/borg/constants.py +++ b/src/borg/constants.py @@ -111,6 +111,8 @@ class KeyBlobStorage: class KeyType: + # legacy crypto + # upper 4 bits are ciphersuite, 0 == legacy AES-CTR KEYFILE = 0x00 # repos with PASSPHRASE mode could not be created any more since borg 1.0, see #97. # in borg 1.3 all of its code and also the "borg key migrate-to-repokey" command was removed. @@ -123,6 +125,16 @@ class KeyType: BLAKE2REPO = 0x05 BLAKE2AUTHENTICATED = 0x06 AUTHENTICATED = 0x07 + # new crypto + # upper 4 bits are ciphersuite, lower 4 bits are keytype + AESOCBKEYFILE = 0x10 + AESOCBREPO = 0x11 + CHPOKEYFILE = 0x20 + CHPOREPO = 0x21 + BLAKE2AESOCBKEYFILE = 0x30 + BLAKE2AESOCBREPO = 0x31 + BLAKE2CHPOKEYFILE = 0x40 + BLAKE2CHPOREPO = 0x41 REPOSITORY_README = """This is a Borg Backup repository. diff --git a/src/borg/crypto/key.py b/src/borg/crypto/key.py index 3747995718..3691051b0a 100644 --- a/src/borg/crypto/key.py +++ b/src/borg/crypto/key.py @@ -23,7 +23,7 @@ from .nonces import NonceManager from .low_level import AES, bytes_to_long, long_to_bytes, bytes_to_int, num_cipher_blocks, hmac_sha256, blake2b_256, hkdf_hmac_sha512 -from .low_level import AES256_CTR_HMAC_SHA256, AES256_CTR_BLAKE2b +from .low_level import AES256_CTR_HMAC_SHA256, AES256_CTR_BLAKE2b, AES256_OCB, CHACHA20_POLY1305 class UnsupportedPayloadError(Error): @@ -156,8 +156,9 @@ def __init__(self, repository): def id_hash(self, data): """Return HMAC hash using the "id" HMAC key """ + raise NotImplementedError - def encrypt(self, chunk): + def encrypt(self, id, data): pass def decrypt(self, id, data, decompress=True): @@ -263,8 +264,8 @@ def detect(cls, repository, manifest_data): def id_hash(self, data): return sha256(data).digest() - def encrypt(self, chunk): - data = self.compressor.compress(chunk) + def encrypt(self, id, data): + data = self.compressor.compress(data) return b''.join([self.TYPE_STR, data]) def decrypt(self, id, data, decompress=True): @@ -335,12 +336,12 @@ class AESKeyBase(KeyBase): PAYLOAD_OVERHEAD = 1 + 32 + 8 # TYPE + HMAC + NONCE - CIPHERSUITE = AES256_CTR_HMAC_SHA256 + CIPHERSUITE = None # override in derived class logically_encrypted = True - def encrypt(self, chunk): - data = self.compressor.compress(chunk) + def encrypt(self, id, data): + data = self.compressor.compress(data) next_iv = self.nonce_manager.ensure_reservation(self.cipher.next_iv(), self.cipher.block_count(len(data))) return self.cipher.encrypt(data, header=self.TYPE_STR, iv=next_iv) @@ -382,7 +383,9 @@ def init_ciphers(self, manifest_data=None): self.nonce_manager = NonceManager(self.repository, nonce) -class FlexiKeyBase(AESKeyBase): +class FlexiKey: + FILE_ID = 'BORG_KEY' + @classmethod def detect(cls, repository, manifest_data): key = cls(repository) @@ -405,12 +408,6 @@ def detect(cls, repository, manifest_data): key._passphrase = passphrase return key - def find_key(self): - raise NotImplementedError - - def load(self, target, passphrase): - raise NotImplementedError - def _load(self, key_data, passphrase): cdata = a2b_base64(key_data) data = self.decrypt_key_file(cdata, passphrase) @@ -488,18 +485,6 @@ def create(cls, repository, args): logger.info('Keep this key safe. Your data will be inaccessible without it.') return key - def save(self, target, passphrase, create=False): - raise NotImplementedError - - def get_new_target(self, args): - raise NotImplementedError - - -class FlexiKey(ID_HMAC_SHA_256, FlexiKeyBase): - TYPES_ACCEPTABLE = {KeyType.KEYFILE, KeyType.REPO, KeyType.PASSPHRASE} - - FILE_ID = 'BORG_KEY' - def sanity_check(self, filename, id): file_id = self.FILE_ID.encode() + b' ' repo_id = hexlify(id) @@ -624,40 +609,43 @@ def remove(self, target): raise TypeError('Unsupported borg key storage type') -class KeyfileKey(FlexiKey): +class KeyfileKey(ID_HMAC_SHA_256, AESKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.KEYFILE, KeyType.REPO, KeyType.PASSPHRASE} TYPE = KeyType.KEYFILE NAME = 'key file' ARG_NAME = 'keyfile' STORAGE = KeyBlobStorage.KEYFILE + CIPHERSUITE = AES256_CTR_HMAC_SHA256 -class RepoKey(FlexiKey): +class RepoKey(ID_HMAC_SHA_256, AESKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.KEYFILE, KeyType.REPO, KeyType.PASSPHRASE} TYPE = KeyType.REPO NAME = 'repokey' ARG_NAME = 'repokey' STORAGE = KeyBlobStorage.REPO + CIPHERSUITE = AES256_CTR_HMAC_SHA256 -class Blake2FlexiKey(ID_BLAKE2b_256, FlexiKey): +class Blake2KeyfileKey(ID_BLAKE2b_256, AESKeyBase, FlexiKey): TYPES_ACCEPTABLE = {KeyType.BLAKE2KEYFILE, KeyType.BLAKE2REPO} - CIPHERSUITE = AES256_CTR_BLAKE2b - - -class Blake2KeyfileKey(Blake2FlexiKey): TYPE = KeyType.BLAKE2KEYFILE NAME = 'key file BLAKE2b' ARG_NAME = 'keyfile-blake2' STORAGE = KeyBlobStorage.KEYFILE + CIPHERSUITE = AES256_CTR_BLAKE2b -class Blake2RepoKey(Blake2FlexiKey): +class Blake2RepoKey(ID_BLAKE2b_256, AESKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.BLAKE2KEYFILE, KeyType.BLAKE2REPO} TYPE = KeyType.BLAKE2REPO NAME = 'repokey BLAKE2b' ARG_NAME = 'repokey-blake2' STORAGE = KeyBlobStorage.REPO + CIPHERSUITE = AES256_CTR_BLAKE2b -class AuthenticatedKeyBase(FlexiKey): +class AuthenticatedKeyBase(AESKeyBase, FlexiKey): STORAGE = KeyBlobStorage.REPO # It's only authenticated, not encrypted. @@ -676,8 +664,8 @@ def init_ciphers(self, manifest_data=None): if manifest_data is not None: self.assert_type(manifest_data[0]) - def encrypt(self, chunk): - data = self.compressor.compress(chunk) + def encrypt(self, id, data): + data = self.compressor.compress(data) return b''.join([self.TYPE_STR, data]) def decrypt(self, id, data, decompress=True): @@ -690,7 +678,7 @@ def decrypt(self, id, data, decompress=True): return data -class AuthenticatedKey(AuthenticatedKeyBase): +class AuthenticatedKey(ID_HMAC_SHA_256, AuthenticatedKeyBase): TYPE = KeyType.AUTHENTICATED TYPES_ACCEPTABLE = {TYPE} NAME = 'authenticated' @@ -704,8 +692,173 @@ class Blake2AuthenticatedKey(ID_BLAKE2b_256, AuthenticatedKeyBase): ARG_NAME = 'authenticated-blake2' +# ------------ new crypto ------------ + + +class AEADKeyBase(KeyBase): + """ + Chunks are encrypted and authenticated using some AEAD ciphersuite + + Layout: suite:4 keytype:4 reserved:8 messageIV:48 sessionID:192 auth_tag:128 payload:... [bits] + ^-------------------- AAD ----------------------------^ + Offsets:0 1 2 8 32 48 [bytes] + + suite: 1010b for new AEAD crypto, 0000b is old crypto + keytype: see constants.KeyType (suite+keytype) + reserved: all-zero, for future use + messageIV: a counter starting from 0 for all new encrypted messages of one session + sessionID: 192bit random, computed once per session (the session key is derived from this) + auth_tag: authentication tag output of the AEAD cipher (computed over payload and AAD) + payload: encrypted chunk data + """ + + PAYLOAD_OVERHEAD = 1 + 1 + 6 + 24 + 16 # [bytes], see Layout + + CIPHERSUITE = None # override in subclass + + logically_encrypted = True + + MAX_IV = 2 ** 48 - 1 + + def encrypt(self, id, data): + # to encrypt new data in this session we use always self.cipher and self.sessionid + data = self.compressor.compress(data) + reserved = b'\0' + iv = self.cipher.next_iv() + if iv > self.MAX_IV: # see the data-structures docs about why the IV range is enough + raise IntegrityError("IV overflow, should never happen.") + iv_48bit = iv.to_bytes(6, 'big') + header = self.TYPE_STR + reserved + iv_48bit + self.sessionid + return self.cipher.encrypt(data, header=header, iv=iv, aad=id) + + def decrypt(self, id, data, decompress=True): + # to decrypt existing data, we need to get a cipher configured for the sessionid and iv from header + self.assert_type(data[0], id) + iv_48bit = data[2:8] + sessionid = data[8:32] + iv = int.from_bytes(iv_48bit, 'big') + cipher = self._get_cipher(sessionid, iv) + try: + payload = cipher.decrypt(data, aad=id) + except IntegrityError as e: + raise IntegrityError(f"Chunk {bin_to_hex(id)}: Could not decrypt [{str(e)}]") + if not decompress: + return payload + data = self.decompress(payload) + self.assert_id(id, data) + return data + + def init_from_random_data(self): + data = os.urandom(100) + self.enc_key = data[0:32] + self.enc_hmac_key = data[32:64] + self.id_key = data[64:96] + self.chunk_seed = bytes_to_int(data[96:100]) + # Convert to signed int32 + if self.chunk_seed & 0x80000000: + self.chunk_seed = self.chunk_seed - 0xffffffff - 1 + + def _get_session_key(self, sessionid): + assert len(sessionid) == 24 # 192bit + key = hkdf_hmac_sha512( + ikm=self.enc_key + self.enc_hmac_key, + salt=sessionid, + info=b'borg-session-key-' + self.CIPHERSUITE.__name__.encode(), + output_length=32 + ) + return key + + def _get_cipher(self, sessionid, iv): + assert isinstance(iv, int) + key = self._get_session_key(sessionid) + cipher = self.CIPHERSUITE(key=key, iv=iv, header_len=1+1+6+24, aad_offset=0) + return cipher + + def init_ciphers(self, manifest_data=None, iv=0): + # in every new session we start with a fresh sessionid and at iv == 0, manifest_data and iv params are ignored + self.sessionid = os.urandom(24) + self.cipher = self._get_cipher(self.sessionid, iv=0) + + +class AESOCBKeyfileKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.AESOCBKEYFILE, KeyType.AESOCBREPO} + TYPE = KeyType.AESOCBKEYFILE + NAME = 'key file AES-OCB' + ARG_NAME = 'keyfile-aes-ocb' + STORAGE = KeyBlobStorage.KEYFILE + CIPHERSUITE = AES256_OCB + + +class AESOCBRepoKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.AESOCBKEYFILE, KeyType.AESOCBREPO} + TYPE = KeyType.AESOCBREPO + NAME = 'repokey AES-OCB' + ARG_NAME = 'repokey-aes-ocb' + STORAGE = KeyBlobStorage.REPO + CIPHERSUITE = AES256_OCB + + +class CHPOKeyfileKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.CHPOKEYFILE, KeyType.CHPOREPO} + TYPE = KeyType.CHPOKEYFILE + NAME = 'key file ChaCha20-Poly1305' + ARG_NAME = 'keyfile-chacha20-poly1305' + STORAGE = KeyBlobStorage.KEYFILE + CIPHERSUITE = CHACHA20_POLY1305 + + +class CHPORepoKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.CHPOKEYFILE, KeyType.CHPOREPO} + TYPE = KeyType.CHPOREPO + NAME = 'repokey ChaCha20-Poly1305' + ARG_NAME = 'repokey-chacha20-poly1305' + STORAGE = KeyBlobStorage.REPO + CIPHERSUITE = CHACHA20_POLY1305 + + +class Blake2AESOCBKeyfileKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.BLAKE2AESOCBKEYFILE, KeyType.BLAKE2AESOCBREPO} + TYPE = KeyType.BLAKE2AESOCBKEYFILE + NAME = 'key file Blake2b AES-OCB' + ARG_NAME = 'keyfile-blake2-aes-ocb' + STORAGE = KeyBlobStorage.KEYFILE + CIPHERSUITE = AES256_OCB + + +class Blake2AESOCBRepoKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.BLAKE2AESOCBKEYFILE, KeyType.BLAKE2AESOCBREPO} + TYPE = KeyType.BLAKE2AESOCBREPO + NAME = 'repokey Blake2b AES-OCB' + ARG_NAME = 'repokey-blake2-aes-ocb' + STORAGE = KeyBlobStorage.REPO + CIPHERSUITE = AES256_OCB + + +class Blake2CHPOKeyfileKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.BLAKE2CHPOKEYFILE, KeyType.BLAKE2CHPOREPO} + TYPE = KeyType.BLAKE2CHPOKEYFILE + NAME = 'key file Blake2b ChaCha20-Poly1305' + ARG_NAME = 'keyfile-blake2-chacha20-poly1305' + STORAGE = KeyBlobStorage.KEYFILE + CIPHERSUITE = CHACHA20_POLY1305 + + +class Blake2CHPORepoKey(ID_BLAKE2b_256, AEADKeyBase, FlexiKey): + TYPES_ACCEPTABLE = {KeyType.BLAKE2CHPOKEYFILE, KeyType.BLAKE2CHPOREPO} + TYPE = KeyType.BLAKE2CHPOREPO + NAME = 'repokey Blake2b ChaCha20-Poly1305' + ARG_NAME = 'repokey-blake2-chacha20-poly1305' + STORAGE = KeyBlobStorage.REPO + CIPHERSUITE = CHACHA20_POLY1305 + + AVAILABLE_KEY_TYPES = ( PlaintextKey, KeyfileKey, RepoKey, AuthenticatedKey, Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey, + # new crypto + AESOCBKeyfileKey, AESOCBRepoKey, + CHPOKeyfileKey, CHPORepoKey, + Blake2AESOCBKeyfileKey, Blake2AESOCBRepoKey, + Blake2CHPOKeyfileKey, Blake2CHPORepoKey, ) diff --git a/src/borg/crypto/low_level.pyx b/src/borg/crypto/low_level.pyx index b43d4a79fa..5692fa23c9 100644 --- a/src/borg/crypto/low_level.pyx +++ b/src/borg/crypto/low_level.pyx @@ -42,7 +42,7 @@ from cpython cimport PyMem_Malloc, PyMem_Free from cpython.buffer cimport PyBUF_SIMPLE, PyObject_GetBuffer, PyBuffer_Release from cpython.bytes cimport PyBytes_FromStringAndSize -API_VERSION = '1.2_01' +API_VERSION = '1.3_01' cdef extern from "openssl/crypto.h": int CRYPTO_memcmp(const void *a, const void *b, size_t len) @@ -77,9 +77,9 @@ cdef extern from "openssl/evp.h": int EVP_DecryptFinal_ex(EVP_CIPHER_CTX *ctx, unsigned char *out, int *outl) int EVP_CIPHER_CTX_ctrl(EVP_CIPHER_CTX *ctx, int type, int arg, void *ptr) - int EVP_CTRL_GCM_GET_TAG - int EVP_CTRL_GCM_SET_TAG - int EVP_CTRL_GCM_SET_IVLEN + int EVP_CTRL_AEAD_GET_TAG + int EVP_CTRL_AEAD_SET_TAG + int EVP_CTRL_AEAD_SET_IVLEN const EVP_MD *EVP_sha256() nogil @@ -152,7 +152,7 @@ class UNENCRYPTED: self.header_len = header_len self.set_iv(iv) - def encrypt(self, data, header=b'', iv=None): + def encrypt(self, data, header=b'', iv=None, aad=None): """ IMPORTANT: it is called encrypt to satisfy the crypto api naming convention, but this does NOT encrypt and it does NOT compute and store a MAC either. @@ -162,7 +162,7 @@ class UNENCRYPTED: assert self.iv is not None, 'iv needs to be set before encrypt is called' return header + data - def decrypt(self, envelope): + def decrypt(self, envelope, aad=None): """ IMPORTANT: it is called decrypt to satisfy the crypto api naming convention, but this does NOT decrypt and it does NOT verify a MAC either, because data @@ -184,10 +184,10 @@ class UNENCRYPTED: cdef class AES256_CTR_BASE: - # Layout: HEADER + MAC 32 + IV 8 + CT (same as attic / borg < 1.2 IF HEADER = TYPE_BYTE, no AAD) + # Layout: HEADER + MAC 32 + IV 8 + CT (same as attic / borg < 1.3 IF HEADER = TYPE_BYTE, no AAD) cdef EVP_CIPHER_CTX *ctx - cdef unsigned char *enc_key + cdef unsigned char enc_key[32] cdef int cipher_blk_len cdef int iv_len, iv_len_short cdef int aad_offset @@ -235,7 +235,7 @@ cdef class AES256_CTR_BASE: """ raise NotImplementedError - def encrypt(self, data, header=b'', iv=None): + def encrypt(self, data, header=b'', iv=None, aad=None): """ encrypt data, compute mac over aad + iv + cdata, prepend header. aad_offset is the offset into the header where aad starts. @@ -252,7 +252,7 @@ cdef class AES256_CTR_BASE: ilen + self.cipher_blk_len) # play safe, 1 extra blk if not odata: raise MemoryError - cdef int olen + cdef int olen = 0 cdef int offset cdef Py_buffer idata = ro_buffer(data) cdef Py_buffer hdata = ro_buffer(header) @@ -264,15 +264,12 @@ cdef class AES256_CTR_BASE: offset += self.mac_len self.store_iv(odata+offset, self.iv) offset += self.iv_len_short - rc = EVP_EncryptInit_ex(self.ctx, EVP_aes_256_ctr(), NULL, self.enc_key, self.iv) - if not rc: + if not EVP_EncryptInit_ex(self.ctx, EVP_aes_256_ctr(), NULL, self.enc_key, self.iv): raise CryptoError('EVP_EncryptInit_ex failed') - rc = EVP_EncryptUpdate(self.ctx, odata+offset, &olen, idata.buf, ilen) - if not rc: + if not EVP_EncryptUpdate(self.ctx, odata+offset, &olen, idata.buf, ilen): raise CryptoError('EVP_EncryptUpdate failed') offset += olen - rc = EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen) - if not rc: + if not EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen): raise CryptoError('EVP_EncryptFinal_ex failed') offset += olen self.mac_compute( hdata.buf+aoffset, alen, @@ -285,19 +282,18 @@ cdef class AES256_CTR_BASE: PyBuffer_Release(&hdata) PyBuffer_Release(&idata) - def decrypt(self, envelope): + def decrypt(self, envelope, aad=None): """ authenticate aad + iv + cdata, decrypt cdata, ignore header bytes up to aad_offset. """ cdef int ilen = len(envelope) cdef int hlen = self.header_len - assert hlen == self.header_len cdef int aoffset = self.aad_offset cdef int alen = hlen - aoffset cdef unsigned char *odata = PyMem_Malloc(ilen + self.cipher_blk_len) # play safe, 1 extra blk if not odata: raise MemoryError - cdef int olen + cdef int olen = 0 cdef int offset cdef unsigned char mac_buf[32] assert sizeof(mac_buf) == self.mac_len @@ -311,14 +307,12 @@ cdef class AES256_CTR_BASE: if not EVP_DecryptInit_ex(self.ctx, EVP_aes_256_ctr(), NULL, self.enc_key, iv): raise CryptoError('EVP_DecryptInit_ex failed') offset = 0 - rc = EVP_DecryptUpdate(self.ctx, odata+offset, &olen, - idata.buf+hlen+self.mac_len+self.iv_len_short, - ilen-hlen-self.mac_len-self.iv_len_short) - if not rc: + if not EVP_DecryptUpdate(self.ctx, odata+offset, &olen, + idata.buf+hlen+self.mac_len+self.iv_len_short, + ilen-hlen-self.mac_len-self.iv_len_short): raise CryptoError('EVP_DecryptUpdate failed') offset += olen - rc = EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen) - if rc <= 0: + if not EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen): raise CryptoError('EVP_DecryptFinal_ex failed') offset += olen self.blocks += self.block_count(offset) @@ -335,8 +329,7 @@ cdef class AES256_CTR_BASE: if isinstance(iv, int): iv = iv.to_bytes(self.iv_len, byteorder='big') assert isinstance(iv, bytes) and len(iv) == self.iv_len - for i in range(self.iv_len): - self.iv[i] = iv[i] + self.iv = iv self.blocks = 0 # how many AES blocks got encrypted with this IV? def next_iv(self): @@ -360,7 +353,7 @@ cdef class AES256_CTR_BASE: cdef class AES256_CTR_HMAC_SHA256(AES256_CTR_BASE): - cdef unsigned char *mac_key + cdef unsigned char mac_key[32] def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1): assert isinstance(mac_key, bytes) and len(mac_key) == 32 @@ -377,7 +370,7 @@ cdef class AES256_CTR_HMAC_SHA256(AES256_CTR_BASE): const unsigned char *data2, int data2_len, unsigned char *mac_buf): data = data1[:data1_len] + data2[:data2_len] - mac = hmac.HMAC(self.mac_key, data, hashlib.sha256).digest() + mac = hmac.digest(self.mac_key[:self.mac_len], data, 'sha256') for i in range(self.mac_len): mac_buf[i] = mac[i] @@ -390,7 +383,7 @@ cdef class AES256_CTR_HMAC_SHA256(AES256_CTR_BASE): cdef class AES256_CTR_BLAKE2b(AES256_CTR_BASE): - cdef unsigned char *mac_key + cdef unsigned char mac_key[128] def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1): assert isinstance(mac_key, bytes) and len(mac_key) == 128 @@ -423,15 +416,16 @@ ctypedef const EVP_CIPHER * (* CIPHER)() cdef class _AEAD_BASE: - # Layout: HEADER + MAC 16 + IV 12 + CT + # new crypto used in borg >= 1.3 + # Layout: HEADER + MAC 16 + CT cdef CIPHER cipher cdef EVP_CIPHER_CTX *ctx - cdef unsigned char *enc_key + cdef unsigned char key[32] cdef int cipher_blk_len cdef int iv_len cdef int aad_offset - cdef int header_len + cdef int header_len_expected cdef int mac_len cdef unsigned char iv[12] cdef long long blocks @@ -441,83 +435,87 @@ cdef class _AEAD_BASE: """check whether library requirements for this ciphersuite are satisfied""" raise NotImplemented # override / implement in child class - def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1): - assert mac_key is None - assert isinstance(enc_key, bytes) and len(enc_key) == 32 + def __init__(self, key, iv=None, header_len=0, aad_offset=0): + """ + init AEAD crypto + + :param key: 256bit encrypt-then-mac key + :param iv: 96bit initialisation vector / nonce + :param header_len: expected length of header + :param aad_offset: where in the header the authenticated data starts + """ + assert isinstance(key, bytes) and len(key) == 32 self.iv_len = sizeof(self.iv) - self.header_len = 1 + self.header_len_expected = header_len assert aad_offset <= header_len self.aad_offset = aad_offset - self.header_len = header_len self.mac_len = 16 - self.enc_key = enc_key + self.key = key if iv is not None: self.set_iv(iv) else: self.blocks = -1 # make sure set_iv is called before encrypt - def __cinit__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1): + def __cinit__(self, key, iv=None, header_len=0, aad_offset=0): self.ctx = EVP_CIPHER_CTX_new() def __dealloc__(self): EVP_CIPHER_CTX_free(self.ctx) - def encrypt(self, data, header=b'', iv=None): + def encrypt(self, data, header=b'', iv=None, aad=b''): """ - encrypt data, compute mac over aad + iv + cdata, prepend header. - aad_offset is the offset into the header where aad starts. + encrypt data, compute auth tag over aad + header + cdata. + return header + auth tag + cdata. + aad_offset is the offset into the header where the authenticated header part starts. + aad is additional authenticated data, which won't be included in the returned data, + but only used for the auth tag computation. """ if iv is not None: self.set_iv(iv) assert self.blocks == 0, 'iv needs to be set before encrypt is called' - # AES-GCM, AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte) + # AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte) # IV we provide, thus we must not encrypt more than 2^32 cipher blocks with same IV). block_count = self.block_count(len(data)) if block_count > 2**32: raise ValueError('too much data, would overflow internal 32bit counter') cdef int ilen = len(data) cdef int hlen = len(header) - assert hlen == self.header_len + assert hlen == self.header_len_expected cdef int aoffset = self.aad_offset cdef int alen = hlen - aoffset - cdef unsigned char *odata = PyMem_Malloc(hlen + self.mac_len + self.iv_len + + cdef int aadlen = len(aad) + cdef unsigned char *odata = PyMem_Malloc(hlen + self.mac_len + ilen + self.cipher_blk_len) if not odata: raise MemoryError - cdef int olen + cdef int olen = 0 cdef int offset cdef Py_buffer idata = ro_buffer(data) cdef Py_buffer hdata = ro_buffer(header) + cdef Py_buffer aadata = ro_buffer(aad) try: offset = 0 for i in range(hlen): odata[offset+i] = header[i] offset += hlen offset += self.mac_len - self.store_iv(odata+offset, self.iv) - rc = EVP_EncryptInit_ex(self.ctx, self.cipher(), NULL, NULL, NULL) - if not rc: + if not EVP_EncryptInit_ex(self.ctx, self.cipher(), NULL, NULL, NULL): raise CryptoError('EVP_EncryptInit_ex failed') - if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_SET_IVLEN, self.iv_len, NULL): + if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_SET_IVLEN, self.iv_len, NULL): raise CryptoError('EVP_CIPHER_CTX_ctrl SET IVLEN failed') - rc = EVP_EncryptInit_ex(self.ctx, NULL, NULL, self.enc_key, self.iv) - if not rc: + if not EVP_EncryptInit_ex(self.ctx, NULL, NULL, self.key, self.iv): raise CryptoError('EVP_EncryptInit_ex failed') - rc = EVP_EncryptUpdate(self.ctx, NULL, &olen, hdata.buf+aoffset, alen) - if not rc: + if not EVP_EncryptUpdate(self.ctx, NULL, &olen, aadata.buf, aadlen): raise CryptoError('EVP_EncryptUpdate failed') - if not EVP_EncryptUpdate(self.ctx, NULL, &olen, odata+offset, self.iv_len): + if not EVP_EncryptUpdate(self.ctx, NULL, &olen, hdata.buf+aoffset, alen): raise CryptoError('EVP_EncryptUpdate failed') - offset += self.iv_len - rc = EVP_EncryptUpdate(self.ctx, odata+offset, &olen, idata.buf, ilen) - if not rc: + if not EVP_EncryptUpdate(self.ctx, odata+offset, &olen, idata.buf, ilen): raise CryptoError('EVP_EncryptUpdate failed') offset += olen - rc = EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen) - if not rc: + if not EVP_EncryptFinal_ex(self.ctx, odata+offset, &olen): raise CryptoError('EVP_EncryptFinal_ex failed') offset += olen - if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_GET_TAG, self.mac_len, odata+hlen): + if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_GET_TAG, self.mac_len, odata + hlen): raise CryptoError('EVP_CIPHER_CTX_ctrl GET TAG failed') self.blocks = block_count return odata[:offset] @@ -525,53 +523,50 @@ cdef class _AEAD_BASE: PyMem_Free(odata) PyBuffer_Release(&hdata) PyBuffer_Release(&idata) + PyBuffer_Release(&aadata) - def decrypt(self, envelope): + def decrypt(self, envelope, aad=b''): """ - authenticate aad + iv + cdata, decrypt cdata, ignore header bytes up to aad_offset. + authenticate aad + header + cdata (from envelope), ignore header bytes up to aad_offset., + return decrypted cdata. """ - # AES-GCM, AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte) + # AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte) # IV we provide, thus we must not decrypt more than 2^32 cipher blocks with same IV): approx_block_count = self.block_count(len(envelope)) # sloppy, but good enough for borg if approx_block_count > 2**32: raise ValueError('too much data, would overflow internal 32bit counter') cdef int ilen = len(envelope) - cdef int hlen = self.header_len - assert hlen == self.header_len + cdef int hlen = self.header_len_expected cdef int aoffset = self.aad_offset cdef int alen = hlen - aoffset + cdef int aadlen = len(aad) cdef unsigned char *odata = PyMem_Malloc(ilen + self.cipher_blk_len) if not odata: raise MemoryError - cdef int olen + cdef int olen = 0 cdef int offset cdef Py_buffer idata = ro_buffer(envelope) + cdef Py_buffer aadata = ro_buffer(aad) try: if not EVP_DecryptInit_ex(self.ctx, self.cipher(), NULL, NULL, NULL): raise CryptoError('EVP_DecryptInit_ex failed') - iv = self.fetch_iv( idata.buf+hlen+self.mac_len) - self.set_iv(iv) - if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_SET_IVLEN, self.iv_len, NULL): + if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_SET_IVLEN, self.iv_len, NULL): raise CryptoError('EVP_CIPHER_CTX_ctrl SET IVLEN failed') - if not EVP_DecryptInit_ex(self.ctx, NULL, NULL, self.enc_key, iv): + if not EVP_DecryptInit_ex(self.ctx, NULL, NULL, self.key, self.iv): raise CryptoError('EVP_DecryptInit_ex failed') - if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_GCM_SET_TAG, self.mac_len, idata.buf+hlen): - raise CryptoError('EVP_CIPHER_CTX_ctrl SET TAG failed') - rc = EVP_DecryptUpdate(self.ctx, NULL, &olen, idata.buf+aoffset, alen) - if not rc: + if not EVP_DecryptUpdate(self.ctx, NULL, &olen, aadata.buf, aadlen): raise CryptoError('EVP_DecryptUpdate failed') - if not EVP_DecryptUpdate(self.ctx, NULL, &olen, - idata.buf+hlen+self.mac_len, self.iv_len): + if not EVP_DecryptUpdate(self.ctx, NULL, &olen, idata.buf+aoffset, alen): raise CryptoError('EVP_DecryptUpdate failed') offset = 0 - rc = EVP_DecryptUpdate(self.ctx, odata+offset, &olen, - idata.buf+hlen+self.mac_len+self.iv_len, - ilen-hlen-self.mac_len-self.iv_len) - if not rc: + if not EVP_DecryptUpdate(self.ctx, odata+offset, &olen, + idata.buf+hlen+self.mac_len, + ilen-hlen-self.mac_len): raise CryptoError('EVP_DecryptUpdate failed') offset += olen - rc = EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen) - if rc <= 0: + if not EVP_CIPHER_CTX_ctrl(self.ctx, EVP_CTRL_AEAD_SET_TAG, self.mac_len, idata.buf + hlen): + raise CryptoError('EVP_CIPHER_CTX_ctrl SET TAG failed') + if not EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen): # a failure here means corrupted or tampered tag (mac) or data. raise IntegrityError('Authentication / EVP_DecryptFinal_ex failed') offset += olen @@ -580,6 +575,7 @@ cdef class _AEAD_BASE: finally: PyMem_Free(odata) PyBuffer_Release(&idata) + PyBuffer_Release(&aadata) def block_count(self, length): return num_cipher_blocks(length, self.cipher_blk_len) @@ -590,71 +586,48 @@ cdef class _AEAD_BASE: if isinstance(iv, int): iv = iv.to_bytes(self.iv_len, byteorder='big') assert isinstance(iv, bytes) and len(iv) == self.iv_len - for i in range(self.iv_len): - self.iv[i] = iv[i] + self.iv = iv self.blocks = 0 # number of cipher blocks encrypted with this IV def next_iv(self): # call this after encrypt() to get the next iv (int) for the next encrypt() call - # AES-GCM, AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit + # AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit # (12 byte) IV we provide, thus we only need to increment the IV by 1. iv = int.from_bytes(self.iv[:self.iv_len], byteorder='big') return iv + 1 - cdef fetch_iv(self, unsigned char * iv_in): - return iv_in[0:self.iv_len] - - cdef store_iv(self, unsigned char * iv_out, unsigned char * iv): - cdef int i - for i in range(self.iv_len): - iv_out[i] = iv[i] - - def extract_iv(self, envelope): - offset = self.header_len + self.mac_len - return bytes_to_long(envelope[offset:offset+self.iv_len]) - - -cdef class _AES_BASE(_AEAD_BASE): - def __init__(self, *args, **kwargs): - self.cipher_blk_len = 16 - super().__init__(*args, **kwargs) - - -cdef class _CHACHA_BASE(_AEAD_BASE): - def __init__(self, *args, **kwargs): - self.cipher_blk_len = 64 - super().__init__(*args, **kwargs) - -cdef class AES256_OCB(_AES_BASE): +cdef class AES256_OCB(_AEAD_BASE): @classmethod def requirements_check(cls): if is_libressl: raise ValueError('AES OCB is not implemented by LibreSSL (yet?).') - def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1): + def __init__(self, key, iv=None, header_len=0, aad_offset=0): self.requirements_check() self.cipher = EVP_aes_256_ocb - super().__init__(mac_key, enc_key, iv=iv, header_len=header_len, aad_offset=aad_offset) + self.cipher_blk_len = 16 + super().__init__(key, iv=iv, header_len=header_len, aad_offset=aad_offset) -cdef class CHACHA20_POLY1305(_CHACHA_BASE): +cdef class CHACHA20_POLY1305(_AEAD_BASE): @classmethod def requirements_check(cls): if is_libressl: raise ValueError('CHACHA20-POLY1305 is not implemented by LibreSSL (yet?).') - def __init__(self, mac_key, enc_key, iv=None, header_len=1, aad_offset=1): + def __init__(self, key, iv=None, header_len=0, aad_offset=0): self.requirements_check() self.cipher = EVP_chacha20_poly1305 - super().__init__(mac_key, enc_key, iv=iv, header_len=header_len, aad_offset=aad_offset) + self.cipher_blk_len = 64 + super().__init__(key, iv=iv, header_len=header_len, aad_offset=aad_offset) cdef class AES: """A thin wrapper around the OpenSSL EVP cipher API - for legacy code, like key file encryption""" cdef CIPHER cipher cdef EVP_CIPHER_CTX *ctx - cdef unsigned char *enc_key + cdef unsigned char enc_key[32] cdef int cipher_blk_len cdef int iv_len cdef unsigned char iv[16] @@ -721,7 +694,7 @@ cdef class AES: if not EVP_DecryptUpdate(self.ctx, odata, &olen, idata.buf, ilen): raise Exception('EVP_DecryptUpdate failed') offset += olen - if EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen) <= 0: + if not EVP_DecryptFinal_ex(self.ctx, odata+offset, &olen): # this error check is very important for modes with padding or # authentication. for them, a failure here means corrupted data. # CTR mode does not use padding nor authentication. @@ -742,8 +715,7 @@ cdef class AES: if isinstance(iv, int): iv = iv.to_bytes(self.iv_len, byteorder='big') assert isinstance(iv, bytes) and len(iv) == self.iv_len - for i in range(self.iv_len): - self.iv[i] = iv[i] + self.iv = iv self.blocks = 0 # number of cipher blocks encrypted with this IV def next_iv(self): @@ -752,7 +724,6 @@ cdef class AES: return iv + self.blocks - def hmac_sha256(key, data): return hmac.digest(key, data, 'sha256') @@ -779,7 +750,7 @@ def hkdf_hmac_sha512(ikm, salt, info, output_length): # Step 1. HKDF-Extract (ikm, salt) -> prk if salt is None: salt = bytes(64) - prk = hmac.HMAC(salt, ikm, hashlib.sha512).digest() + prk = hmac.digest(salt, ikm, 'sha512') # Step 2. HKDF-Expand (prk, info, output_length) -> output key n = ceil(output_length / digest_length) @@ -787,6 +758,6 @@ def hkdf_hmac_sha512(ikm, salt, info, output_length): output = b'' for i in range(n): msg = t_n + info + (i + 1).to_bytes(1, 'little') - t_n = hmac.HMAC(prk, msg, hashlib.sha512).digest() + t_n = hmac.digest(prk, msg, 'sha512') output += t_n return output[:output_length] diff --git a/src/borg/helpers/checks.py b/src/borg/helpers/checks.py index 1fe2c1439c..1d14788b95 100644 --- a/src/borg/helpers/checks.py +++ b/src/borg/helpers/checks.py @@ -31,7 +31,7 @@ def check_extension_modules(): raise ExtensionModuleError if compress.API_VERSION != '1.2_02': raise ExtensionModuleError - if borg.crypto.low_level.API_VERSION != '1.2_01': + if borg.crypto.low_level.API_VERSION != '1.3_01': raise ExtensionModuleError if item.API_VERSION != '1.2_01': raise ExtensionModuleError diff --git a/src/borg/helpers/manifest.py b/src/borg/helpers/manifest.py index 84871f736f..dc0580f289 100644 --- a/src/borg/helpers/manifest.py +++ b/src/borg/helpers/manifest.py @@ -261,4 +261,4 @@ def write(self): self.tam_verified = True data = self.key.pack_and_authenticate_metadata(manifest.as_dict()) self.id = self.key.id_hash(data) - self.repository.put(self.MANIFEST_ID, self.key.encrypt(data)) + self.repository.put(self.MANIFEST_ID, self.key.encrypt(self.MANIFEST_ID, data)) diff --git a/src/borg/selftest.py b/src/borg/selftest.py index dfe7abb545..bacdcea2a5 100644 --- a/src/borg/selftest.py +++ b/src/borg/selftest.py @@ -29,7 +29,7 @@ ChunkerTestCase, ] -SELFTEST_COUNT = 35 +SELFTEST_COUNT = 36 class SelfTestResult(TestResult): diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py index 2eb552bd40..62a33fb898 100644 --- a/src/borg/testsuite/archiver.py +++ b/src/borg/testsuite/archiver.py @@ -36,7 +36,7 @@ from ..chunker import has_seek_hole from ..constants import * # NOQA from ..crypto.low_level import bytes_to_long, num_cipher_blocks -from ..crypto.key import FlexiKeyBase, RepoKey, KeyfileKey, Passphrase, TAMRequiredError +from ..crypto.key import FlexiKey, RepoKey, KeyfileKey, Passphrase, TAMRequiredError from ..crypto.keymanager import RepoIdMismatch, NotABorgKeyFile from ..crypto.file_integrity import FileIntegrityError from ..helpers import Location, get_security_dir @@ -2882,7 +2882,7 @@ def test_init_interrupt(self): def raise_eof(*args): raise EOFError - with patch.object(FlexiKeyBase, 'create', raise_eof): + with patch.object(FlexiKey, 'create', raise_eof): self.cmd('init', '--encryption=repokey', self.repository_location, exit_code=1) assert not os.path.exists(self.repository_location) @@ -3806,7 +3806,7 @@ def test_manifest_rebuild_duplicate_archive(self): 'version': 1, }) archive_id = key.id_hash(archive) - repository.put(archive_id, key.encrypt(archive)) + repository.put(archive_id, key.encrypt(archive_id, archive)) repository.commit(compact=False) self.cmd('check', self.repository_location, exit_code=1) self.cmd('check', '--repair', self.repository_location, exit_code=0) @@ -3894,7 +3894,7 @@ class ManifestAuthenticationTest(ArchiverTestCaseBase): def spoof_manifest(self, repository): with repository: _, key = Manifest.load(repository, Manifest.NO_OPERATION_CHECK) - repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb({ + repository.put(Manifest.MANIFEST_ID, key.encrypt(Manifest.MANIFEST_ID, msgpack.packb({ 'version': 1, 'archives': {}, 'config': {}, @@ -3907,7 +3907,7 @@ def test_fresh_init_tam_required(self): repository = Repository(self.repository_path, exclusive=True) with repository: manifest, key = Manifest.load(repository, Manifest.NO_OPERATION_CHECK) - repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb({ + repository.put(Manifest.MANIFEST_ID, key.encrypt(Manifest.MANIFEST_ID, msgpack.packb({ 'version': 1, 'archives': {}, 'timestamp': (datetime.utcnow() + timedelta(days=1)).strftime(ISO_FORMAT), @@ -3929,7 +3929,7 @@ def test_not_required(self): manifest = msgpack.unpackb(key.decrypt(None, repository.get(Manifest.MANIFEST_ID))) del manifest[b'tam'] - repository.put(Manifest.MANIFEST_ID, key.encrypt(msgpack.packb(manifest))) + repository.put(Manifest.MANIFEST_ID, key.encrypt(Manifest.MANIFEST_ID, msgpack.packb(manifest))) repository.commit(compact=False) output = self.cmd('list', '--debug', self.repository_location) assert 'archive1234' in output diff --git a/src/borg/testsuite/crypto.py b/src/borg/testsuite/crypto.py index 6420f36c88..baf8cba2cd 100644 --- a/src/borg/testsuite/crypto.py +++ b/src/borg/testsuite/crypto.py @@ -91,11 +91,10 @@ def test_AES256_CTR_HMAC_SHA256_aad(self): def test_AE(self): # used in legacy-like layout (1 type byte, no aad) - mac_key = None - enc_key = b'X' * 32 - iv = 0 + key = b'X' * 32 + iv_int = 0 data = b'foo' * 10 - header = b'\x23' + header = b'\x23' + iv_int.to_bytes(12, 'big') tests = [ # (ciphersuite class, exp_mac, exp_cdata) ] @@ -111,11 +110,11 @@ def test_AE(self): for cs_cls, exp_mac, exp_cdata in tests: # print(repr(cs_cls)) # encrypt/mac - cs = cs_cls(mac_key, enc_key, iv, header_len=1, aad_offset=1) + cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1) hdr_mac_iv_cdata = cs.encrypt(data, header=header) hdr = hdr_mac_iv_cdata[0:1] - mac = hdr_mac_iv_cdata[1:17] - iv = hdr_mac_iv_cdata[17:29] + iv = hdr_mac_iv_cdata[1:13] + mac = hdr_mac_iv_cdata[13:29] cdata = hdr_mac_iv_cdata[29:] self.assert_equal(hexlify(hdr), b'23') self.assert_equal(hexlify(mac), exp_mac) @@ -123,23 +122,22 @@ def test_AE(self): self.assert_equal(hexlify(cdata), exp_cdata) self.assert_equal(cs.next_iv(), 1) # auth/decrypt - cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1) + cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1) pdata = cs.decrypt(hdr_mac_iv_cdata) self.assert_equal(data, pdata) self.assert_equal(cs.next_iv(), 1) # auth-failure due to corruption (corrupted data) - cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1) + cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1) hdr_mac_iv_cdata_corrupted = hdr_mac_iv_cdata[:29] + b'\0' + hdr_mac_iv_cdata[30:] self.assert_raises(IntegrityError, lambda: cs.decrypt(hdr_mac_iv_cdata_corrupted)) def test_AEAD(self): # test with aad - mac_key = None - enc_key = b'X' * 32 - iv = 0 + key = b'X' * 32 + iv_int = 0 data = b'foo' * 10 - header = b'\x12\x34\x56' + header = b'\x12\x34\x56' + iv_int.to_bytes(12, 'big') tests = [ # (ciphersuite class, exp_mac, exp_cdata) ] @@ -155,11 +153,11 @@ def test_AEAD(self): for cs_cls, exp_mac, exp_cdata in tests: # print(repr(cs_cls)) # encrypt/mac - cs = cs_cls(mac_key, enc_key, iv, header_len=3, aad_offset=1) + cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1) hdr_mac_iv_cdata = cs.encrypt(data, header=header) hdr = hdr_mac_iv_cdata[0:3] - mac = hdr_mac_iv_cdata[3:19] - iv = hdr_mac_iv_cdata[19:31] + iv = hdr_mac_iv_cdata[3:15] + mac = hdr_mac_iv_cdata[15:31] cdata = hdr_mac_iv_cdata[31:] self.assert_equal(hexlify(hdr), b'123456') self.assert_equal(hexlify(mac), exp_mac) @@ -167,16 +165,38 @@ def test_AEAD(self): self.assert_equal(hexlify(cdata), exp_cdata) self.assert_equal(cs.next_iv(), 1) # auth/decrypt - cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1) + cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1) pdata = cs.decrypt(hdr_mac_iv_cdata) self.assert_equal(data, pdata) self.assert_equal(cs.next_iv(), 1) # auth-failure due to corruption (corrupted aad) - cs = cs_cls(mac_key, enc_key, header_len=len(header), aad_offset=1) + cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=1) hdr_mac_iv_cdata_corrupted = hdr_mac_iv_cdata[:1] + b'\0' + hdr_mac_iv_cdata[2:] self.assert_raises(IntegrityError, lambda: cs.decrypt(hdr_mac_iv_cdata_corrupted)) + def test_AEAD_with_more_AAD(self): + # test giving extra aad to the .encrypt() and .decrypt() calls + key = b'X' * 32 + iv_int = 0 + data = b'foo' * 10 + header = b'\x12\x34' + tests = [] + if not is_libressl: + tests += [AES256_OCB, CHACHA20_POLY1305] + for cs_cls in tests: + # encrypt/mac + cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0) + hdr_mac_iv_cdata = cs.encrypt(data, header=header, aad=b'correct_chunkid') + # successful auth/decrypt (correct aad) + cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0) + pdata = cs.decrypt(hdr_mac_iv_cdata, aad=b'correct_chunkid') + self.assert_equal(data, pdata) + # unsuccessful auth (incorrect aad) + cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0) + self.assert_raises(IntegrityError, + lambda: cs.decrypt(hdr_mac_iv_cdata, aad=b'incorrect_chunkid')) + # These test vectors come from https://www.kullo.net/blog/hkdf-sha-512-test-vectors/ # who claims to have verified these against independent Python and C++ implementations. diff --git a/src/borg/testsuite/key.py b/src/borg/testsuite/key.py index 6d2207a9e7..14297d480b 100644 --- a/src/borg/testsuite/key.py +++ b/src/borg/testsuite/key.py @@ -8,7 +8,8 @@ from ..crypto.key import bin_to_hex from ..crypto.key import PlaintextKey, AuthenticatedKey, RepoKey, KeyfileKey, \ - Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey + Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey, \ + AESOCBKeyfileKey, AESOCBRepoKey, CHPOKeyfileKey, CHPORepoKey from ..crypto.key import ID_HMAC_SHA_256, ID_BLAKE2b_256 from ..crypto.key import TAMRequiredError, TAMInvalid, TAMUnsupportedSuiteError, UnsupportedManifestError from ..crypto.key import identify_key @@ -80,6 +81,8 @@ def keys_dir(self, request, monkeypatch, tmpdir): Blake2KeyfileKey, Blake2RepoKey, Blake2AuthenticatedKey, + AESOCBKeyfileKey, AESOCBRepoKey, + CHPOKeyfileKey, CHPORepoKey, )) def key(self, request, monkeypatch): monkeypatch.setenv('BORG_PASSPHRASE', 'test') @@ -111,18 +114,21 @@ def load_key(self): def test_plaintext(self): key = PlaintextKey.create(None, None) chunk = b'foo' - assert hexlify(key.id_hash(chunk)) == b'2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae' - assert chunk == key.decrypt(key.id_hash(chunk), key.encrypt(chunk)) + id = key.id_hash(chunk) + assert hexlify(id) == b'2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae' + assert chunk == key.decrypt(id, key.encrypt(id, chunk)) def test_keyfile(self, monkeypatch, keys_dir): monkeypatch.setenv('BORG_PASSPHRASE', 'test') key = KeyfileKey.create(self.MockRepository(), self.MockArgs()) assert key.cipher.next_iv() == 0 - manifest = key.encrypt(b'ABC') + chunk = b'ABC' + id = key.id_hash(chunk) + manifest = key.encrypt(id, chunk) assert key.cipher.extract_iv(manifest) == 0 - manifest2 = key.encrypt(b'ABC') + manifest2 = key.encrypt(id, chunk) assert manifest != manifest2 - assert key.decrypt(None, manifest) == key.decrypt(None, manifest2) + assert key.decrypt(id, manifest) == key.decrypt(id, manifest2) assert key.cipher.extract_iv(manifest2) == 1 iv = key.cipher.extract_iv(manifest) key2 = KeyfileKey.detect(self.MockRepository(), manifest) @@ -131,7 +137,8 @@ def test_keyfile(self, monkeypatch, keys_dir): assert len({key2.id_key, key2.enc_key, key2.enc_hmac_key}) == 3 assert key2.chunk_seed != 0 chunk = b'foo' - assert chunk == key2.decrypt(key.id_hash(chunk), key.encrypt(chunk)) + id = key.id_hash(chunk) + assert chunk == key2.decrypt(id, key.encrypt(id, chunk)) def test_keyfile_nonce_rollback_protection(self, monkeypatch, keys_dir): monkeypatch.setenv('BORG_PASSPHRASE', 'test') @@ -139,9 +146,11 @@ def test_keyfile_nonce_rollback_protection(self, monkeypatch, keys_dir): with open(os.path.join(get_security_dir(repository.id_str), 'nonce'), "w") as fd: fd.write("0000000000002000") key = KeyfileKey.create(repository, self.MockArgs()) - data = key.encrypt(b'ABC') + chunk = b'ABC' + id = key.id_hash(chunk) + data = key.encrypt(id, chunk) assert key.cipher.extract_iv(data) == 0x2000 - assert key.decrypt(None, data) == b'ABC' + assert key.decrypt(id, data) == chunk def test_keyfile_kfenv(self, tmpdir, monkeypatch): keyfile = tmpdir.join('keyfile') @@ -152,7 +161,7 @@ def test_keyfile_kfenv(self, tmpdir, monkeypatch): assert keyfile.exists() chunk = b'ABC' chunk_id = key.id_hash(chunk) - chunk_cdata = key.encrypt(chunk) + chunk_cdata = key.encrypt(chunk_id, chunk) key = KeyfileKey.detect(self.MockRepository(), chunk_cdata) assert chunk == key.decrypt(chunk_id, chunk_cdata) keyfile.remove() @@ -209,18 +218,20 @@ def test_decrypt_integrity(self, monkeypatch, keys_dir): def test_roundtrip(self, key): repository = key.repository plaintext = b'foo' - encrypted = key.encrypt(plaintext) + id = key.id_hash(plaintext) + encrypted = key.encrypt(id, plaintext) identified_key_class = identify_key(encrypted) assert identified_key_class == key.__class__ loaded_key = identified_key_class.detect(repository, encrypted) - decrypted = loaded_key.decrypt(None, encrypted) + decrypted = loaded_key.decrypt(id, encrypted) assert decrypted == plaintext def test_decrypt_decompress(self, key): plaintext = b'123456789' - encrypted = key.encrypt(plaintext) - assert key.decrypt(None, encrypted, decompress=False) != plaintext - assert key.decrypt(None, encrypted) == plaintext + id = key.id_hash(plaintext) + encrypted = key.encrypt(id, plaintext) + assert key.decrypt(id, encrypted, decompress=False) != plaintext + assert key.decrypt(id, encrypted) == plaintext def test_assert_id(self, key): plaintext = b'123456789' @@ -240,7 +251,8 @@ def test_authenticated_encrypt(self, monkeypatch): assert AuthenticatedKey.id_hash is ID_HMAC_SHA_256.id_hash assert len(key.id_key) == 32 plaintext = b'123456789' - authenticated = key.encrypt(plaintext) + id = key.id_hash(plaintext) + authenticated = key.encrypt(id, plaintext) # 0x07 is the key TYPE, \x0000 identifies no compression. assert authenticated == b'\x07\x00\x00' + plaintext @@ -250,7 +262,8 @@ def test_blake2_authenticated_encrypt(self, monkeypatch): assert Blake2AuthenticatedKey.id_hash is ID_BLAKE2b_256.id_hash assert len(key.id_key) == 128 plaintext = b'123456789' - authenticated = key.encrypt(plaintext) + id = key.id_hash(plaintext) + authenticated = key.encrypt(id, plaintext) # 0x06 is the key TYPE, 0x0000 identifies no compression. assert authenticated == b'\x06\x00\x00' + plaintext diff --git a/src/borg/testsuite/remote.py b/src/borg/testsuite/remote.py index 034ebb4bad..fe94dc414d 100644 --- a/src/borg/testsuite/remote.py +++ b/src/borg/testsuite/remote.py @@ -165,7 +165,7 @@ def key(self, repository, monkeypatch): def _put_encrypted_object(self, key, repository, data): id_ = key.id_hash(data) - repository.put(id_, key.encrypt(data)) + repository.put(id_, key.encrypt(id_, data)) return id_ @pytest.fixture