libressl's AEAD api

[ThomasWaldmann]

it seems that they don't support these ciphers via the same EVP api as openssl:

• AES-OCB
• CHACHA20-POLY1305

For borg 1.3+ it seems like a good idea if we would deprecate the crypto modes based on AES-CTR and recommend the new AEAD modes.

So, how's the situation on OpenBSD? Are there other platforms requiring LibreSSL support?

• Do we need to support LibreSSL to run on OpenBSD or could we use OpenSSL there also?
• If we can neither use OpenSSL there nor some compatible API in LibreSSL, do we want to add some code for LibreSSL to our borg.crypto.low_level module that uses LibreSSL's EVP_AEAD api (which does not exist in OpenSSL)? https://man.openbsd.org/EVP_AEAD_CTX_init.3
• seems like we only get AES-GCM and chacha20-poly1305 via that api. there were objections against gcm, so we went for ocb rather. so guess that would mean with LibreSSL it would only be chacha20-poly1305 (which is fine as it is quite fast).

[bket]

@ThomasWaldmann

So, how's the situation on OpenBSD? Are there other platforms requiring LibreSSL support?

• Do we need to support LibreSSL to run on OpenBSD or could we use OpenSSL there also?

Some background:

A lot of people, including the LibreSSL people, OpenBSD's porters and upstream projects, invested a lot of effort to get LibreSSL to work with the hundreds of tools that exist. OpenBSD for example has ~11000 ports (including flavors and subpackages) of which ~800 use LibreSSL and only 6 explicitly require OpenSSL.

Concerning your questions: Strictly taken it is possible to use OpenSSL on OpenBSD, though it is not preferred. I'm not sure about requirements from other platforms.

• If we can neither use OpenSSL there nor some compatible API in LibreSSL, do we want to add some code for LibreSSL to our borg.crypto.low_level module that uses LibreSSL's EVP_AEAD api (which does not exist in OpenSSL)? https://man.openbsd.org/EVP_AEAD_CTX_init.3

EVP_aead seems to be borrowed from BoringSSL, and the API for EVP_aead is completely incompatible with what Borg currently uses.
I'm hoping that the EVP things for some of the djb ciphers will be eventually implemented in LibreSSL. AES-OCB is rarely used and was patent encumbered, which does not not help discussion on implementation.

It would be really cool if Borg keeps on supporting LibreSSL. However, in the end it is all about effort needed, and return on effort...

• seems like we only get AES-GCM and chacha20-poly1305 via that api. there were objections against gcm, so we went for ocb rather. so guess that would mean with LibreSSL it would only be chacha20-poly1305 (which is fine as it is quite fast).

I do not want to start a discussion on crypto as I do not understand it. I'm wondering though why OCB is better than GCM. Is it because OCB is a bit less fragile than GCM against repeated nonces? Are there other reasons? Is there a hard reason not to use GCM?

[ThomasWaldmann]

I am roughly aware about LibreSSL and other non-OpenSSL efforts (and had my own issues when working with OpenSSL).
But we need something that supports the borg legacy (== fast AES256-CTR / HMAC-SHA256) as well as a better future (AEAD ciphers: OCB and CHPO).

The OCB patents issue is long resolved for FOSS, see: https://en.wikipedia.org/wiki/OCB_mode and as a recent update also this: https://mailarchive.ietf.org/arch/msg/cfrg/qLTveWOdTJcLn4HP3ev-vrj05Vg/

I can not give you the details about the GCM concerns, but when searching for usable AEAD ciphers I got the impression that quite some cryptographers feel bad about GCM. Considering that OCB is also faster made it clear for me to choose OCB. Years ago, there was the availability issue, but this is also resolved for most platforms since openssl 1.1.0 (and all openssl versions without AES-OCB not receiving security support any more).

So, if openssl is working on openbsd, maybe we should switch to that for borg 1.3+?

Alternatively, maybe some openbsd python/cython developer who is into crypto stuff could work on borg.crypto.low_level for the alternative code needed for LibreSSL? As long as they don't have OCB, one can't read such repos on LibreSSL though.

[ThomasWaldmann]

good news: i have a vagrant based openbsd-current (future 7.1) test box now. all master tests passing currently.

[ThomasWaldmann]

Update: The AES-OCB and chacha20-poly1305 based crypto is now in master branch (was #6463).

[ThomasWaldmann]

I put this into 1.3.0b1 milestone (first beta release).

@bket When the alpha releases get successful testing on openbsd with openssl, I guess we can close this because we'll just use openssl then. In case libressl catches up later regarding API and available algorithms, we can reopen this.

[ThomasWaldmann]

@bket could you help fixing the Vagrantfile in master branch, so it works based on openssl 1.1.1?

Currently it tries building on libressl, which will fail the tests.

[bket]

@ThomasWaldmann, the Vagrantfile needs a small addition:

diff --git Vagrantfile Vagrantfile
index eea43d8c..359ae6d6 100644
--- Vagrantfile
+++ Vagrantfile
@@ -78,6 +78,7 @@ def packages_openbsd
     pkg_add lz4
     pkg_add zstd
     pkg_add git  # no fakeroot
+    pkg_add openssl%1.1
     pkg_add py3-pip
     pkg_add py3-virtualenv
   EOF

Unfortunately this is not enough as we need to explicity tell the build to link against libcrypto from the installed package. I have the diff below sitting in my tree, which passes some flags. Setting BORG_OPENSSL_PREFIX is not going to cut it because of some renaming. With this diff pkgconfig does the right thing, and everything builds/tests OK. Still thinking about an alternative solution.

Index: setup.py
--- setup.py.orig
+++ setup.py
@@ -161,12 +161,13 @@ if not on_rtd:
             pc, 'BORG_OPENSSL_PREFIX', 'libcrypto', 'libcrypto', '>=1.1.1', lib_subdir='')
     else:
         crypto_ext_lib = lib_ext_kwargs(
-            pc, 'BORG_OPENSSL_PREFIX', 'crypto', 'libcrypto', '>=1.1.1')
+            pc, 'BORG_OPENSSL_PREFIX', 'crypto', 'libecrypto11', '>=1.1.1')
 
     crypto_ext_kwargs = members_appended(
         dict(sources=[crypto_ll_source, crypto_helpers]),
         crypto_ext_lib,
         dict(extra_compile_args=cflags),
+        dict(extra_link_args=['-Wl,-rpath,/usr/local/lib/eopenssl11']),
     )
 
     compress_ext_kwargs = members_appended(

[ThomasWaldmann]

thanks for the info!

pkg_add: ah, it needs a percent separator. I tried - and @ and ..., but didn't find that.

the renaming also explains why i did not even manually find the right crypto.pc file...

setup.py: if there is no easier way, guess we need to add a platform check there and just switch these values based on the platform.

[ThomasWaldmann]

#6593 removes LibreSSL support completely. So we currently have nothing holding us back from using AES-OCB and CHACHA20-POLY1305 with borg master (and maybe getting rid of AES-CTR and other old crypto completely).

[sthen]

Be aware that using the OpenSSL shared libraries means that if you use Python extensions which use the LibreSSL shared libraries (including hashlib/ssl as well as external extensions) you'll have symbol conflicts and likely crashes at runtime. That is the reason for the non-standard library naming to prevent software from picking it up automatically in configure scripts etc. Using the static libraries instead should avoid that problem. (Care also needs to be taken to make sure the correct headers are used).

BTW, EVP_chacha20_poly1305() is in -current OpenBSD now, so that will be in 7.2, but no OCB yet.

[ThomasWaldmann]

@sthen Thanks for the hint!

I am not very familiar with such low-level stuff (and how to do that from setup.py, guess it needs specific compiler/linker flags?), neither with OpenBSD.

So, could you do a PR to fix this?

Did you (or @bket maybe) see any actual conflicts / crashes?

[sthen]

I'm not sure to be honest, I haven't tried doing this with Python as everything else using Python in the ports tree is happy with LibreSSL. There are only about 5 ports which require openssl at all, and only one (postfix) needs static-linking so it can coexist with other libraries (as far as ports is concerned, pulling in openssl is a last resort, the preferred approach is to work with libressl devs to get the relevant support added if possible - postfix is about the worst case as they seem to be actively trying to use as much openssl-only functionality as they can ;-)

I'll take a look at @bket's WIP port and see if there's anything I can figure out from there..

[ThomasWaldmann]

Statically linking OpenSSL fixed by #7019, thanks to @bket !

[ThomasWaldmann]

https://www.openbsd.org/73.html

• Added EVP_chacha20_poly1305() to the list of all ciphers.

But still no AES-OCB AFAICS.

[sthen]

Correct, no OCB yet. While I haven't checked with LibreSSL devs about it recently I don't think it is a priority to add. Very little software uses it - the only ones I know about are mosh (normally using an internal implementation rather than openssl's) and Borg.

[ThomasWaldmann]

I chose it because I wanted a fast and secure AES-based AEAD cipher available via OpenSSL. I noticed that cryptographers have bad feelings about AES-GCM being "fragile" and that the "patent issues" that held back AES-OCB for long were resolved. Didn't find any other AES options.