add easyblock to install OpenSSL wrapper for OpenSSL installed in OS, or build and install OpenSSL from source if not available in OS

[lexming]

Fixes issue easybuilders/easybuild-easyconfigs#11895

This easyblock looks for the installation files of OpenSSL in the host system to wrap them in the EB installation directory with symlinks

• it searches the library in the host system by loading the respective libssl.so in Python and retrieve the path to the library file using dlinfo in Linux and ctypes.macholib.dyld.dyld_find in Mac OS.
• it searches the headers of the required version of OpenSSL (v1.0.x or v1.1.x) in the include paths of gcc in the system.

This method does not need to crawl through any environment variables, is resilient to installations in exotic paths and integrates well within the EB environment.

Why symlinking?

• it provides a transparent way to update OpenSSL without breaking dynamic links and probably without breaking rpaths either. Any packages build with the wrappers in place should use the paths to the installation directory of the wrapper, not the actual libraries in the system or EB.

If the host system lacks the libs or headers of the required version of OpenSSL, the easyblock builds and installs any component included in the easyconfig bundle (EB_OpenSSL_wrapper is based on Bundle). This approach has the advantage of automatically disabling the installation of a fallback OpenSSL in EB if the host system already has the required files. Moreover, if the fallback is installed, the result is identical to any other installation of OpenSSL in EB.

Easyconfigs can be found in easybuilders/easybuild-easyconfigs#12858

edit: requires easybuilders/easybuild-framework#3682 + easybuilders/easybuild-framework#3683

[lexming]

this regex seems fine to me

[boegel]

@lexming Fails for me with:

$ eb --from-pr 12858 --include-easyblocks-from-pr 2429
...
ERROR: Traceback (most recent call last):
  File "/data/gent/400/vsc40023/easybuild_easy_installed/lib/python2.7/site-packages/easybuild_framework-4.3.5.dev0-py2.7.egg/easybuild/main.py", line 117, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/data/gent/400/vsc40023/easybuild_easy_installed/lib/python2.7/site-packages/easybuild_framework-4.3.5.dev0-py2.7.egg/easybuild/framework/easyblock.py", line 3371, in build_and_install_one
    app = app_class(ecdict['ec'])
  File "/tmp/eb-wb7k98av/included-easyblocks-u3uhdwkv/easybuild/easyblocks/openssl_wrapper.py", line 132, in __init__
    os.path.join(lib_dir, self.openssl_engines[self.version]),
  File "/usr/lib64/python3.6/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib64/python3.6/genericpath.py", line 151, in _check_arg_types
    raise TypeError("Can't mix strings and bytes in path components") from None
TypeError: Can't mix strings and bytes in path components

[boegel]

Let's see if a close/re-open wakes up GitHub Actions...

[Micket]

Soo.. do we not want this to find the CentOS7 openssl11 packages? Because they are intentionally not installed into a standard path, since that would break the rest of the system?

[boegel]

Soo.. do we not want this to find the CentOS7 openssl11 packages? Because they are intentionally not installed into a standard path, since that would break the rest of the system?

We could enhance this easyblock to make sure those are found too, but you would need to specify where to find them of course (and we could include the CentOS 7 location as a default in the easyblock)?

[Micket]

Right, well, I'm not sure how to really suggest adding those paths, and, it's pretty much a mix here. We've got

1. /usr/bin/openssl11
2. /usr/lib64/openssl11/ (or /usr/lib64/libssl.so.1.1)
3. /usr/include/openssl11/
   So, optional system_ssl_binary, system_ssl_libpath, ssl_ssl_includepath? I think it's preferable for these to be set by default on centos/rhel 7 systems since it would be to easy to miss for a large portion or users.

Also, the automatic fallback method here; there isn't a practical way to update ever update it without having to revert to a live rebuild (ouch)? (as opposed to the more atomic change like with Java and modulerc)

[lexming]

With the addition of multiple library names per system/version, I moved the code loading and finding the library to its own function in systemtools. See easybuilders/easybuild-framework#3683

[boegel]

@lexming Another crash when OpenSSL-1.0.eb is being built from source after your last update:

ERROR: Traceback (most recent call last):
  File "/home/centos/easybuild/easybuild-framework/easybuild/main.py", line 117, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/home/centos/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3571, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/home/centos/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3465, in run_all_steps
    self.run_step(step_name, step_methods)
  File "/home/centos/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3320, in run_step
    step_method(self)()
  File "/tmp/eb-t7009csl/included-easyblocks-lj9yuliv/easybuild/easyblocks/openssl_wrapper.py", line 252, in sanity_check_step
    ssl_files.extend(os.path.join('lib', libso) for libso in self.openssl_libs)
  File "/tmp/eb-t7009csl/included-easyblocks-lj9yuliv/easybuild/easyblocks/openssl_wrapper.py", line 252, in <genexpr>
    ssl_files.extend(os.path.join('lib', libso) for libso in self.openssl_libs)
  File "/usr/lib64/python3.6/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib64/python3.6/genericpath.py", line 149, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'list'

[lexming]

@boegel will you stop breaking it? I'll have to ban your username from this easyblock...

In fact, I introduced this bug in my last commit. I'll make a better fix 👍

[lexming]

@boegel fixed in fd425a2. I replaced the changing list of OpenSSL libraries in self.openssl_libs into two separate variables, where the target libraries of the installation are always held in a simple list (self.target_ssl_libs). This fixes the last two errors.

I repeated all my previous tests and they are good.

all good now

[boegel]

Tested extensively with easybuilders/easybuild-easyconfigs#12858 across a range of different systems, operating systems, and CPU architectures.

@lexming: Thanks a lot for your efforts on this!