enable system SSL certificates in OpenSSL installations #2683

lexming · 2022-02-18T14:34:40Z

fixes #2674, fixes easybuilders/easybuild-easyconfigs#14901

Changelog:

make OpenSSL wrapper to error out instead of building OpenSSL from source if the only missing pieces in the system are the header files. On error, inform the user to either install the development package of openssl in the system or force the installation from source in EasyBuild with wrap_system_openssl
symlink system certificates (if present) in the installation directory of OpenSSL whenever it is build from source
add ssl_certificates option to openssl easyblock to define the path to system certificates
update sanity checks in openssl easyblock: remove hardcoded library extensions, remove checks for lib and lib64 (now EB installs both), add checks on ssl/certs whenever certificates are symlinked

The link to the system certificates is determined in the following order:

use ssl_certificates path if defined
fall back to OPENSSLDIR from system openssl
fall back to generic path in GENERIC_SSL_CERTS_DIR
fall back to installation without certificates and print warning informing the user

lexming · 2022-02-19T00:30:24Z

Test report by @lexming

Overview of tested easyconfigs (in order)

SUCCESS OpenSSL-1.1.1b-GCCcore-8.2.0.eb
SUCCESS OpenSSL-1.1.1d-GCCcore-8.3.0.eb
SUCCESS OpenSSL-1.1.1e-GCCcore-9.3.0.eb
SUCCESS OpenSSL-1.1.1h-GCCcore-10.2.0.eb
SUCCESS OpenSSL-1.1.1k-GCCcore-10.3.0.eb
SUCCESS OpenSSL-1.1.1k-GCCcore-11.2.0.eb

Build succeeded for 6 out of 6 (6 easyconfigs in total)
node305.hydra.os - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, Python 2.7.5
See https://gist.github.com/5e3788eff4e94ca74dcabd2ef88f9814 for a full test report.

OPENSSLDIR points to %(installdir)s/ssl (its default value).
The resulting ssl folder on CentOS 7

ssl/
├── certs -> /etc/pki/tls/certs
├── ct_log_list.cnf
├── ct_log_list.cnf.dist
├── misc
│   ├── CA.pl
│   ├── tsget -> tsget.pl
│   └── tsget.pl
├── openssl.cnf
├── openssl.cnf.dist
└── private

lexming · 2022-02-19T01:14:19Z

Test report by @lexming

Overview of tested easyconfigs (in order)

SUCCESS OpenSSL-1.0.eb
SUCCESS OpenSSL-1.1.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node300.hydra.os - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, Python 2.7.5
See https://gist.github.com/ff7513d04eb3c9d6e768c6b0bfb3f8f5 for a full test report.

Tests with default easyconfigs. OpenSSL in CentOS 7 is wrapped successfully. Nothing changes in this case. OPENSSLDIR correctly points to /etc/pki/tls

lexming · 2022-02-19T01:55:47Z

Test report by @lexming

Overview of tested easyconfigs (in order)

SUCCESS OpenSSL-1.0.eb
SUCCESS OpenSSL-1.1.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node301.hydra.os - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, Python 2.7.5
See https://gist.github.com/6641022a002257c9e7fa4ce1ba599a04 for a full test report.

Tests with wrap_system_openssl=False. OpenSSL is successfully built from source. OPENSSLDIR points to %(installdir)s/ssl (its default value).

ssl folder for v1.0 on CentOS 7

OpenSSL/1.0/ssl
├── certs -> /etc/pki/tls/certs
├── man
│   ├── man1
│   ├── man3
│   ├── man5
│   └── man7
├── misc
│   ├── CA.pl
│   ├── CA.sh
│   ├── c_hash
│   ├── c_info
│   ├── c_issuer
│   ├── c_name
│   └── tsget
├── openssl.cnf
└── private

ssl folder for v1.1 on CentOS 7

OpenSSL/1.1/ssl
├── certs -> /etc/pki/tls/certs
├── ct_log_list.cnf
├── ct_log_list.cnf.dist
├── misc
│   ├── CA.pl
│   ├── tsget -> tsget.pl
│   └── tsget.pl
├── openssl.cnf
├── openssl.cnf.dist
└── private

boegel · 2022-03-04T20:19:23Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS OpenSSL-1.0.eb
SUCCESS OpenSSL-1.1.eb
SUCCESS OpenSSL-1.1.1k-GCCcore-11.2.0.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
node2682.swalot.os - Linux openSUSE Leap 15.3, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.15
See https://gist.github.com/49637d370970760f83a921f8a3a83507 for a full test report.

$ grep PRETTY /etc/os-release
PRETTY_NAME="openSUSE Leap 15.3"
$ rpm -qa | grep ^openssl
openssl-1.1.1d-1.46.noarch
openssl-1_1-1.1.1d-11.38.1.x86_64

$ ls OpenSSL/1.0/ssl
certs  man  misc  openssl.cnf  private
$ ls -l OpenSSL/1.0/ssl/certs
lrwxrwxrwx 1 vsc40023 vsc40023 14 Mar  4 20:14 OpenSSL/1.0/ssl/certs -> /etc/ssl/certs

$ ls -l OpenSSL/1.1/bin/openssl OpenSSL/1.1/lib/libssl.so
lrwxrwxrwx 1 vsc40023 vsc40023 16 Mar  4 20:14 OpenSSL/1.1/bin/openssl -> /usr/bin/openssl
lrwxrwxrwx 1 vsc40023 vsc40023 13 Mar  4 20:14 OpenSSL/1.1/lib/libssl.so -> libssl.so.1.1

boegel · 2022-03-04T20:50:15Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS OpenSSL-1.0.eb
SUCCESS OpenSSL-1.1.1d-GCCcore-8.3.0.eb
SUCCESS OpenSSL-1.1.1h-GCCcore-10.2.0.eb
SUCCESS OpenSSL-1.1.1k-GCCcore-10.3.0.eb
SUCCESS OpenSSL-1.1.1k-GCCcore-11.2.0.eb
SUCCESS OpenSSL-1.1.eb

Build succeeded for 6 out of 6 (6 easyconfigs in total)
node3502.doduo.os - Linux RHEL 8.4, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/6f4228a16c0a63d4865fbfc442bef896 for a full test report.

$ grep PRETTY /etc/os-release
PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
$ rpm -qa | grep ^openssl
openssl-devel-1.1.1g-15.el8_3.x86_64
openssl-libs-1.1.1g-15.el8_3.x86_64
openssl-1.1.1g-15.el8_3.x86_64
openssl-libs-1.1.1g-15.el8_3.i686

$ ls -l OpenSSL/1.0/ssl/certs
lrwxrwxrwx 1 vsc40023 vsc40023 18 Mar  4 21:30 OpenSSL/1.0/ssl/certs -> /etc/pki/tls/certs
$ ls -l OpenSSL/1.1.1k-GCCcore-11.2.0/ssl/certs
lrwxrwxrwx 1 vsc40023 vsc40023 18 Mar  4 21:49 OpenSSL/1.1.1k-GCCcore-11.2.0/ssl/certs -> /etc/pki/tls/certs

boegel · 2022-03-04T21:14:07Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS OpenSSL-1.0.eb
SUCCESS OpenSSL-1.1.eb
SUCCESS OpenSSL-1.1.1k-GCCcore-11.2.0.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
node2682.swalot.os - Linux Ubuntu 20.04.3 LTS (Focal Fossa), x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.8.10
See https://gist.github.com/895ff8cd16466705083f7cd70611effc for a full test report.

$ grep PRETTY /etc/os-release
PRETTY_NAME="Ubuntu 20.04.3 LTS"
$ apt list openssl | grep installed
openssl/focal-updates,now 1.1.1f-1ubuntu2.10 amd64 [installed,automatic]

$ ls OpenSSL/1.0/ssl
certs  man  misc  openssl.cnf  private
$ ls -l OpenSSL/1.0/ssl/certs
lrwxrwxrwx 1 vsc40023 vsc40023 18 Mar  4 21:09 OpenSSL/1.0/ssl/certs -> /usr/lib/ssl/certs

$ ls -l OpenSSL/1.1/{bin,lib64}/{openssl,libssl.so} 2>/dev/null
lrwxrwxrwx 1 vsc40023 vsc40023 16 Mar  4 21:09 OpenSSL/1.1/bin/openssl -> /usr/bin/openssl
lrwxrwxrwx 1 vsc40023 vsc40023 13 Mar  4 21:09 OpenSSL/1.1/lib64/libssl.so -> libssl.so.1.1

boegel

lgtm

lexming added 6 commits February 18, 2022 01:34

trigger error if system OpenSSL only lacks the headers

a48af77

add ssl_certificates option to install system certificates with OpenSSL

c8d6332

find SSL certificates in system from OPENSSLDIR

571e741

streamline sanity checks of OpenSSL

4e79821

add SSL certificates to sanity check of OpenSSL

d4ef108

print warning for installations of OpenSSL without certificates

954cf76

lexming added the enhancement label Feb 18, 2022

lexming added this to the next release (4.5.4?) milestone Feb 18, 2022

This comment was marked as outdated.

Sign in to view

hardcode openssl name in paths and binaries

ce5941d

boegel changed the title ~~Enable system SSL certificates in OpenSSL installations~~ enable system SSL certificates in OpenSSL installations Mar 1, 2022

boegel approved these changes Mar 4, 2022

View reviewed changes

boegel merged commit a1752d9 into easybuilders:develop Mar 4, 2022

lexming deleted the ssl-certs branch March 4, 2022 21:57

bedroge mentioned this pull request May 11, 2022

{tools}[foss/2021a] jax v0.3.9 w/ Python 3.9.5 easybuilders/easybuild-easyconfigs#15422

Merged

boegel mentioned this pull request Jun 4, 2022

sanity check for TensorFlow-2.6.0-foss-2021a.eb fails (download problem when running TensorFlow-2.x_mnist-test.py test case) easybuilders/easybuild-easyconfigs#14058

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable system SSL certificates in OpenSSL installations #2683

enable system SSL certificates in OpenSSL installations #2683

lexming commented Feb 18, 2022 •

edited by boegel

Loading

This comment was marked as outdated.

lexming commented Feb 19, 2022 •

edited

Loading

lexming commented Feb 19, 2022 •

edited

Loading

lexming commented Feb 19, 2022 •

edited

Loading

boegel commented Mar 4, 2022 •

edited

Loading

boegel commented Mar 4, 2022 •

edited

Loading

boegel commented Mar 4, 2022 •

edited

Loading

boegel left a comment

enable system SSL certificates in OpenSSL installations #2683

enable system SSL certificates in OpenSSL installations #2683

Conversation

lexming commented Feb 18, 2022 • edited by boegel Loading

This comment was marked as outdated.

lexming commented Feb 19, 2022 • edited Loading

Overview of tested easyconfigs (in order)

lexming commented Feb 19, 2022 • edited Loading

Overview of tested easyconfigs (in order)

lexming commented Feb 19, 2022 • edited Loading

Overview of tested easyconfigs (in order)

boegel commented Mar 4, 2022 • edited Loading

Overview of tested easyconfigs (in order)

boegel commented Mar 4, 2022 • edited Loading

Overview of tested easyconfigs (in order)

boegel commented Mar 4, 2022 • edited Loading

Overview of tested easyconfigs (in order)

boegel left a comment

Choose a reason for hiding this comment

lexming commented Feb 18, 2022 •

edited by boegel

Loading

lexming commented Feb 19, 2022 •

edited

Loading

lexming commented Feb 19, 2022 •

edited

Loading

lexming commented Feb 19, 2022 •

edited

Loading

boegel commented Mar 4, 2022 •

edited

Loading

boegel commented Mar 4, 2022 •

edited

Loading

boegel commented Mar 4, 2022 •

edited

Loading