Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SHA-1 GPG keys are deprecated, request to update GPG-keys and RPM repository packages #3124

Closed
Jounimk opened this issue Jan 23, 2023 · 33 comments
Assignees
Labels
enhancement New Enhancement question Further information is requested rpm signing v2.8.0

Comments

@Jounimk
Copy link

Jounimk commented Jan 23, 2023

Is your feature request related to a problem? Please describe

SHA-1 keys are deprecated and it's not possible to install OpenSearch packages from the repository to RHEL 9, which does not accept anymore SHA-1 signed RPMs packages by default (there are considered distrusted).

“Importing GPG key 0x9310D3FC:
Userid : "OpenSearch project <[email protected]>"
Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC
From : https://artifacts.opensearch.org/publickeys/opensearch.pgp
Is this ok [y/N]: y
warning: Signature not supported. Hash algorithm SHA1 not available.
Key import failed (code 2). Failing package is: opensearch-2.4.1-1.x86_64”

https://www.redhat.com/en/blog/rhel-security-sha-1-package-signatures-distrusted-rhel-9

Describe the solution you'd like

I would like that OpenSearch GPG keys would be updated to SHA-256 (or SHA-512) algoritm and RPM packages in the repository would be signed with this new key to make them RHEL 9 compatible by default.

Describe alternatives you've considered

No response

Additional context

No response

Acceptance Criteria

@Jounimk Jounimk added enhancement New Enhancement untriaged Issues that have not yet been triaged labels Jan 23, 2023
@peterzhuamazon
Copy link
Member

Hi @Jounimk we have not officially support RHEL9 related distros yet.
https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/#operating-system-compatibility

Tho this is a good call out and we probably need to just disable sha1 checksum in the build process.
Would label this as a future improvement for now.

Also, @bbarani @setiah please chime in on your thoughts.
Thanks.

@peterzhuamazon peterzhuamazon added question Further information is requested rpm and removed untriaged Issues that have not yet been triaged labels Jan 24, 2023
@peterzhuamazon
Copy link
Member

peterzhuamazon commented Jan 24, 2023

Interesting part is we already use sha512 for signing but seems like sha1 checksum is still somewhat being verified by the host on default so this needs more researching.

@dblock
Copy link
Member

dblock commented Jan 25, 2023

@peterzhuamazon where is this code? Maybe @Jounimk can try and contribute a fix?

@Jounimk
Copy link
Author

Jounimk commented Jan 26, 2023

@dblock This issue is not in the code itself and I'm afraid it cannot be mitigated in the RPM/Deb build process. Problem lies inside old opensearch GPG keypair which was done using SHA-1 hash. Therefore opensearch.pgp public key cannot be imported anymore into the RHEL9 (or similar distributions). So package installation fails
in the preliminary step where opensearch.pgp public key import fails. Whoever is reponsible of the opensearch GPG keys should generate a new key pair with (with GnuPG 2.3.3 version or newer : "gpg --full-generate-key"). Then all .deb and .rpm packages need to be signed with the new private GPG-key. This will solve the described issue.

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Jan 26, 2023

@dblock This issue is not in the code itself and I'm afraid it cannot be mitigated in the RPM/Deb build process. Problem lies inside old opensearch GPG keypair which was done using SHA-1 hash. Therefore opensearch.pgp public key cannot be imported anymore into the RHEL9 (or similar distributions). So package installation fails in the preliminary step where opensearch.pgp public key import fails. Whoever is reponsible of the opensearch GPG keys should generate a new key pair with (with GnuPG 2.3.3 version or newer : "gpg --full-generate-key"). Then all .deb and .rpm packages need to be signed with the new private GPG-key. This will solve the described issue.

Thanks @Jounimk this is a much bigger issue than I initially anticipated.
The best window for this might be after the current subkey expire again:

This means we also need to post 2 public keys for people to verify, old artifacts vs new ones later (possibly)

Need thoughts from @bbarani @CEHENKLE @dblock as this is not extending the key into a new signing subkey, but completely generate a new master key.

All steps are in this issue:

Thanks.

@prudhvigodithi
Copy link
Member

This info could be helpful https://old.nixaid.com/gpg-migration-sha1-to-sha2/ to migrate to sha2. Can we try this @peterzhuamazon ?
Thanks

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Jan 26, 2023

This info could be helpful https://old.nixaid.com/gpg-migration-sha1-to-sha2/ to migrate to sha2. Can we try this @peterzhuamazon ? Thanks

Yes @prudhvigodithi after doing some research I think if we can just migrate from sha1 to sha2, it will be the best case.

@dblock
Copy link
Member

dblock commented Jan 26, 2023

Thanks @peterzhuamazon for the details! @bbarani I think someone will need to take this on your team.

@peterzhuamazon peterzhuamazon self-assigned this Jan 30, 2023
@prudhvigodithi
Copy link
Member

Updated the Acceptance Criteria section of the issue.
Thank you

@peterzhuamazon
Copy link
Member

It requires some research on the migration without completely revoking the key.
The key will expire (public key) in May per the extension last year, we will trying to migrate the secret key from sha1 to sha2 before extending the public key again.

@bbarani bbarani added the v2.8.0 label Mar 29, 2023
@bbarani
Copy link
Member

bbarani commented Mar 29, 2023

We are targeting to migrate to SHA-2 from 2.8.0 version. CC: @peterzhuamazon @prudhvigodithi

@bbarani
Copy link
Member

bbarani commented Apr 5, 2023

@peterzhuamazon @prudhvigodithi Lets do the analysis and report the findings soon.

@peterzhuamazon
Copy link
Member

Starting to do the key migration now before expiration day 20230512.

@peterzhuamazon
Copy link
Member

Test on a rockylinux9 server on this:

# gpg -vv --version
gpg (GnuPG) 2.3.3
libgcrypt 1.10.0-unknown
Copyright (C) 2021 Free Software Foundation, Inc.
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Pubkey: RSA (1), ELG (16), DSA (17), ECDH (18), ECDSA (19), EDDSA (22)
Cipher: IDEA (S1), 3DES (S2), CAST5 (S3), BLOWFISH (S4), AES (S7),
        AES192 (S8), AES256 (S9), TWOFISH (S10), CAMELLIA128 (S11),
        CAMELLIA192 (S12), CAMELLIA256 (S13)
AEAD: EAX (A1), OCB (A2)
Hash: SHA1 (H2), RIPEMD160 (H3), SHA256 (H8), SHA384 (H9), SHA512 (H10),
      SHA224 (H11)
Compression: Uncompressed (Z0), ZIP (Z1), ZLIB (Z2), BZIP2 (Z3)

@peterzhuamazon
Copy link
Member

peterzhuamazon commented May 2, 2023

Old key confirm to have sha1 (hash / digest-algo 2)

:signature packet: algo 1, keyid 39D319879310D3FC
......
digest algo 2
......
algo: 3, SHA1 protection, hash: 2

@peterzhuamazon
Copy link
Member

peterzhuamazon commented May 2, 2023

We will try to change the cipher from CAST5 to AES256 (S3 to S9) while change hash from sha1 to sha512 (H2 to H10) in line with the rpm signature internally signed within the package.

@peterzhuamazon
Copy link
Member

peterzhuamazon commented May 2, 2023

Note that the change means both subprivate key and master private key need to have this change.
This does not affect the detached gpg signature but also affect package signing like rpm, which is currently still using master private key to sign due to not all the rpm version support subkey signing.

@peterzhuamazon
Copy link
Member

A related bug seems not able to let me force a sha512:
https://dev.gnupg.org/T1800

@peterzhuamazon
Copy link
Member

Use this gpg.conf:


personal-cipher-preferences AES256
personal-digest-preferences SHA512
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
cipher-algo AES256
digest-algo SHA512
cert-digest-algo SHA512
compress-algo ZLIB
disable-cipher-algo 3DES
#weak-digest SHA1
s2k-cipher-algo AES256
s2k-digest-algo SHA512
s2k-mode 3
s2k-count 65011712

@peterzhuamazon
Copy link
Member

Trying to use a older version of gpg:


gpg (GnuPG) 2.2.20
libgcrypt 1.8.5
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Pubkey: RSA (1), ELG (16), DSA (17), ECDH (18), ECDSA (19), EDDSA (22)
Cipher: IDEA (S1), 3DES (S2), CAST5 (S3), BLOWFISH (S4), AES (S7),
        AES192 (S8), AES256 (S9), TWOFISH (S10), CAMELLIA128 (S11),
        CAMELLIA192 (S12), CAMELLIA256 (S13)
Hash: SHA1 (H2), RIPEMD160 (H3), SHA256 (H8), SHA384 (H9), SHA512 (H10),
      SHA224 (H11)
Compression: Uncompressed (Z0), ZIP (Z1), ZLIB (Z2), BZIP2 (Z3)

@peterzhuamazon
Copy link
Member

peterzhuamazon commented May 2, 2023

Seems like it will not able to change the existing sha1 key to sha2, period, unless you recreate a key initially in sha2.
This is kinda contradict from many of the post online claiming it is possible.
Yet I only see the results similar to this one:

https://security.stackexchange.com/questions/259483/changing-the-encryption-algorithm-used-for-pgp-gpg-private-key

It will keep showing as this:

    iter+salt S2K, algo: 7, SHA1 protection, hash: 2,

@peterzhuamazon
Copy link
Member

peterzhuamazon commented May 2, 2023

RH official post: https://www.redhat.com/en/blog/rhel-security-sha-1-package-signatures-distrusted-rhel-9

                                                                                                                                                                                        104 MB/s | 679 MB     00:06
OpenSearch 2.x                                                                                                                                                                                                                                                                               33 kB/s | 3.1 kB     00:00
Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <[email protected]>"
 Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC
 From       : https://artifacts.opensearch.org/publickeys/opensearch.pgp
Is this ok [y/N]: y
warning: Signature not supported. Hash algorithm SHA1 not available.
Key import failed (code 2). Failing package is: opensearch-2.7.0-1.x86_64
 GPG Keys are configured as: https://artifacts.opensearch.org/publickeys/opensearch.pgp
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED

@peterzhuamazon
Copy link
Member

Seems like as long as the public key is in sha2 then rpm install will let it through

:signature packet: algo 1, keyid 39D319879310D3FC
	version 4, created <>, md5len <>, sigclass <>
	digest algo 10, begin of digest <>

Installed:
  opensearch-2.7.0-1.x86_64
# cat /etc/*release
NAME="Rocky Linux"
VERSION="9.1 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.1 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.1"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"
Rocky Linux release 9.1 (Blue Onyx)
Rocky Linux release 9.1 (Blue Onyx)
Rocky Linux release 9.1 (Blue Onyx)

@peterzhuamazon
Copy link
Member

peterzhuamazon commented May 3, 2023

Use this config and only change hash to SHA512 (H10), this results in the public key generated having 2 signature packets of both H2 and H10, to suit for both older and newer requirements.

gpg (GnuPG) 2.2.20
libgcrypt 1.8.5
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Pubkey: RSA (1), ELG (16), DSA (17), ECDH (18), ECDSA (19), EDDSA (22)
Cipher: IDEA (S1), 3DES (S2), CAST5 (S3), BLOWFISH (S4), AES (S7),
        AES192 (S8), AES256 (S9), TWOFISH (S10), CAMELLIA128 (S11),
        CAMELLIA192 (S12), CAMELLIA256 (S13)
Hash: SHA1 (H2), RIPEMD160 (H3), SHA256 (H8), SHA384 (H9), SHA512 (H10),
      SHA224 (H11)
Compression: Uncompressed (Z0), ZIP (Z1), ZLIB (Z2), BZIP2 (Z3)
personal-digest-preferences SHA512
digest-algo SHA512
cert-digest-algo SHA512
s2k-digest-algo SHA512
s2k-mode 3
s2k-count 65011712

The cipher remained to be on CAST5 for now.

Expire in one year at about 20240512.

Test on Rockylinux9:

Old key:

Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <[email protected]>"
 Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC
 From       : https://artifacts.opensearch.org/publickeys/opensearch.pgp
Is this ok [y/N]: y
warning: Signature not supported. Hash algorithm SHA1 not available.
Key import failed (code 2). Failing package is: opensearch-2.4.0-1.x86_64
 GPG Keys are configured as: https://artifacts.opensearch.org/publickeys/opensearch.pgp
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED

New key: (IGNORE post install script issues as I am testing on docker without systemd now)

Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <[email protected]>"
 Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC
 From       : /testcerts/opensearch_20240512.pgp
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                                                    1/1
  Running scriptlet: opensearch-2.4.0-1.x86_64                                                                                                                                                                                                                                                                          1/1
  Installing       : opensearch-2.4.0-1.x86_64                                                                                                                                                                                                                                                                          1/1
  Running scriptlet: opensearch-2.4.0-1.x86_64                                                                                                                                                                                                                                                                          1/1
warning: %post(opensearch-2.4.0-1.x86_64) scriptlet failed, exit status 127

Error in POSTIN scriptlet in rpm package opensearch
  Verifying        : opensearch-2.4.0-1.x86_64                                                                                                                                                                                                                                                                          1/1

Installed:
  opensearch-2.4.0-1.x86_64

@peterzhuamazon
Copy link
Member

To clear old repo gpg check key use:

sudo rm -rf /var/cache/dnf/* (or specific to the */pubring or similar)

To clear old rpm imported gpg key:

rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n'
rpm -e <keyid>

@peterzhuamazon
Copy link
Member

peterzhuamazon commented May 3, 2023

Ubuntu 20.04 ok: (IGNORE post install script issues as I am testing on docker without systemd now)

# apt-key list
......
pub   rsa4096 2021-05-11 [SC]
      C5B7 4989 65EF D1C2 924B  A9D5 39D3 1987 9310 D3FC
uid           [ unknown] OpenSearch project <[email protected]>
sub   rsa2048 2021-05-11 [S] [expires: 2024-05-12]
......

# apt update
Get:1 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable InRelease [7535 B]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 Packages [928 B]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Fetched 344 kB in 1s (507 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

# apt install opensearch
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  opensearch
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 748 MB of archives.
After this operation, 976 MB of additional disk space will be used.
Get:1 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 opensearch amd64 2.7.0 [748 MB]
Fetched 748 MB in 9s (84.4 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package opensearch.
(Reading database ... 7978 files and directories currently installed.)
Preparing to unpack .../opensearch_2.7.0_amd64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.7.0) ...
Setting up opensearch (2.7.0) ...
Running OpenSearch Post-Installation Script
/var/lib/dpkg/info/opensearch.postinst: line 56: systemd-tmpfiles: command not found
dpkg: error processing package opensearch (--configure):
 installed opensearch package post-installation script subprocess returned error exit status 127
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Errors were encountered while processing:
 opensearch
E: Sub-process /usr/bin/dpkg returned an error code (1)

@peterzhuamazon
Copy link
Member

peterzhuamazon commented May 3, 2023

Try centos7 ok: (IGNORE errors as there is no systemd here)

                                                                                                                                                       | 679 MB  00:00:08
Retrieving key from file:///<>
Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <[email protected]>"
 Fingerprint: c5b7 4989 65ef d1c2 924b a9d5 39d3 1987 9310 d3fc
 From       : <>
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Failed to get D-Bus connection: Operation not permitted
Failed to get D-Bus connection: Operation not permitted
  Installing : opensearch-2.7.0-1.x86_64                                                                                                                                                                                                                                                                                1/1
Failed to get D-Bus connection: Operation not permitted
warning: %post(opensearch-2.7.0-1.x86_64) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package opensearch-2.7.0-1.x86_64
  Verifying  : opensearch-2.7.0-1.x86_64                                                                                                                                                                                                                                                                                1/1

Installed:
  opensearch.x86_64 0:2.7.0-1

@peterzhuamazon
Copy link
Member

Test verify detached signature on the oldest releases:

# gpg --verify opensearch-1.0.0-linux-x64.tar.gz.sig
gpg: assuming signed data in 'opensearch-1.0.0-linux-x64.tar.gz'
gpg: Signature made Mon Jul 12 19:22:11 2021 UTC
gpg:                using RSA key C2EE2AF6542C03B4
gpg: Good signature from "OpenSearch project <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C5B7 4989 65EF D1C2 924B  A9D5 39D3 1987 9310 D3FC
     Subkey fingerprint: 2187 3199 B103 0FCD 49DA  83F8 C2EE 2AF6 542C 03B4

@peterzhuamazon
Copy link
Member

Update this:
#2040 (comment)

@peterzhuamazon
Copy link
Member

We will publish the new key as part of this issue:

@peterzhuamazon
Copy link
Member

We have released the new key now to public:
https://opensearch.org/verify-signatures.html

Thanks.

@peterzhuamazon
Copy link
Member

Is you try to install rpm without yum, but directly using the rpm package by rpm --import or similar, RHEL9 might not accept the new public key, tho yum accept.

You can always enable the legacy crypto:

  • Switch to legacy crypto policy: update-crypto-policies --set LEGACY OR Explicitly allow SHA-1: update-crypto-policies --set DEFAULT:SHA1
  • rpm --import <key> && dnf install <>
  • update-crypto-policies --set DEFAULT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New Enhancement question Further information is requested rpm signing v2.8.0
Projects
Development

No branches or pull requests

5 participants