[Bug]: RHEL8 yum/dnf repo: Bad GPG signature

[draeath]

Describe the bug

Repository metadata for https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/2.x/yum currently has an apparently invalid GPG signature, as far as DNF on RHEL8 is concerned.

To reproduce

1. Set up the yum repo
2. Import the gpg key with rpm --import
3. Attempt to utilize the repository

Expected behavior

Normal function

Screenshots

No response

Host / Environment

• Red Hat Enterprise Linux release 8.7 (Ootpa)
• x86_64

Additional context

The repomd.xml i know to find (https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/yum/repodata/repomd.xml) appears to have a good signature from the appropriate key? I have no idea what's wrong, unless there's a different repo metadata file DNF is complaining about (i wish it would give URLs when complaining, lol)

draeath@ginnungagap:~> wget https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/yum/repodata/repomd.xml https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/yum/repodata/repomd.xml.asc https://artifacts.opensearch.org/publickeys/opensearch.pgp
# output snipped

draeath@ginnungagap:~> file opensearch.pgp repomd.xml repomd.xml.asc 
opensearch.pgp: PGP public key block Public-Key (old)
repomd.xml:     XML 1.0 document, ASCII text
repomd.xml.asc: PGP signature Signature (old)

draeath@ginnungagap:~> gpg --import ./opensearch.pgp
# output snipped

draeath@ginnungagap:~> gpg --verify repomd.xml.asc
gpg: assuming signed data in 'repomd.xml'
gpg: Signature made Mon 01 May 2023 03:57:43 PM EDT
gpg:                using RSA key C2EE2AF6542C03B4
gpg: Good signature from "OpenSearch project <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C5B7 4989 65EF D1C2 924B  A9D5 39D3 1987 9310 D3FC
    Subkey fingerprint: 2187 3199 B103 0FCD 49DA  83F8 C2EE 2AF6 542C 03B4

draeath@ginnungagap:~> cat opensearch.pgp | gpg --with-colons --import-options show-only --import
pub:-:4096:1:39D319879310D3FC:1620766954:::f:::scSC::::::23::0:
fpr:::::::::C5B7498965EFD1C2924BA9D539D319879310D3FC:
uid:-::::1620766954::09D151299DD6D3D114D0A3558C1AB8329B4AE259::OpenSearch project <[email protected]>::::::::::0:
sub:-:2048:1:C2EE2AF6542C03B4:1620767456:1715546574:::::s::::::23:
fpr:::::::::21873199B1030FCD49DA83F8C2EE2AF6542C03B4:

Relevant log output

[root@REDACTED ~]# dnf --verbose --disablerepo=* --enablerepo=opensearch-2.x makecache
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, diff, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
DNF version: 4.7.0
cachedir: /var/cache/dnf
Making cache files for all metadata files.
opensearch-2.x: has expired and will be refreshed.
repo: downloading from remote: opensearch-2.x
OpenSearch 2.x                                                  1.2 kB/s | 498  B     00:00    
OpenSearch 2.x                                                  122 kB/s | 4.2 kB     00:00    
repo opensearch-2.x: 0x39D319879310D3FC already imported
OpenSearch 2.x                                                  5.6 kB/s | 498  B     00:00    
Error: Failed to download metadata for repo 'opensearch-2.x': repomd.xml GPG signature verification error: Bad GPG signature

[draeath]

My local repo config, should it matter or be incorrect:

[opensearch-2.x]
name=OpenSearch 2.x
baseurl=https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/yum
enabled=1
repo_gpgcheck=1
gpgcheck=1
gpgkey=https://artifacts.opensearch.org/publickeys/opensearch.pgp
type=rpm-md

This was working as-of at least a few weeks ago. The host currently has opensearch-2.7.0-1.x86_64 installed, if that helps narrow down when the repo broke.

[draeath]

Oh, and we are not using FIPS mode (and can't, it breaks things on us)

[root@REDACTED ~]# fips-mode-setup --check
Installation of FIPS modules is not completed.
FIPS mode is disabled.

[peterzhuamazon]

Hi @draeath not sure why it is showing as invalid on your end.
It is showing as a good signature.

gpg: Good signature from "OpenSearch project <[email protected]>" [unknown]

As for the yum installation I just did a clean try on a centos8, not seeing issues.

# yum install opensearch
Last metadata expiration check: 0:00:35 ago on Thu 18 May 2023 04:50:45 PM UTC.
Dependencies resolved.
============================================================================================================================================================================================================================================================================================================================
 Package                                                                       Architecture                                                              Version                                                                    Repository                                                                         Size
============================================================================================================================================================================================================================================================================================================================
Installing:
 opensearch                                                                    x86_64                                                                    2.7.0-1                                                                    opensearch-2.x                                                                    679 M

Transaction Summary
============================================================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 679 M
Installed size: 936 M
Is this ok [y/N]: y
Downloading Packages:
opensearch-2.7.0-linux-x64.rpm                                                                                                                                                                                                                                                              117 MB/s | 679 MB     00:05
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                                                                       117 MB/s | 679 MB     00:05
OpenSearch 2.x                                                                                                                                                                                                                                                                              131 kB/s | 4.2 kB     00:00
Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <[email protected]>"
 Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC
 From       : https://artifacts.opensearch.org/publickeys/opensearch.pgp
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                                                    1/1
  Running scriptlet: opensearch-2.7.0-1.x86_64                                                                                                                                                                                                                                                                          1/1
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

  Installing       : opensearch-2.7.0-1.x86_64 [=====

Thanks.

[peterzhuamazon]

# dnf --verbose --disablerepo=* --enablerepo=opensearch-2.x makecache
DNF version: 4.7.0
cachedir: /var/cache/dnf
Unknown configuration option: autorefresh = 1 in /etc/yum.repos.d/opensearch-2.x.repo
Making cache files for all metadata files.
opensearch-2.x: has expired and will be refreshed.
OpenSearch 2.x                                                                                                                                                                                                                                                                              4.3 kB/s | 498  B     00:00
reviving: 'opensearch-2.x' can be revived - repomd matches.
opensearch-2.x: using metadata from Mon 01 May 2023 07:57:06 PM UTC.
User-Agent: constructed: 'libdnf (CentOS Linux 8; generic; Linux.x86_64)'
Metadata cache created.

[peterzhuamazon]

CentOS Linux release 8.5.2111
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
CentOS Linux release 8.5.2111
CentOS Linux release 8.5.2111

[peterzhuamazon]

Can you do these things to clean your repo key and rpm key before trying again?

sudo yum clean all
sudo rm -rf /var/cache/dnf/*
rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n' 
rpm -e <keyid of opensearch>
sudo yum repolist

Then run your yum or dnf to install or cache to local

Thanks.

[draeath]

It's working now. Either that fixed it, or it was a transient issue?