Flatcar fails to install current (GPG signature check failed)

[Scarjit]

Description

flatcar-install fails to install flatcar, using the "stable" release channel and the "current" version.
The error occurs due to a missing public key for the image signer ([email protected] @ E9426D8B67E35DF476BD048185F7C8868837E271)

Impact

This bug prevents us and our customers from installing our stack, in all cases where we use "stable"/"current" instead of our pinned older version (3227.2.2).

Environment and steps to reproduce

1. Set-up: We use iPXE and matchbox to deploy our stack, but it isn't required a simple flatcar-install -d /dev/sda -C stable -V current -v is enough to replicate the bug.
2. Task: This happens while the installer is verifing the installer img.
3. Action(s):
   a. Our iPXE chainloads 3227.2.2 as stage-0
   b. stage-0 executes flatcar-install -d /dev/sda -C stable -V current -v
4. Error: GPG failed to verify the image, due to a missing public key

Expected behavior

The key for the most recent stable version should be valid and available.

Additional information

stage-0 /etc/os-release

NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3227.2.2
VERSION_ID=3227.2.2
BUILD_ID=2022-08-29-1855
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3227.2.2 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3227.2.2:*:*:*:*:*:*:*"

Install output (this is from a manual start, instead of the service, but the error remains the same):

flatcar-0-install / # flatcar-install \
  -d /dev/sda \
  -C stable \
  -V current \
  -v
+ getopts V:B:C:DI:d:o:c:e:i:t:b:k:f:nsyvh OPTION
+ [[ -z '' ]]
+ [[ -z /dev/sda ]]
+ [[ -n '' ]]
++ lsblk -n -d -o TYPE /dev/sda
+ [[ disk =~ ^(disk|mpath|loop|lvm)$ ]]
+ [[ ! -w /dev/sda ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ _disk_status=
++ mktemp --tmpdir -d flatcar-install.XXXXXXXXXX
+ WORKDIR=/tmp/flatcar-install.UFcghRSc9b
+ trap 'error_output ; is_modified && wipefs --all --backup "${DEVICE}" ; rm -rf "${WORKDIR}"' EXIT
+ [[ '' = \t\r\u\e ]]
+ '[' -n '' ']'
+ install_from_url
+ prep_url
+ IMAGE_NAME=flatcar_production_image.bin.bz2
+ [[ -n '' ]]
+ [[ -n 1 ]]
+ [[ -z 1 ]]
+ [[ current =~ ^(alpha|beta|stable|edge)$ ]]
+ [[ -z '' ]]
+ BASE_URL=https://stable.release.flatcar-linux.net/amd64-usr
+ [[ current == \c\u\r\r\e\n\t ]]
+ local VERSIONTXT_URL=https://stable.release.flatcar-linux.net/amd64-usr/current/version.txt
++ wget --tries 10 --timeout=20 --retry-connrefused -qO- https://stable.release.flatcar-linux.net/amd64-usr/current/version.txt
++ sed -n 's/^FLATCAR_VERSION=//p'
+ VERSION_ID=3510.2.6
+ [[ -z 3510.2.6 ]]
+ echo 'Current version of Flatcar Container Linux stable is 3510.2.6'
Current version of Flatcar Container Linux stable is 3510.2.6
+ IMAGE_URL=https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2
+ SIG_NAME=flatcar_production_image.bin.bz2.sig
+ SIG_URL=https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2.sig
+ wget --tries 10 --timeout=20 --retry-connrefused --spider --quiet https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2
+ wget --tries 10 --timeout=20 --retry-connrefused --spider --quiet https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2.sig
+ export GNUPGHOME=/tmp/flatcar-install.UFcghRSc9b/gnupg
+ GNUPGHOME=/tmp/flatcar-install.UFcghRSc9b/gnupg
+ mkdir -p /tmp/flatcar-install.UFcghRSc9b/gnupg
+ '[' -n '' ']'
+ gpg --batch --quiet --import
+ echo 'Downloading the signature for https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2...'
Downloading the signature for https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2...
+ wget --tries 10 --timeout=20 --retry-connrefused --no-verbose -O /tmp/flatcar-install.UFcghRSc9b/flatcar_production_image.bin.bz2.sig https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2.sig
2023-08-28 15:09:05 URL:https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2.sig [594/594] -> "/tmp/flatcar-install.UFcghRSc9b/flatcar_production_image.bin.bz2.sig" [1]
+ VERSION_SUMMARY+=' stable 3510.2.6'
+ echo 'Downloading, writing and verifying flatcar_production_image.bin.bz2...'
Downloading, writing and verifying flatcar_production_image.bin.bz2...
++ write_to_disk
++ mkfifo -m 0600 /tmp/flatcar-install.UFcghRSc9b/disk_modified
+ wget --tries 10 --timeout=20 --retry-connrefused --no-verbose -O - https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2
+ gpg --batch --trusted-key E25D9AED0593B34A --verify /tmp/flatcar-install.UFcghRSc9b/flatcar_production_image.bin.bz2.sig -
+ tee /dev/fd/62
++ lbzip2 -cd
++ trap '(exec 2>/dev/null ; echo done > "${WORKDIR}/disk_modified") &' RETURN
+++ blockdev --getsz /dev/sda
++ dd conv=nocreat count=1024 if=/dev/zero of=/dev/sda seek=67107840 status=none
++ dd bs=1M conv=nocreat of=/dev/sda status=none
2023-08-28 15:09:19 URL:https://stable.release.flatcar-linux.net/amd64-usr/3510.2.6/flatcar_production_image.bin.bz2 [383304086/383304086] -> "-" [1]
gpg: Signature made Mon Aug  7 16:50:35 2023 UTC
gpg:                using RSA key E9426D8B67E35DF476BD048185F7C8868837E271
gpg:                issuer "[email protected]"
gpg: Can't check signature: No public key
+ EEND=('0' '0' '2')
+ local EEND
+ '[' 0 -ne 0 ']'
+ '[' 0 -ne 0 ']'
+ '[' 2 -ne 0 ']'
+ echo '2: GPG signature verification failed for flatcar_production_image.bin.bz2'
2: GPG signature verification failed for flatcar_production_image.bin.bz2
+ exit 1
+ error_output
+ echo 'Error: return code 1 from exit 1'
Error: return code 1 from exit 1
+ is_modified
+ [[ -e /tmp/flatcar-install.UFcghRSc9b/disk_modified ]]
+ wipefs --all --backup /dev/sda
/dev/sda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sda: calling ioctl to re-read partition table: Success
+ rm -rf /tmp/flatcar-install.UFcghRSc9b
++ udevadm settle
flatcar-0-install / # ++ local try
++ for try in 0 1 2 4
++ sleep 0
++ blockdev --rereadpt /dev/sda
++ unset try
++ break
++ '[' -z '' ']'
++ udevadm settle
+++ exec
^C

[Scarjit]

I have now tried the following combinations:

stage-0 (current) & to-install (current):
works

stage-0 (3227.2.2) & to-install (3227.2.2):
works

stage-0 (3227.2.2) & to-install (current):
does not work

[pothos]

That's expected, 3510.2.6 is the latest Stable version and the outdated 3227.2.2 doesn't have the key. A workaround for old images would be to first download flatcar-install from https://raw.githubusercontent.com/flatcar/init/flatcar-master/bin/flatcar-install where the new key is embedded. That said, why do you want to use this old version? What prevents you from using a newer version or LTS?

[Scarjit]

We use 3227.2.2 primarly since it has undergone lots of internal testing.
But ill try to move our default install to lts instead.

[jepio]

flatcar-install also supports supplying the gpg key to use for verification. https://github.com/flatcar/init/blob/flatcar-master/bin/flatcar-install#L105