Missing /boot/efi/EFI/almalinux/grubenv

[gcoretech]

I migrated a server running CentOS 8.3 to AlmaLinux 8.3.
After the final reboot the server was unreachable.

It seems /boot/grub2/grubenv is not a file anymore but a softlink pointing to /boot/efi/EFI/centos/grubenv. This has been changed to /boot/efi/EFI/almalinux/grubenv during migration but the directory /boot/efi/EFI/almalinux/ does not contain a file named grubenv afterwards.

[root@server ~]# ls -l /boot/grub2/grubenv 
lrwxrwxrwx 1 root root 28 Mar 15 19:28 /boot/grub2/grubenv -> ../efi/EFI/almalinux/grubenv

[root@server ~]# ls -l /boot/efi/EFI/almalinux/grubenv
ls: cannot access '/boot/efi/EFI/almalinux/grubenv': No such file or directory

The server booted again after copying /boot/efi/EFI/centos/grubenv to /boot/efi/EFI/almalinux/grubenv

(Boot via BIOS, not UEFI)

[andrewlukoshko]

Hello.
File /boot/efi/EFI/almalinux/grubenv belongs to grub2-efi-x64-2.02-90.el8_3.1.alma.x86_64 package.
Was it installed during migration?

[gcoretech]

Hello Andrew,
yes, that package was installed:

[root@server ~]# rpm -qi grub2-efi-x64-2.02-90.el8_3.1.alma.x86_64
Name        : grub2-efi-x64
Epoch       : 1
Version     : 2.02
Release     : 90.el8_3.1.alma
Architecture: x86_64
Install Date: Wed 07 Apr 2021 01:19:58 PM CEST

[root@server ~]# rpm -ql grub2-efi-x64-2.02-90.el8_3.1.alma.x86_64
/boot/efi/EFI/almalinux/fonts
/boot/efi/EFI/almalinux/grub.cfg
/boot/efi/EFI/almalinux/grubenv
/boot/efi/EFI/almalinux/grubx64.efi
/boot/grub2/grubenv
/boot/loader/entries
/etc/grub2-efi.cfg

Maybe grubenv is generated, the rpm does not contain the file:

[root@server tmp]# curl -O http://repo.almalinux.org/almalinux/8.3/BaseOS/x86_64/os/Packages/grub2-efi-x64-2.02-90.el8_3.1.alma.x86_64.rpm

[root@server tmp]# rpm2cpio < grub2-efi-x64-2.02-90.el8_3.1.alma.x86_64.rpm | cpio -id

[root@server tmp]# ls -l boot/efi/EFI/almalinux/
total 1860
drwx------ 2 root root    4096 Apr  7 19:55 fonts
-rwx------ 1 root root 1900112 Apr  7 19:55 grubx64.efi

[gcoretech]

spec file says:
%ghost %config(noreplace) /boot/grub2/grubenv

This defines the file as owned by the package but it is not included

[andrewlukoshko]

Thank you I'm looking into it

[andrewlukoshko]

What did you actually mean by "Boot via BIOS, not UEFI" in first message?
You had to switch to BIOS after migration or your system boots in BIOS mode always?

I wasn't able to reproduce the issue with CentOS 8.3 which has latest updates installed:

[root@localhost almalinux]# pwd
/boot/efi/EFI/almalinux
[root@localhost almalinux]# ls -l
total 5444
-rwx------. 1 root root     122 Mar 15 21:44 BOOTX64.CSV
drwx------. 2 root root    4096 Mar 15 21:28 fonts
-rwx------. 1 root root    6544 Apr  7 21:50 grub.cfg
-rwx------. 1 root root    1024 Apr  7 21:50 grubenv
-rwx------. 1 root root 1900112 Mar 15 21:28 grubx64.efi
-rwx------. 1 root root 1171320 Mar 15 21:44 mmx64.efi
-rwx------. 1 root root 1240144 Mar 15 21:44 shimx64-almalinux.efi
-rwx------. 1 root root 1237503 Mar 15 21:44 shimx64.efi

I'll continue testing.

[gcoretech]

I wanted to say the server was not started using an EFI partition:

[root@server ~]# efibootmgr 
EFI variables are not supported on this system.

I'll see if I can reinstall another server and try again

[gcoretech]

I've reinstalled a testserver with CentOS 8.3 and even tried with CentOS 8.2 + upgrade to CentOS 8.3 before migrating to AlmaLinux and I could not reproduce the problem.

This testserver has harddisks whereas the previous server had NVMe disks only. Unfortunately I cannot reinstall the NVMe server, but on a similar hardware with CentOS 8.3 I noticed the following:

[root@server2 ~]# ls -l /boot/grub2/grub*
-rw-r--r--. 1 root root 8923 Jan 22 15:58 /boot/grub2/grub.cfg
-rw-------. 1 root root 1024 Mar 24 20:48 /boot/grub2/grubenv
lrwxrwxrwx  1 root root   25 Sep  8  2020 /boot/grub2/grubenv.rpmnew -> ../efi/EFI/centos/grubenv

[root@server2 ~]# ls -l /boot/efi/EFI/centos/grubenv
ls: cannot access '/boot/efi/EFI/centos/grubenv': No such file or directory

There is a grubenv.rpmnew file that is linked to efi/EFI/centos/grubenv whereas grubenv itself is a plain file, not a softlink.

Could it be possible the problem occurs with NVMe disks only?
Maybe even triggered by a CentOS update (because of the .rpmnew file)?

[fwarnke]

I am able to reproduce the problem on a server using UEFI to boot a 4TB SSD. Did you boot in to maint mode to make the above change? I must be doing something wrong because that did not work for me.

[OliverKellyATech]

I have a similar issue on a UEFI VM, where my /boot folders and associated symlink sare similarly mixed up between centos and almalinux folder. See #31 (comment)

My /boot/grub2/grubenv symlink links to /boot/efi/EFI/centos/grubenv, which is non-exsistent:

# ls -l /boot/grub2/
total 0
lrwxrwxrwx 1 root root 28 Apr 22 17:46 grubenv -> /boot/efi/EFI/centos/grubenv

# stat /boot/efi/EFI/centos/grubenv
stat: cannot stat '/boot/efi/EFI/centos/grubenv': No such file or directory

# stat /boot/efi/EFI/almalinux/grubenv
  File: /boot/efi/EFI/almalinux/grubenv
  Size: 1024      	Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d	Inode: 9           Links: 1
Access: (0700/-rwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-04-22 04:52:54.000000000 +1000
Modify: 2021-04-12 12:08:52.000000000 +1000
Change: 2021-04-12 12:08:52.000000000 +1000
 Birth: -

[andrewlukoshko]

Is problem still here with 8.4?

[gcoretech]

It took some time to get a new server with nvme disks.
The conversion from CentOS 8.4 to AlmaLinux 8.4 works flawlessly.
Softlinks / boot files are set correctly now. Thanks!