The latest CLOUDHV_EFI.fd can not start aarch64 VM

[peng6662001]

Describe the bug
I reference docs/uefi.md to build cloudhv_efi.fd,then the VM starts fail
I reference scripts/common-aarch64.sh to rebuild cloudhv_efi.fd and run the VM with the same command,the VM can work.

VM configuration

./cloud-hypervisor --kernel CLOUDHV_EFI.fd --disk path=../focal-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img --cmdline 'root=/dev/vda1 console=ttyS0 console=hvc0 rw' --cpus boot=1 --mem
ory size=2G --serial tty --console off -v

Logs

Linux kernel output:
log_fail.txt
log_good.txt

[michael2012z]

Ping @jongwu, could you take a look?

[jongwu]

Hi @peng6662001 -, I build the latest edk2 and it works well. Can you give the uefi boot log here?

[jongwu]

As the docs direct you build the release version of edk2, you should rebuild it to "DEBUG" version to print the log. Use the followling cmdline instead of the last line in the doc.

$ build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtCloudHv.dsc -b DEBUG

[peng6662001]

@jongwu Please check Log.zip,I uploaded the build log and VM log
Log.zip

[jongwu]

Hi @peng6662001 -, I see serval grub files installed in your disk. Can you remove them and reinstall grub then try again?

[peng6662001]

Hi @jongwu ,I have only one grub file now,but the build always fails,I can't fix it.
Please check blog_0217.txt,how to print more build log?
blog_0217.txt

[peng6662001]

Hi @jongwu ,I can still build success when I follow scripts/common-aarch64.sh

[jongwu]

Hi @peng6662001 -, can you build the latest edk2 now? Once done, you can run clh again. Make sure your guest image just has one grub under /boot/efi/ as I saw lots of grub files in your rlog:

FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
FSOpen: Open 'EFI' Success
FSOpen: Open 'ubuntu' Success
FSOpen: Open 'BOOTAA64.CSV' Success
FSOpen: Open '\EFI\ubuntu\BOOTAA64.CSV' Success
FSOpen: Open '\EFI\ubuntu\shimaa64.efi' Success

[peng6662001]

@jongwu Do you know how to fix the build error in blog_0217.txt:
/home/dom/cloud-hypervisor/uefi/edk2/ArmVirtPkg/CloudHvPlatformHasAcpiDtDxe/CloudHvHasAcpiDtDxe.inf(39): error 4000: Value of Guid [gUefiOvmfPkgTokenSpaceGuid] is not found under [Guids] section in
/home/dom/cloud-hypervisor/uefi/edk2/ArmVirtPkg/ArmVirtPkg.dec
/home/dom/cloud-hypervisor/uefi/edk2/EmbeddedPkg/EmbeddedPkg.dec
/home/dom/cloud-hypervisor/uefi/edk2/MdeModulePkg/MdeModulePkg.dec
/home/dom/cloud-hypervisor/uefi/edk2/MdePkg/MdePkg.dec

[jongwu]

Hi @peng6662001 -, It's a bug of the latest edk2. I will fix it. For now, try the following patch:

`diff --git a/ArmVirtPkg/CloudHvPlatformHasAcpiDtDxe/CloudHvHasAcpiDtDxe.inf b/ArmVirtPkg/CloudHvPlatformHasAcpiDtDxe/CloudHvHasAcpiDtDxe.inf
index 7cad40e11f..6ff80b24d0 100644
--- a/ArmVirtPkg/CloudHvPlatformHasAcpiDtDxe/CloudHvHasAcpiDtDxe.inf
+++ b/ArmVirtPkg/CloudHvPlatformHasAcpiDtDxe/CloudHvHasAcpiDtDxe.inf
@@ -23,6 +23,7 @@
EmbeddedPkg/EmbeddedPkg.dec
MdeModulePkg/MdeModulePkg.dec
MdePkg/MdePkg.dec
+OvmfPkg/OvmfPkg.dec

[LibraryClasses]
BaseLib
`

[peng6662001]

Hi @peng6662001 -, can you build the latest edk2 now? Once done, you can run clh again. Make sure your guest image just has one grub under /boot/efi/ as I saw lots of grub files in your rlog:

FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
FSOpen: Open 'EFI' Success
FSOpen: Open 'ubuntu' Success
FSOpen: Open 'BOOTAA64.CSV' Success
FSOpen: Open '\EFI\ubuntu\BOOTAA64.CSV' Success
FSOpen: Open '\EFI\ubuntu\shimaa64.efi' Success

@jongwu Please check new log
log_vm_1702.txt
I backup /boot/efi in /home/cloud/efi and remain some necessary files in /boot/efi.
My disk is focal-server-cloudimg-arm64.raw.

root@cloud:/home/cloud# find efi
efi
efi/EFI
efi/EFI/ubuntu
efi/EFI/ubuntu/shimaa64.efi
efi/EFI/ubuntu/grubaa64.efi
efi/EFI/ubuntu/mmaa64.efi
efi/EFI/ubuntu/grub.cfg
efi/EFI/ubuntu/BOOTAA64.CSV
efi/EFI/BOOT
efi/EFI/BOOT/BOOTAA64.EFI
efi/EFI/BOOT/mmaa64.efi
efi/EFI/BOOT/fbaa64.efi
root@cloud:/home/cloud# 
root@cloud:/home/cloud# 
root@cloud:/home/cloud# find /boot/efi/
/boot/efi/
/boot/efi/EFI
/boot/efi/EFI/BOOT
/boot/efi/EFI/BOOT/BOOTAA64.EFI
/boot/efi/EFI/BOOT/grubaa64.efi
/boot/efi/EFI/ubuntu
/boot/efi/EFI/ubuntu/shimaa64.efi
/boot/efi/EFI/ubuntu/BOOTAA64.CSV
/boot/efi/EFI/ubuntu/grub.cfg

[jongwu]

Hi @peng6662001 -, I encounter a similar issue when start ubuntu:22.04 based on clh/qemu:

Loading driver at 0x0007C2C9000 EntryPoint=0x0007C2E7000
Loading driver at 0x0007C2C9000 EntryPoint=0x0007C2E7000
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7EE40918
ProtectUefiImageCommon - 0x7E9AA040
0x000000007C2C9000 - 0x00000000000D7000
SetUefiImageMemoryAttributes - 0x000000007C2C9000 - 0x000000000001E000 (0x0000000000004008)
SetUefiImageMemoryAttributes - 0x000000007C2E7000 - 0x0000000000067000 (0x0000000000020008)
SetUefiImageMemoryAttributes - 0x000000007C34E000 - 0x0000000000052000 (0x0000000000004008)
InstallProtocolInterface: 605DAB50-E046-4300-ABB6-3DD810DD8B23 7C37C9A0
FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
FSOpen: Open '\EFI\BOOT\fbaa64.efi' Success
SetMemoryAttributes: BaseAddress == 0x7C297000, Length == 0x19000, Attributes == 0x4000
ClearMemoryAttributes: BaseAddress == 0x7C297000, Length == 0x19000, Attributes == 0x22000

Synchronous Exception at 0x000000007C29B000

Synchronous Exception at 0x000000007C29B000
PC 0x00007C29B000
PC 0x00007C2E93F4
PC 0x00007C2E9608
PC 0x00007C2EA32C
PC 0x00007C2E7030
PC 0x00007FAA276C (0x00007FA9B000+0x0000776C) [ 1] DxeCore.dll
PC 0x00007F7E6080 (0x00007F7D8000+0x0000E080) [ 2] BdsDxe.dll
PC 0x00007F7DAA00 (0x00007F7D8000+0x00002A00) [ 2] BdsDxe.dll
PC 0x00007F7E2360 (0x00007F7D8000+0x0000A360) [ 2] BdsDxe.dll
PC 0x00007FAA5ED4 (0x00007FA9B000+0x0000AED4) [ 3] DxeCore.dll
[ 1] /root/jong/firm/Build/ArmVirtCloudHv-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
[ 2] /root/jong/firm/Build/ArmVirtCloudHv-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/BdsDxe/BdsDxe/DEBUG/BdsDxe.dll
[ 3] /root/jong/firm/Build/ArmVirtCloudHv-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll

X0 0x000000007EE5D798 X1 0x000000007FA50018 X2 0x000000007C29B000 X3 0x0000000000000000
...
ESR 0x8600000F FAR 0x000000007C29B000

ESR : EC 0x21 IL 0x1 ISS 0x0000000F

Instruction abort: Permission fault, third level

...

ASSERT [ArmCpuDxe] /root/jong/firm/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333): ((BOOLEAN)(0==1))

If this is also your case, here is a workaround in edk2:

`diff --git a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibCore.c b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibCore.c
index 7ed758fbbc..4349f2828d 100644
--- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibCore.c
+++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibCore.c
@@ -529,6 +529,7 @@ ArmSetMemoryRegionNoExec (
{
UINT64 Val;

+return EFI_SUCCESS;^M
if (ArmReadCurrentEL () == AARCH64_EL1) {
Val = TT_PXN_MASK | TT_UXN_MASK;
} else {`

[jongwu]

It maybe a bug for ubuntu shim. I try to rebuild ubuntu shim and replace "/boot/efi/EFI/BOOT/BOOTAA64.EFI" to fix it locally.

[peng6662001]

@jongwu I built CLOUDHV_EFI.fd with the latest code,the vm worked well with focal-server-cloudimg-arm64-custom-20210929-0.raw,but it can't run with jammy-server-cloudimg-arm64-custom-20220329-0.raw,could you please take a look?

[jongwu]

Can you try the edk2 patch above?

[peng6662001]

@jongwu I can't find ArmSetMemoryRegionNoExec in ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibCore.c

[jongwu]

Hi @peng6662001 -, I think this issue is no related with Cloud Hypervisor or edk2. There is a same issue for Qemu. I think this bug derives from UEFI shim. I can find a workaround in edk2 but it's meaningless.

I raise an issue in shim. Hope to get some feedback there.

[rbradford]

Looks like this issue wasn't related to CH in the end.