HAOS for ARM/AArch64 CPU (ESXi Arm Edition)

[shirou93]

Why not release hassos ova for arm? VMware released ESXi Arm Edition and RPI4 8gb make sense.

Is it possible convert rpi.img to working vmdk?

[agners]

@shirou93 patch welcome 😄 Yeah you can convert the disk image to a vmdk and try create a machine which boots. The utility qemu-img can do that.

[ipha]

I tried building a generic arm vm image, but I couldn't get a bootloader working. Best I can tell barebox doesn't support arm EFI.

GRUB could be an option, but I don't know if it supports the A/B partitions.

[shirou93]

@ipha @agners i try but not booting. Maybe bootloader was broken when convert img to vmdk.

[agners]

Hm, yeah bootloader is a problem: We use U-Boot in all Arm builds. It seems that ESXi Arm is expecting a UEFI bootloader such as Grub2. Gleening at this blog post it seems that booting something else is possible in theory, but it seems that requires some hacking: https://blogs.vmware.com/arm/2020/10/23/uefi-less-vms-with-esxi-arm-fling-1-1/

[myericbaba]

anyone try to follow this guideline for HA?
https://www.virtuallyghetto.com/2020/10/how-to-run-raspberry-pi-os-as-a-vm-on-esxi-arm.html

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Grandma-Betty]

I am interested in this too. Would be so nice to run HA OS on ESXi ARM!
Edit: Made the feature request here:
https://community.home-assistant.io/t/support-haos-for-vmware-esxi-arm/365534

[agners]

As noted in #1549, there are now images named generic-aarch64 which support UEFI boot on arm64. I haven't tested them on ESXi for ARM, I am keen to here if the image also works on ESXi.

There are development builds available at https://os-builds.home-assistant.io/8.0.dev20220223/.

[Grandma-Betty]

As noted in #1549, there are now images named generic-aarch64 which support UEFI boot on arm64. I haven't tested them on ESXi for ARM, I am keen to here if the image also works on ESXi.

There are development builds available at https://os-builds.home-assistant.io/8.0.dev20220223/.

Just tried it with the same settings as HAOS runs on ESXi-ARM, without any luck.
I have used the .vmdk disk image as well as the .img file from source.
I also tried to play around with hard disk controller settings too, no luck.
It's not booting.

[agners]

@Grandma-Betty you did use the generic-aarch64 images right? Did anything come up at all (Grub boot loader maybe?)

[Grandma-Betty]

Yes, I used the said images.
No, it was not even reaching a boot loader. With the .vmdk image I was not even able to start the VM itself from the Hypervisor.
With the .img file the VM did of course start but was waiting forever for PXE boot.

[agners]

Can you configure the UEFI BIOS in ESXi somehow? Maybe secure boot or something?

[tweetysat]

Also tired with both (latest) generic vmdk and ova vmdk.
After adding the disk, changed sata to ide and verified that the boot is EFI, the vm is not starting.
It tells me : Failed - Disconnected from virtual machine.
And :

Power On VM
Key
haTask-2-vim.VirtualMachine.powerOn-4294853845

Description
Power On this virtual machine

Virtual machine
HomeAssistant OS
State
Failed - Disconnected from virtual machine.

Errors
Disconnected from virtual machine.
Failed to establish transport connection (9): There is no VMware process running for config file /vmfs/volumes/5f4e56d3-d3071bfa-bf11-dca632e04cec/HomeAssistant OS/HomeAssistant OS.vmx.
Remote disconnected

[Grandma-Betty]

Yes, I had the same issue. Just forgot to report back, apologies @agners.

[Grandma-Betty]

Hm, yeah bootloader is a problem: We use U-Boot in all Arm builds. It seems that ESXi Arm is expecting a UEFI bootloader such as Grub2. Gleening at this blog post it seems that booting something else is possible in theory, but it seems that requires some hacking: https://blogs.vmware.com/arm/2020/10/23/uefi-less-vms-with-esxi-arm-fling-1-1/

Any news on this? Seems pretty interesting!

[agners]

No updates, I don't plan to look into ESXi Arm myself.

Home Assistant OS uses regular GRUB2 built for aarch64. The bootloader is present on the ESP (EFI System Partition) as it is common in UEFI boot flows. ESXi really should be able to pick up that GRUB2 and start booting, it's unclear to me why it can't.

Unfortunately I haven't seen a useful log so far. If that is really all ESXi is logging, then its pretty useless to be honest 😢

I recommend using KVM, as it works and offers also logging which helps us developer to debug the problem. Plus its open source 💪

[Grandma-Betty]

@agners Well, I was able to get the time now to try again. I located more VM specific logfiles which I attached. Maybe this gives you any hints in any way?
I was trying to boot with the HAOS .img file btw. (latest version from here)
Thank you!
vmware.log
vmware-1.log
vmware-2.log
vmware-3.log

[agners]

Are those multiple boots? In log 1-3 it seems to crash similarly every time:

2022-05-12T11:37:22.420Z| vcpu-0| I125: VIDE: Register ide0:0) vscsiID = 0, channel = 56
2022-05-12T11:37:22.420Z| vcpu-0| W115: [0 ] bad0003          [1 ] fffffccf92e0     [2 ] 0                [3 ] 427f00000000    
2022-05-12T11:37:22.420Z| vcpu-0| W115: [4 ] 0                [5 ] 128cfd           [6 ] 451a07ca1900     [7 ] ffffffffff758116
2022-05-12T11:37:22.420Z| vcpu-0| W115: [8 ] 2                [9 ] 5                [10] 2                [11] 451a1a4a19e8    
2022-05-12T11:37:22.420Z| vcpu-0| W115: [12] 451a07ca19e8     [13] 420040800080     [14] 41ffd0dd7c08     [15] 8               
2022-05-12T11:37:22.420Z| vcpu-0| W115: [16] 1                [17] 4                [18] 3                [19] fffffcca12a8    
2022-05-12T11:37:22.420Z| vcpu-0| W115: [20] 0                [21] 0                [22] fffffcca1208     [23] fffffc044500    
2022-05-12T11:37:22.420Z| vcpu-0| W115: [24] 0                [25] fffffc044000     [26] fffffcca1208     [27] 0               
2022-05-12T11:37:22.420Z| vcpu-0| W115: [28] 0                [29] fffffc407f90     [30] fffffc04448c    
2022-05-12T11:37:22.420Z| vcpu-0| W115: pc fffffc0444e4 sp fffffc407f30 cpsr 40000248
2022-05-12T11:37:22.420Z| vcpu-0| W115: (relative to vmm .text fffffc000000: pc 444e4 x30 4448c)
2022-05-12T11:37:22.420Z| vcpu-0| W115: Backtrace 'ASSERT':
2022-05-12T11:37:22.420Z| vcpu-0| W115: [ 0] fffffc009324
2022-05-12T11:37:22.420Z| vcpu-0| W115: MONITOR PANIC: vcpu-0:VERIFY devices/vide/iovmk/videVMK-vmkio.c:1904
2022-05-12T11:37:22.420Z| vcpu-0| I125: Core dump with build build-19076756

From the logs, it seems there are two disks attached?

...
ide0:0.fileName = "HAOS_ARM_test.vmdk"
...
sata0:1.fileName = "/vmfs/volumes/6203b160-68b10596-af83-ac1f6b96bfb0/haos_generic-aarch64-8.0.dev20220510.img"
...

Can you try attaching just the vmdk and just the img once? Ideally, as a S-ATA drive, but it probably doesn't matter.

[Grandma-Betty]

Yes, those are multiple boots. I tried to boot with IDE controller first, but the VM does not even start with IDE. Seems like the "Other 4.x or later Linux (64-bit)" template does not support IDE controllers:

At last I tried SATA controller and then the VM started but never reached the GRUB2 bootloader.
The latest logs should be with the SATA controller setting.

[Grandma-Betty]

Wait, let me create the VM again and only boot once with the correct settings. I'll post the new logs in a minute.

[agners]

The last log (vmware-3.log) does has that vdmk file still, I am not sure what that is/if that interferes maybe:

...
ide0:0.fileName = "HAOS_ARM_test.vmdk"
...

Is HAOS_ARM_test the name of the VM? Is that something automatically generated?

[Grandma-Betty]

HAOS_ARM_test is the name I have given manually to the VM.
Here are the new logs after creating the VM and only booting once:
vmware.log

[Grandma-Betty]

@agners Does this image already include open-vm-tools and your latest NVMe driver commits?

https://os-builds.home-assistant.io/9.0.dev20220518/

EDIT: I tested it now. NVMe controller indeed works now in my case but open-vm-tools seems not to be recognized by ESXi yet:

Also all features that open-vm-tools are providing are not working so it's not implemented correctly yet.

[Grandma-Betty]

Tested the latest dev image from here:
https://os-builds.home-assistant.io/9.0.dev20220519/
Still the same as above. NVMe works, open-vm-tools not yet.

[agners]

Hm, open-vm-tools should be part of that build. Can you get OS shell access (using login) and check if the udev rules are present?

ls -la /usr/lib/udev/rules.d/99-vmware-scsi-udev.rules

[Grandma-Betty]

Nope, there's not even the /usr/lib/udev directory available:

It's completely missing.

[agners]

Hm, I see, the build system prevents the package from being selected under AArch64 🙈 should be a easy fix.

[Grandma-Betty]

Let me know when the next image is ready.

[Grandma-Betty]

I can confirm open-vm-tools and nvme boot works now with the latest 8.1 update.
Thanks a lot @agners for your support mate!
Bischen geile Siech ;-)

[Grandma-Betty]

Just a minor thing I found:
Isn't the system supposed to show "true" for the "Virtual Environment" entry when using a .vmdk as a installation source? I mean, where else would you use a .vmdk file for than within a virtual environment? Or am I misunderstanding the meaning of this?

[mondface]

Now I also run home assistant 8.1 with "vmware tools" on my esxi on arm.
Thank you Grandma-Betty!

so far I had my ha running on an intel NUC as vm under esxi 7.
since today also runs a ha on a raspi 4 with 8GB RAM as vm under esxi on arm.
first download the vmdk "haos_generic-aarch64-8.0.vmdk.zip":

https://github.com/home-assistant/operating-system/releases/tag/8.0

Create a vm as "other 4th linux-system 64-bit", but without disk
then copy the unzipped vmdk to the esx and import it there (please adjust paths):

vmkfstools -i /vmfs/volumes/iso/haos_generic-aarch64-8.0.dev20220223.vmdk /vmfs/volumes/datastore1/haos/haos.vmdk

Include vmdk in ha's configuration
first go to 8.1 after installation, now you also have "vmware-tools"

[Grandma-Betty]

@mondface You're welcome in any way, but don't thank me, thank @agners and @claplace, they're the real heroes here ;-)

[claplace]

oh I need to upgrade to 8.1 then!

Detecting VMware virtualization is possible by looking at the device tree. e.g. I added those lines to systemd: https://github.com/systemd/systemd/blob/df423851fcc05cf02281d11aab6aee7b476c1c3b/src/basic/virt.c#L134

[mondface]

I have a zwave-me stick, but it doesn't run smoothly. USB I have to assign 3.1 for the vm.
how did you implement that?

[Grandma-Betty]

@mondface Well I don't use zwave, but it's kinda common that you have to use a USB 3.1 controller. I was never able for any distribution running on ESXi ARM to make USB 2.0 controller working reliably. But since USB 3.1 controller never gave me any issues it's not really a disadvantage to me. I think the USB backwards compatibility is always given so I don't care. You just have to know it, that's all ;-)

I remember once writing an issue about this according to Debian distro which may also be interesting to you:
https://flings.vmware.com/esxi-arm-edition/comments/18497#comment_18505

[Grandma-Betty]

@mondface @claplace Off-topic but the following issue also might be interesting to you as I am not the only one struggling with NUT on ESXi since months now:
networkupstools/nut#1004
This might be useful to be aware of as you can read there are also users struggling with NUT runnig on Home Assistant running on ESXi. Also one of the latest posts of the user running 29 ESXi servers which are all affected in the same way about the USB issue proofs that there is something wrong.

[Grandma-Betty]

@agners As the aarch64 image went stable now it would make sense to include it here, wouldn't it?
https://www.home-assistant.io/installation/
I mean the more people are aware of it, the more people will use it, the more it gets the chance to get "matured".
I can only tell it works like a charm in my case and I really did many(!) tests over the weekend with the add-ons/integrations I have mentioned above. Nothing to complain about compared to my previous x86 virtual setup. In fact I am really more happy now with HA running on a Raspberry Pi based ESXi hypervisor.

[agners]

@Grandma-Betty @claplace the "Virtual Environment" information is from Home Assistant Core, and indicates if the installation makes use of a Python Virtual Environment. So it is unrelated to the virtualization technology we are discussing.

@Grandma-Betty I'd still count aarch64 virtualization quite niche still. If we would add it to the official documentation would add maintenance load on our end. At this point I don't think we would accept it to the official documentation.

I recommend starting with a Community Guide for now. A good title and it will be the first Google search result 😄 Tips like the USB 2.0/3.1 issue etc, can then be collected there. The #1050 (comment) is probably a good start already.

I'll close the issue here as there is nothing open on HAOS side at this point. If you create a community guide, feel free to link it here. If more work is to be done on HAOS side, please create a new issue.

[Grandma-Betty]

I recommend starting with a Community Guide for now.

https://community.home-assistant.io/t/installing-haos-on-vmware-flings-esxi-arm/426604

[claplace]

Hi @Grandma-Betty, this guide is very well written! Thank you very much.

[thiscantbeserious]

Just in case: I used the generic-aarch64 image successfully including USB-Passtrought with a Zigbee-Stick (HAOS 9.3) on Pimox 7 - with a Raspberry Pi 4/8GB ... works surprisingly smooth. Used the installer from here:

https://github.com/tteck/Proxmox/blob/main/vm/haos-vm-v4.sh

Simply replaced ova with generic-aarch64 image in it and afterwards coverted the disks to qcow2 (also had to remove the EFI boot disk and readd it).

Works surprisingly smooth - even with just 2 cores assigned.

Might be a better Option then ESiX in terms of performance, specifically on something as limited as ARM SBCs.

Just be sure to not use a Board like the NanoPi M4v2 with BIG.little architecture - because that's something KVM/QEMU refuses to work with - multiple different core types.

[tteck]

There's actually a PiMox script to install HAOS at https://tteck.github.io/Proxmox/

(https://github.com/tteck/Proxmox/raw/main/vm/pimox-haos-vm-v4.sh)

[Grandma-Betty]

Might be a better Option then ESiX in terms of performance, specifically on something as limited as ARM SBCs.

Not sure about that. The ESXi hypervisor does not consume much resources.
But indeed it is a very interesting option and I'm very impressed by the progressive process of virtualization generally.
Also it's nice that PiMox is open source and it does not have any license limitations as the annoying 180 days ESXi ARM limitation, which you only can get around when assigning a license via vCenter to it.

Because of license reasons I was considering too about jumping my homelab from vSphere to a Proxmox environment, but the missing ARM part for Proxmox was holding me back currently. Now I can think it over again.
So thanks for sharing!

[thiscantbeserious]

There's actually a PiMox script to install HAOS at https://tteck.github.io/Proxmox/

(https://github.com/tteck/Proxmox/raw/main/vm/pimox-haos-vm-v4.sh)

Dang, didn't see that, thanks! ... but it works anyway. :)

Might be a better Option then ESiX in terms of performance, specifically on something as limited as ARM SBCs.

Not sure about that. The ESXi hypervisor does not consume much resources. But indeed it is a very interesting option and I'm very impressed by the progressive process of virtualization generally. Also it's nice that PiMox is open source and it does not have any license limitations as the annoying 180 days ESXi ARM limitation, which you only can get around when assigning a license via vCenter to it.

Because of license reasons I was considering too about jumping my homelab from vSphere to a Proxmox environment, but the missing ARM part for Proxmox was holding me back currently. Now I can think it over again. So thanks for sharing!

Just be carefull: Any BIG.little architecture will give you troubles - as long as its a Raspberrypi with 4 Cores of the same type it wont have any troubles.

[thiscantbeserious]

@tteck if you modify your create_lxc.sh script to actually be able to select a locally stored images you can actually use all lxc scripts also with aarch64/arm64 on the Raspberry Pi 4B for example with Pimox (because the tool shipping with promox only having and downloading amd64 variants - which will obviously also fail to start because of the wrong architecture) ... currently running a hard coded variant over here to test them ... zigbee2mqtt and homeassistant lxc work all fine too including usb passtrought

[tteck]

@thiscantbeserious Maybe use uname -m to detect the architecture. If aarch64/arm64 is detected, download the appropriate image. 🤷🏻‍♂️