open-vm-tools exceptions while boot on ESXI 6.7

[Amos-85]

RancherOS Version: 1.5.6

Where are you running RancherOS? ( Vsphere baremetal)

Issue:
after booting from iso of [rancher os v1.5.6 / v1.5.5] (https://github.com/rancher/os/releases/download/v1.5.6/rancheros-vmware.iso),
open-vm-tools has some exceptions relate to non compatible binaries to the rancher os.

open-vm-tools logs in /var/log/vmware-vmsvc.log:

[2020-10-26T15:09:45.296Z] [ message] [vmsvc] Log caching is enabled with maxCacheEntries=4096.
[2020-10-26T15:09:45.297Z] [ message] [vmsvc] Core dump limit set to -1
[2020-10-26T15:09:45.297Z] [ message] [vmtoolsd] Tools Version: 10.3.10.10540 (build-12406962)
[2020-10-26T15:09:45.307Z] [ warning] [vmtoolsd] Opening plugin '._libhgfsServer.so' failed: /usr/local/lib/open-vm-tools/plugins/common/._libhgfsServer.so: invalid ELF header.
[2020-10-26T15:09:45.307Z] [ warning] [vmtoolsd] Opening plugin '._libvix.so' failed: /usr/local/lib/open-vm-tools/plugins/common/._libvix.so: invalid ELF header.
[2020-10-26T15:09:45.308Z] [ warning] [vmtoolsd] Opening plugin '._libdeployPkgPlugin.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libdeployPkgPlugin.so: invalid ELF header.
[2020-10-26T15:09:45.308Z] [ warning] [vmtoolsd] Opening plugin '._libguestInfo.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libguestInfo.so: invalid ELF header.
[2020-10-26T15:09:45.309Z] [ warning] [vmtoolsd] Opening plugin '._libpowerOps.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libpowerOps.so: invalid ELF header.
[2020-10-26T15:09:45.309Z] [ warning] [vmtoolsd] Opening plugin '._libresolutionKMS.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libresolutionKMS.so: invalid ELF header.
[2020-10-26T15:09:45.309Z] [ warning] [vmtoolsd] Opening plugin '._libtimeSync.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libtimeSync.so: invalid ELF header.
[2020-10-26T15:09:45.309Z] [ warning] [vmtoolsd] Opening plugin '._libvmbackup.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libvmbackup.so: invalid ELF header.
[2020-10-26T15:09:45.309Z] [ warning] [vmtoolsd] Opening plugin 'libdeployPkgPlugin.so' failed: libmspack.so.0: cannot open shared object file: No such file or directory.
[2020-10-26T15:09:45.310Z] [ message] [vmsvc] Cannot load message catalog for domain 'hgfsServer', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-26T15:09:45.310Z] [ message] [vmtoolsd] Plugin 'hgfsServer' initialized.
[2020-10-26T15:09:45.311Z] [ message] [vmsvc] Cannot load message catalog for domain 'vix', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-26T15:09:45.311Z] [ message] [vmtoolsd] Plugin 'vix' initialized.
[2020-10-26T15:09:45.311Z] [ message] [vmsvc] Cannot load message catalog for domain 'guestInfo', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-26T15:09:45.311Z] [ message] [vmtoolsd] Plugin 'guestInfo' initialized.
[2020-10-26T15:09:45.311Z] [ message] [vmsvc] Cannot load message catalog for domain 'powerops', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-26T15:09:45.311Z] [ message] [vmtoolsd] Plugin 'powerops' initialized.
[2020-10-26T15:09:45.311Z] [ message] [vmsvc] Cannot load message catalog for domain 'timeSync', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-26T15:09:45.311Z] [ message] [vmtoolsd] Plugin 'timeSync' initialized.
[2020-10-26T15:09:45.311Z] [ message] [vmsvc] Cannot load message catalog for domain 'vmbackup', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-26T15:09:45.311Z] [ message] [vmtoolsd] Plugin 'vmbackup' initialized.
[2020-10-26T15:09:45.315Z] [ message] [vix] VixTools_ProcessVixCommand: command 62
[2020-10-26T15:12:09.388Z] [ message] [vmtoolsd] Unloading plugin 'vmbackup'.
[2020-10-26T15:12:09.388Z] [ message] [vmtoolsd] Unloading plugin 'timeSync'.
[2020-10-26T15:12:09.388Z] [ message] [vmtoolsd] Unloading plugin 'powerops'.
[2020-10-26T15:12:09.388Z] [ message] [vmtoolsd] Unloading plugin 'guestInfo'.
[2020-10-26T15:12:09.388Z] [ message] [vmtoolsd] Unloading plugin 'vix'.
[2020-10-26T15:12:09.388Z] [ message] [vmtoolsd] Unloading plugin 'hgfsServer'.
[2020-10-31T00:55:04.100Z] [ message] [vmsvc] Log caching is enabled with maxCacheEntries=4096.
[2020-10-31T00:55:04.106Z] [ message] [vmsvc] Core dump limit set to -1
[2020-10-31T00:55:04.106Z] [ message] [vmtoolsd] Tools Version: 10.3.10.10540 (build-12406962)
[2020-10-31T00:55:04.122Z] [ warning] [vmtoolsd] Opening plugin '._libhgfsServer.so' failed: /usr/local/lib/open-vm-tools/plugins/common/._libhgfsServer.so: invalid ELF header.
[2020-10-31T00:55:04.122Z] [ warning] [vmtoolsd] Opening plugin '._libvix.so' failed: /usr/local/lib/open-vm-tools/plugins/common/._libvix.so: invalid ELF header.
[2020-10-31T00:55:04.127Z] [ warning] [vmtoolsd] Opening plugin '._libdeployPkgPlugin.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libdeployPkgPlugin.so: invalid ELF header.
[2020-10-31T00:55:04.128Z] [ warning] [vmtoolsd] Opening plugin '._libguestInfo.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libguestInfo.so: invalid ELF header.
[2020-10-31T00:55:04.128Z] [ warning] [vmtoolsd] Opening plugin '._libpowerOps.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libpowerOps.so: invalid ELF header.
[2020-10-31T00:55:04.128Z] [ warning] [vmtoolsd] Opening plugin '._libresolutionKMS.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libresolutionKMS.so: invalid ELF header.
[2020-10-31T00:55:04.129Z] [ warning] [vmtoolsd] Opening plugin '._libtimeSync.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libtimeSync.so: invalid ELF header.
[2020-10-31T00:55:04.129Z] [ warning] [vmtoolsd] Opening plugin '._libvmbackup.so' failed: /usr/local/lib/open-vm-tools/plugins/vmsvc/._libvmbackup.so: invalid ELF header.
[2020-10-31T00:55:04.131Z] [ warning] [vmtoolsd] Opening plugin 'libdeployPkgPlugin.so' failed: libmspack.so.0: cannot open shared object file: No such file or directory.
[2020-10-31T00:55:04.136Z] [ message] [vmsvc] Cannot load message catalog for domain 'hgfsServer', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-31T00:55:04.136Z] [ message] [vmtoolsd] Plugin 'hgfsServer' initialized.
[2020-10-31T00:55:04.136Z] [ message] [vmsvc] Cannot load message catalog for domain 'vix', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-31T00:55:04.136Z] [ message] [vmtoolsd] Plugin 'vix' initialized.
[2020-10-31T00:55:04.136Z] [ message] [vmsvc] Cannot load message catalog for domain 'guestInfo', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-31T00:55:04.136Z] [ message] [vmtoolsd] Plugin 'guestInfo' initialized.
[2020-10-31T00:55:04.136Z] [ message] [vmsvc] Cannot load message catalog for domain 'powerops', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-31T00:55:04.136Z] [ message] [vmtoolsd] Plugin 'powerops' initialized.
[2020-10-31T00:55:04.136Z] [ message] [vmsvc] Cannot load message catalog for domain 'timeSync', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-31T00:55:04.136Z] [ message] [vmtoolsd] Plugin 'timeSync' initialized.
[2020-10-31T00:55:04.136Z] [ message] [vmsvc] Cannot load message catalog for domain 'vmbackup', language 'C', catalog dir '/usr/local/share/open-vm-tools'.
[2020-10-31T00:55:04.136Z] [ message] [vmtoolsd] Plugin 'vmbackup' initialized.
[2020-10-31T00:55:04.140Z] [ message] [powerops] Executing script: '/etc/vmware-tools/poweron-vm-default'
[2020-10-31T00:55:04.213Z] [ message] [powerops] Script exit code: 0, success = 1
[2020-10-31T00:55:04.223Z] [ message] [vix] VixTools_ProcessVixCommand: command 62

This issue leads me to another issue with terraform vshpere module to provision a new vm from rancher os template cause the customization wasn't finish with this exception while running terraform apply:

Virtual machine customization failed on "/Datacenter/vm/somefolder/anotherfolder/vmname":
An error occurred while customizing VM vmname. For details reference the log file <No Log> in the guest OS.
The virtual machine has not been deleted to assist with troubleshooting. If
corrective steps are taken without modifying the "customize" block of the
resource configuration, the resource will need to be tainted before trying
again. For more information on how to do this, see the following page:
https://www.terraform.io/docs/commands/taint.html

the customization isn't finished and the NIC state left disconnected.
the guest os I've chose was Linux
guest os version: other 64 bit

vm vmware-.log:
vmware.log

VMX File:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "14"
nvram = "Rancher.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
floppy0.present = "FALSE"
numvcpus = "4"
memSize = "4096"
tools.upgrade.policy = "manual"
sched.cpu.units = "mhz"
sched.cpu.affinity = "all"
sched.cpu.latencySensitivity = "normal"
vm.createDate = "1603446853039486"
scsi0.virtualDev = "lsilogic"
scsi0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "Rancher.vmdk"
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
scsi0:0.present = "TRUE"
ethernet0.virtualDev = "vmxnet3"
ethernet0.dvs.switchId = "50 10 11 7e 23 1a 53 bb-22 96 18 58 1a 72 64 88"
ethernet0.dvs.portId = "20540"
ethernet0.dvs.portgroupId = "dvportgroup-29"
ethernet0.dvs.connectionId = "994556013"
ethernet0.shares = "normal"
ethernet0.addressType = "vpx"
ethernet0.generatedAddress = "00:50:56:90:df:d7"
ethernet0.uptCompatibility = "TRUE"
ethernet0.present = "TRUE"
displayName = "Rancher"
guestOS = "coreos-64"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
uuid.bios = "42 10 73 84 41 22 32 d3-ba fc b4 66 68 9a d4 77"
vc.uuid = "50 10 65 3b 23 1a 82 3d-a6 73 fe 24 e8 e9 58 17"
migrate.hostLog = "Rancher-472cb6cd.hlog"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
migrate.encryptionMode = "opportunistic"
numa.autosize.cookie = "40001"
numa.autosize.vcpu.maxPerVirtualNode = "4"
sched.swap.derivedName = "/vmfs/volumes/vsan:52ddd636e9bc3a88-8e5869c4a6f73e51/41a8925f-b6fb-e5dc-07df-e4434b8a1b10/Rancher-60c919ef.vswp"
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "16"
ethernet0.pciSlotNumber = "160"
vmci0.pciSlotNumber = "32"
vmci0.id = "1754977399"
monitor.phys_bits_used = "43"
vmotion.checkpointFBSize = "4194304"
vmotion.checkpointSVGAPrimarySize = "4194304"
cleanShutdown = "FALSE"
softPowerOff = "FALSE"
tools.syncTime = "FALSE"
bios.bootDelay = "500"
ide0:0.deviceType = "cdrom-image"
ide0:0.fileName = "/vmfs/volumes/vsan:52ddd636e9bc3a88-8e5869c4a6f73e51/9147d55d-6ee6-b179-7881-e4434b8a1b50/Rancher-os/rancheros_vmware_v1.5.6.iso"
ide0:0.present = "TRUE"
uuid.location = "56 4d 53 95 8a 43 15 e5-8a 40 3b ec db 72 4a f1"
cpuid.coresPerSocket = "4"
vhv.enable = "TRUE"
firmware = "efi"
uefi.secureBoot.enabled = "TRUE"

Expected Behavior:
Boot Rancher OS without open-vm-tools exception and able to customize rancher.

[olljanat]

@Amos-85 RancherOS is EOL #3000

However it would be nice to hear if my recently created BurmillaOS beta release works any better? I don't have access to ESXi 6.7 environment. You can find it and release notes which describes differences to RancherOS https://github.com/burmilla/os/releases/tag/v1.9.0-rc1