AuroraBoot missing dependencies when running binary, not starting pixieboot when running in container.

[tbrasser]

Kairos version:
quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1

CPU architecture, OS, and Version:
Linux spacebar 6.4.10-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 11 Aug 2023 11:03:36 +0000 x86_64 GNU/Linux

Describe the bug
When I run AuroraBoot in a container, pixieboot does not seem to start. When I run AuroraBoot directly it only works with github artifacts not with container images, then it is missing dependencies to build/serve netboot artifacts.

To Reproduce

• Running AuroraBoot container

sudo podman run --privileged --net host --rm -i -t -v $PWD/aurora.yaml:/aurora.yaml -v $PWD/build:/tmp/auroraboot -v $PWD/data:/tmp/data quay.io/kairos/auroraboot --debug /aurora.yaml

• Running AuroraBoot binary

sudo ./AuroraBoot --debug ./aurora.yaml

Expected behavior
When I use AuroraBoot binary, it should be able to build/serve iso/netboot artifacts.
When I use AuroraBoot in a container with host networking, it should have no problems serving all it needs to.

Logs

• Running binary:

5:07PM INF Pulling container image 'quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1' to '/home/thomas/kairos/build/temp-rootfs' (local: true)
5:08PM DBG Output 'INFO   Downloading quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1 to /home/thomas/kairos/build/temp-rootfs
 INFO   Size: 2.402GiB'
5:08PM INF Generating iso 'kairos' from '/home/thomas/kairos/build/temp-rootfs' to '/home/thomas/kairos/build/iso'
5:08PM DBG Output '/bin/sh: line 1: /entrypoint.sh: No such file or directory'
5:08PM ERR Failed generating iso 'kairos' from '/home/thomas/kairos/build/temp-rootfs'. Error: exit status 127
4 errors occurred:
	* exit status 127
	* 'extract-netboot' deps gen-iso failed
	* 'start-httpserver' deps gen-iso failed
	* 'start-netboot' deps extract-netboot failed

• Running in container:

3:33PM DBG #cloud-config
install:
  auto: true
  device: /dev/nvme1n1
  reboot: true
hostname: kairos-node-{{ trunc 4 .MachineID }}
users:
  - name: kairos
    passwd: kairos
    ssh_authorized_keys:
      - github:tbrasser
debug: true
strict: true
k3s:
  enable: true
bundles:
  - targets:
      - run://quay.io/kairos/community-bundles:system-upgrade-controller_latest
      - run://quay.io/kairos/community-bundles:cert-manager_latest
      - run://quay.io/kairos/community-bundles:kairos_latest
kairos:
  entangle:
    enable: true
  entangleProxy:
    enable: true
p2p:
  role: master
  network_token: |
    <redacted>
reset:
  reboot: true
  reset-persistent: true
  reset-oem: true
upgrade:
  reboot: true

3:33PM DBG 1.
3:33PM DBG  <prepare-iso> (background: false)
3:33PM DBG  <prepare-netboot> (background: false)
3:33PM DBG  <prepare-temp> (background: false)
3:33PM DBG
3:33PM DBG 2.
3:33PM DBG  <container-pull> (background: false)
3:33PM DBG  <copy-cloud-config> (background: false)
3:33PM DBG
3:33PM DBG 3.
3:33PM DBG  <gen-iso> (background: false)
3:33PM DBG
3:33PM DBG 4.
3:33PM DBG  <start-httpserver> (background: true)
3:33PM DBG  <extract-netboot> (background: false)
3:33PM DBG
3:33PM DBG 5.
3:33PM DBG  <start-netboot> (background: true)
3:33PM DBG
3:33PM INF Pulling container image 'quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1' to '/home/thomas/kairos/build/temp-rootfs' (local: false)
3:37PM DBG Output ' INFO    Downloading  quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1  to  /home/thomas/kairos/build/temp-rootfs
 INFO    Pulled: sha256:e01b4602b73d840b452dcfe3750d9e331b11adea3baaaa11be81f5dd8afe6239 quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1
 INFO    Size: 2.402GiB
'
3:37PM INF Generating iso 'kairos' from '/home/thomas/kairos/build/temp-rootfs' to '/home/thomas/kairos/build/iso'
3:40PM DBG Output '+ enki --config-dir /config --debug --name kairos build-iso --squash-no-compression --overlay-iso /tmp/geniso69068545 --date=false --output /home/thomas/kairos/build/iso dir:/home/thomas/kairos/build/temp-rootfs
INFO[2023-08-22T15:37:57Z] Starting enki version 0.0.1                  
DEBU[2023-08-22T15:37:57Z] {Version:0.0.1 GitCommit: GoVersion:go1.20.6} 
DEBU[2023-08-22T15:37:57Z] Full config loaded: &v1.BuildConfig{
  Date: false,
  Name: "kairos",
  OutDir: "/home/thomas/kairos/build/iso",
  Config: v1.Config{
    Logger: &v1.logrusWrapper{ // p0
      Logger: &logrus.Logger{
        Out: &os.File{},
        Hooks: logrus.LevelHooks{},
        Formatter: &logrus.TextFormatter{
          ForceColors: true,
          DisableColors: false,
          ForceQuote: false,
          DisableQuote: false,
          EnvironmentOverrideColors: false,
          DisableTimestamp: false,
          FullTimestamp: true,
          TimestampFormat: "",
          DisableSorting: false,
          SortingFunc: ,
          DisableLevelTruncation: false,
          PadLevelText: false,
          QuoteEmptyFields: false,
          FieldMap: logrus.FieldMap(nil),
          CallerPrettyfier: ,
        },
        ReportCaller: false,
        Level: 5,
        ExitFunc: os.Exit,
        BufferPool: nil,
      },
    },
    Fs: &vfs.osfs{}, // p1
    Mounter: &mount.Mounter{},
    Runner: &v1.RealRunner{ // p2
      Logger: p0,
    },
    Syscall: &v1.RealSyscall{},
    CloudInitRunner: &cloudinit.YipCloudInitRunner{},
    ImageExtractor: nil,
    Client: &http.Client{},
    Platform: &v1.Platform{
      OS: "linux",
      Arch: "x86_64",
      GolangArch: "amd64",
    },
    Cosign: false,
    Verify: false,
    CosignPubKey: "",
    Repos: []v1.Repository{},
    Arch: "x86_64",
    SquashFsCompressionConfig: []string{},
    SquashFsNoCompression: true,
  },
} 
DEBU[2023-08-22T15:37:57Z] Loaded LiveISO: &v1.LiveISO{
  RootFS: nil,
  UEFI: []*v1.ImageSource{
    &v1.ImageSource{},
  },
  Image: []*v1.ImageSource{
    &v1.ImageSource{},
    &v1.ImageSource{},
  },
  Label: "COS_LIVE",
  GrubEntry: "Kairos",
  BootloaderInRootFs: false,
} 
INFO[2023-08-22T15:37:57Z] Preparing squashfs root...                   
INFO[2023-08-22T15:37:57Z] Copying /home/thomas/kairos/build/temp-rootfs source... 
INFO[2023-08-22T15:37:57Z] Starting rsync...                            
DEBU[2023-08-22T15:37:57Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /home/thomas/kairos/build/temp-rootfs/ /tmp/enki-iso324095657/rootfs/' 
DEBU[2023-08-22T15:38:02Z] Syncing data...                              
DEBU[2023-08-22T15:38:07Z] Syncing data...                              
INFO[2023-08-22T15:38:07Z] Finished syncing                             
INFO[2023-08-22T15:38:07Z] Finished copying /home/thomas/kairos/build/temp-rootfs into /tmp/enki-iso324095657/rootfs 
INFO[2023-08-22T15:38:07Z] Preparing EFI image...                       
INFO[2023-08-22T15:38:07Z] Copying /efi source...                       
INFO[2023-08-22T15:38:07Z] Starting rsync...                            
DEBU[2023-08-22T15:38:07Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /efi/ /tmp/enki-iso324095657/uefi/' 
INFO[2023-08-22T15:38:07Z] Finished syncing                             
INFO[2023-08-22T15:38:07Z] Finished copying /efi into /tmp/enki-iso324095657/uefi 
INFO[2023-08-22T15:38:07Z] Preparing ISO image root tree...             
INFO[2023-08-22T15:38:07Z] Copying /efi source...                       
INFO[2023-08-22T15:38:07Z] Starting rsync...                            
DEBU[2023-08-22T15:38:07Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /efi/ /tmp/enki-iso324095657/iso/' 
INFO[2023-08-22T15:38:07Z] Finished syncing                             
INFO[2023-08-22T15:38:07Z] Finished copying /efi into /tmp/enki-iso324095657/iso 
INFO[2023-08-22T15:38:07Z] Copying /grub2 source...                     
INFO[2023-08-22T15:38:07Z] Starting rsync...                            
DEBU[2023-08-22T15:38:07Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /grub2/ /tmp/enki-iso324095657/iso/' 
INFO[2023-08-22T15:38:07Z] Finished syncing                             
INFO[2023-08-22T15:38:07Z] Finished copying /grub2 into /tmp/enki-iso324095657/iso 
INFO[2023-08-22T15:38:07Z] Copying /tmp/geniso69068545 source...        
INFO[2023-08-22T15:38:07Z] Starting rsync...                            
DEBU[2023-08-22T15:38:07Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /tmp/geniso69068545/ /tmp/enki-iso324095657/iso/' 
INFO[2023-08-22T15:38:07Z] Finished syncing                             
INFO[2023-08-22T15:38:07Z] Finished copying /tmp/geniso69068545 into /tmp/enki-iso324095657/iso 
DEBU[2023-08-22T15:38:07Z] Copying Kernel file /tmp/enki-iso324095657/rootfs/boot/vmlinuz-6.2.0-27-generic to iso root tree 
DEBU[2023-08-22T15:38:07Z] Copying initrd file /tmp/enki-iso324095657/rootfs/boot/initrd-6.2.0-27-generic to iso root tree 
INFO[2023-08-22T15:38:07Z] Creating squashfs...                         
DEBU[2023-08-22T15:38:07Z] Running cmd: 'mksquashfs /tmp/enki-iso324095657/rootfs /tmp/enki-iso324095657/iso/rootfs.squashfs -b 1024k' 
INFO[2023-08-22T15:40:29Z] Creating EFI image...                        
INFO[2023-08-22T15:40:29Z] Creating file system image /tmp/enki-iso324095657/iso/boot/uefi.img 
DEBU[2023-08-22T15:40:29Z] Running cmd: 'mkfs.vfat -n COS_GRUB /tmp/enki-iso324095657/iso/boot/uefi.img' 
DEBU[2023-08-22T15:40:29Z] Running cmd: 'mcopy -s -i /tmp/enki-iso324095657/iso/boot/uefi.img /tmp/enki-iso324095657/uefi/EFI ::' 
DEBU[2023-08-22T15:40:29Z] Running cmd: 'mcopy -s -i /tmp/enki-iso324095657/iso/boot/uefi.img /tmp/enki-iso324095657/uefi/var ::' 
INFO[2023-08-22T15:40:29Z] Creating ISO image...                        
DEBU[2023-08-22T15:40:29Z] Running cmd: 'xorriso -volid COS_LIVE -joliet on -padding 0 -outdev /home/thomas/kairos/build/iso/kairos.iso -map /tmp/enki-iso324095657/iso / -chmod 0755 -- -boot_image grub bin_path=/boot/x86_64/loader/eltorito.img -boot_image grub grub2_mbr=/tmp/enki-iso324095657/iso//boot/x86_64/loader/boot_hybrid.img -boot_image grub grub2_boot_info=on -boot_image any partition_offset=16 -boot_image any cat_path=/boot/x86_64/boot.catalog -boot_image any cat_hidden=on -boot_image any boot_info_table=on -boot_image any platform_id=0x00 -boot_image any emul_type=no_emulation -boot_image any load_size=2048 -append_partition 2 0xef /tmp/enki-iso324095657/iso/boot/uefi.img -boot_image any next -boot_image any efi_path=--interval:appended_partition_2:all:: -boot_image any platform_id=0xef -boot_image any emul_type=no_emulation' 
DEBU[2023-08-22T15:40:32Z] Xorriso: xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev '/home/thomas/kairos/build/iso/kairos.iso'
Media current: stdio file, overwriteable
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data,  182g free
xorriso : UPDATE : 1168 files added in 1 seconds
Added to ISO image: directory '/'='/tmp/enki-iso324095657/iso'
xorriso : NOTE : Copying to System Area: 512 bytes from file '/tmp/enki-iso324095657/iso/boot/x86_64/loader/boot_hybrid.img'
xorriso : UPDATE : Writing:      12896s    2.0%   fifo   9%  buf  50%
xorriso : UPDATE : Writing:      64528s   10.2%   fifo  41%  buf  50%
xorriso : UPDATE : Writing:     566736s   89.3%   fifo  99%  buf  50%  741.9xD 
ISO image produced: 634343 sectors
Written to medium : 634352 sectors at LBA 48
Writing to '/home/thomas/kairos/build/iso/kairos.iso' completed successfully.
 
'
3:40PM INF Extracting netboot artifacts 'kairos' from '/home/thomas/kairos/build/iso/kairos.iso' to '/home/thomas/kairos/build/netboot'
3:40PM INF Listening on 0.0.0.0:8080...
3:40PM DBG Output ''
3:40PM INF Start pixiecore

[tbrasser]

Tried building with osbuilder on kind and getting better results (@2.4.0-rc1)!

netboot: true works, but gkeImage: true doesn't. At least I got a WoW now that is usable.

Can update here or close and create new issues when they come up.

[audiato]

I'm getting the same errors when trying to use ghcr.io/kairos-io/provider-rke2:v2.3.3 with Auroraboot in Docker.

#1793

[jimmykarily]

@tbrasser I don't have podman installed so I just replaced podman with docker in your command:

sudo docker run --privileged --net host --rm -i -t -v $PWD/aurora.yaml:/aurora.yaml -v $PWD/build:/tmp/auroraboot -v $PWD/data:/tmp/data quay.io/kairos/auroraboot --debug /aurora.yaml

also I don't know what your aurora.yaml looks like. I created one that would pull the same image as in the description:

container_image: quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1

It works just fine. Among the rest of the output, I see this line:

6:52AM INF Pulling container image 'quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1' to '/tmp/temp-rootfs' (local: false)

Your output says local: true. Are you prefixing your image in the aurora config with docker:// ? This will make auroraboot try to use a local image and not pull it.

[tbrasser]

I tried various combinarions, will try again with 2.4.0 release. What is the suggested WoW? osbuilder or auroraboot? (also encountering some race conditions in the main/sidecar containers there, seems to me all should be initContainers with exporter to nginx as single main container. (also probably job instead of bare pod))

[jimmykarily]

Regarding osbuilder, I agree. On osbuilder master branch, we've already switched to InitContainers (not tagged or released yet). Also, it used to be a Job but was later converted to a Pod. I still think it should be a Job and we'll probably change it back to a Job in the future.

There is a lot of work planned on osbuilder and when we are done, auroraboot will be used internally by osbuilder but will also be used directly by users that don't want to use Kubernetes for building.

Relevant issue with more information: #1633

[jimmykarily]

@tbrasser since we can't reproduce it, I'll close this. Please give it a try with 2.4.0 and feel free to re-open is the issue persists. Thanks for reporting!