omicron/baseline service fails to start in bootenv created with helios-build onu

[luqmana]

I created a boot env with helios-build onu to test some illumos changes and wanted to run omicron. But ran into trouble early on setting up the softnpu zone:

softnpu zone setup
configure zone
install zone
Error: Failed to parse command output: exit code 1
stdout:
A ZFS file system has been created for this zone.
INFO: omicron: installing zone sidecar_softnpu @ "/sidecar/sidecar_softnpu"...
INFO: omicron: replicating /usr tree...
INFO: omicron: replicating /lib tree...
INFO: omicron: replicating /sbin tree...
INFO: omicron: pruning SMF manifests...
INFO: omicron: pruning global-only files...
stderr:
Error: could not locate "gzonly.txt" in any baseline directory

Caused by:
    exit code 1
    stdout:
    A ZFS file system has been created for this zone.
    INFO: omicron: installing zone sidecar_softnpu @ "/sidecar/sidecar_softnpu"...
    INFO: omicron: replicating /usr tree...
    INFO: omicron: replicating /lib tree...
    INFO: omicron: replicating /sbin tree...
    INFO: omicron: pruning SMF manifests...
    INFO: omicron: pruning global-only files...
    stderr:
    Error: could not locate "gzonly.txt" in any baseline directory

Turns out the omicron/baseline service was failing to come up:

svc:/system/omicron/baseline:default (Omicron brand baseline generation)
 State: maintenance since February  8, 2024 at 12:06:14 PM PST
Reason: Start method failed repeatedly, last exited with status 1.
   See: http://illumos.org/msg/SMF-8000-KS
   See: man -M /usr/share/man -s 7 omicron1
   See: /var/svc/log/system-omicron-baseline:default.log
Impact: This service is not running.

[ Feb  8 12:06:03 Executing start method ("/usr/lib/brand/omicron1/baseline -w /var/run/brand/omicron1/baseline"). ]
incorp = pkg://helios-dev/consolidation/osnet/[email protected]:20240204T130208Z
entire = pkg:/entire@latest
install = pkg://helios-dev/network/[email protected]:20240105T152834Z
install = pkg://helios-dev/service/network/[email protected]:20230804T102734Z
tempdir @ "/tmp/.tmpA7JFbi/tmp"
image root @ "/tmp/.tmpA7JFbi/tmp/root"
creating image...
copying publishers...
adding properties...
installing packages...
Error: pkg error: "pkg exact-install: No matching version of consolidation/osnet/osnet-incorporation can be installed:\n  Reject:  pkg://helios-dev/consolidation/osnet/[email protected]\n  Reason:  No version matching 'incorporate' dependency [email protected] can be installed\n    ----------------------------------------\n    Reject:  pkg://helios-dev/[email protected]\n    Reason:  Higher ranked publisher on-nightly was selected\n    ----------------------------------------\nNo matching version of service/network/chrony can be installed:\n  Reject:  pkg://helios-dev/service/network/[email protected]\n  Reason:  No version matching 'require' dependency [email protected] can be installed\n    ----------------------------------------\n    Reject:  pkg://helios-dev/[email protected]\n    Reason:  Rejected by 'incorporate' dependency in proposed package 'consolidation/osnet/osnet-incorporation'\n    Reason:  Rejected by 'require' dependency in proposed package 'network/openssh-server'\n      ----------------------------------------\n      Reject:  pkg://helios-dev/network/[email protected]\n      Reason:  No version matching 'require' dependency [email protected] can be installed\n        ----------------------------------------\n        Reject:  pkg://helios-dev/[email protected]\n                   to\n                 pkg://helios-dev/[email protected]\n        Reason:  Rejected by 'incorporate' dependency in proposed package 'consolidation/osnet/osnet-incorporation'\n        Reason:  Higher ranked publisher on-nightly was selected\n        ----------------------------------------\n      ----------------------------------------\n    Reason:  Higher ranked publisher on-nightly was selected\n    Reject:  pkg://helios-dev/[email protected]\n               to\n             pkg://helios-dev/[email protected]\n    Reason:  Rejected by 'incorporate' dependency in proposed package 'consolidation/osnet/osnet-incorporation'\n    Reason:  Rejected by 'require' dependency in proposed package 'network/openssh-server'\n    Reason:  Higher ranked publisher on-nightly was selected\n    Reject:  pkg://helios-dev/[email protected]\n               to\n             pkg://helios-dev/[email protected]\n    Reason:  [already rejected; see above]\n    ----------------------------------------\nNo matching version of network/openssh-server can be installed:\n  Reject:  pkg://helios-dev/network/[email protected]\n  Reason:  [already rejected; see above]\nNo matching version of entire can be installed:\n  Reject:  pkg://helios-dev/[email protected]\n  Reason:  No version matching 'require' dependency SUNWcs can be installed\n    ----------------------------------------\n    Reject:  pkg://helios-dev/[email protected]\n             pkg://helios-dev/[email protected]\n    Reason:  Rejected by 'incorporate' dependency in proposed package 'consolidation/osnet/osnet-incorporation'\n    Reason:  Rejected by 'require' dependency in proposed package 'network/openssh-server'\n    Reason:  Rejected by 'require' dependency in proposed package 'service/network/chrony'\n    Reason:  Higher ranked publisher on-nightly was selected\n    Reject:  pkg://on-nightly/[email protected]\n    Reason:  Rejected by 'incorporate' dependency in proposed package 'consolidation/osnet/osnet-incorporation'\n    Reason:  Rejected by 'require' dependency in proposed package 'network/openssh-server'\n    Reason:  Rejected by 'require' dependency in proposed package 'service/network/chrony'\n    ----------------------------------------"
[ Feb  8 12:06:14 Method "start" exited with status 1. ]

$ pkg info omicron1
             Name: system/zones/brand/omicron1
          Summary: Oxide omicron1 brand support
         Category: System/Virtualization
            State: Installed
        Publisher: helios-dev
          Version: 1.0.16
           Branch: None
   Packaging Date: June  1, 2023 at 09:38:27 PM
Last Install Time: March 17, 2022 at 10:17:37 PM
 Last Update Time: June  4, 2023 at 10:38:30 PM
             Size: 7.24 MB
             FMRI: pkg://helios-dev/system/zones/brand/[email protected]:20230601T213827Z

$ pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
on-nightly                  origin   online F file:///home/luqman/helios/tmp/onu/repo.redist/
helios-dev     (non-sticky) origin   online F https://pkg.oxide.computer/helios/2/dev/

[luqmana]

Moving the helios-dev publisher above on-nightly gets things working.

$ pfexec pkg set-publisher --search-first helios-dev

$ pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
helios-dev     (non-sticky) origin   online F https://pkg.oxide.computer/helios/2/dev/
on-nightly                  origin   online F file:///home/luqman/helios/tmp/onu/repo.redist/

$ svcadm clear svc:/system/omicron/baseline

$ svcs baseline
STATE          STIME    FMRI
online         13:16:53 svc:/system/omicron/baseline:default