/boot is full and system is boot looping #819

mrguitar · 2021-05-06T20:46:55Z

Describe the bug
It appears old kernels aren't getting cleaned up on my system which causes rpm-ostree stages to fail, and zincati tries to reapply updates every 10 minutes, rinse and repeat.

Reproduction steps
Steps to reproduce the behavior:

I'm not sure this is reproducible. This system was deployed pre stable stream.

Expected behavior
A clear and concise description of what you expected to happen.

Ideally zincati, or ostree, would degrade in a way that doesn't just keep rebooting like this.

Actual behavior
A clear and concise description of what actually happened.
Logs are below.

System details
-bare metal

Fedora CoreOS version
currently 33.20210328.3.0, originally deployed w/ a very old version.

Ignition config
I just used the basic template to inject my ssh key

Additional information
Add any other information about the problem here.

-- Logs begin at Fri 2019-10-04 02:38:54 CDT, end at Thu 2021-05-06 09:00:41 CDT. --
May 06 08:59:57 frito systemd[1]: Stopped Zincati Update Agent.
May 06 08:59:57 frito systemd[1]: zincati.service: Succeeded.
May 06 08:59:57 frito systemd[1]: Stopping Zincati Update Agent...
May 06 08:59:39 frito zincati[730]:     
May 06 08:59:39 frito zincati[730]:     error: Cleaning bootversions: Removing boot/loader.0: unlinkat(entries): Read-only file system
May 06 08:59:39 frito zincati[730]: [ERROR] failed to stage deployment: rpm-ostree deploy failed:
May 06 08:56:20 frito zincati[730]: [INFO ] target release '33.20210412.3.0' selected, proceeding to stage it
May 06 08:55:05 frito zincati[730]: [INFO ] current release detected as not a dead-end
May 06 08:54:57 frito systemd[1]: Started Zincati Update Agent.
May 06 08:54:57 frito zincati[730]: [INFO ] deployment 33.20210314.3.0 (db5401f9952d87d93fed66fc13ce1a837f8150ee55b1585434f33eba6f600df9) will be excluded from being a future update target
May 06 08:54:57 frito zincati[730]: [INFO ] found 1 other finalized deployment
May 06 08:54:57 frito zincati[730]: [INFO ] update strategy: immediate
May 06 08:54:57 frito zincati[730]: [INFO ] initialization complete, auto-updates logic enabled
May 06 08:54:54 frito zincati[730]: [INFO ] agent running on node 'aa4e7e319bf24ad796c80653f29194b4', in update group 'default'
May 06 08:54:54 frito zincati[730]: [INFO ] Cincinnati service: https://updates.coreos.fedoraproject.org
May 06 08:54:44 localhost zincati[730]: [INFO ] starting update agent (zincati 0.0.18)
May 06 08:54:44 localhost systemd[1]: Starting Zincati Update Agent...
-- Reboot --
May 06 08:52:05 frito systemd[1]: Stopped Zincati Update Agent.
May 06 08:52:05 frito systemd[1]: zincati.service: Succeeded.
May 06 08:52:05 frito systemd[1]: Stopping Zincati Update Agent...
May 06 08:52:01 frito zincati[723]: [INFO ] staged deployment '33.20210412.3.0' available, proceeding to finalize it and reboot
May 06 08:51:14 frito zincati[723]: [INFO ] target release '33.20210412.3.0' selected, proceeding to stage it
May 06 08:46:08 frito zincati[723]:     
May 06 08:46:08 frito zincati[723]:     error: Cleaning bootversions: Removing boot/loader.0: unlinkat(entries): Read-only file system
May 06 08:46:08 frito zincati[723]: [ERROR] failed to stage deployment: rpm-ostree deploy failed:
May 06 08:42:45 frito zincati[723]: [INFO ] target release '33.20210412.3.0' selected, proceeding to stage it
May 06 08:41:24 frito zincati[723]: [INFO ] current release detected as not a dead-end
May 06 08:41:23 frito systemd[1]: Started Zincati Update Agent.
May 06 08:41:23 frito zincati[723]: [INFO ] deployment 33.20210314.3.0 (db5401f9952d87d93fed66fc13ce1a837f8150ee55b1585434f33eba6f600df9) will be excluded from being a future update target
May 06 08:41:23 frito zincati[723]: [INFO ] found 1 other finalized deployment
May 06 08:41:23 frito zincati[723]: [INFO ] update strategy: immediate
May 06 08:41:23 frito zincati[723]: [INFO ] initialization complete, auto-updates logic enabled
May 06 08:41:20 frito zincati[723]: [INFO ] agent running on node 'aa4e7e319bf24ad796c80653f29194b4', in update group 'default'
May 06 08:41:20 frito zincati[723]: [INFO ] Cincinnati service: https://updates.coreos.fedoraproject.org
May 06 08:41:09 localhost zincati[723]: [INFO ] starting update agent (zincati 0.0.18)
May 06 08:41:09 localhost systemd[1]: Starting Zincati Update Agent...
-- Reboot --

$ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         error: Installing kernel: regfile copy: No space left on device
         check `journalctl -b -1 -u ostree-finalize-staged.service`
Deployments:
  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 33.20210412.3.0 (2021-04-27T18:27:38Z)
                    Commit: 8563aca63f27acdc80c7104486bbc50435dfa47eff28dc7e9209f2496b166076
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
                      Diff: 46 upgraded

● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 33.20210328.3.0 (2021-04-12T15:21:05Z)
                    Commit: 0fc4a4c205dbcdfd6ba68912bfbf2c90911e4a2341b3dda0a254ab6541224b83
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31

  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 33.20210314.3.0 (2021-03-29T14:51:43Z)
                    Commit: db5401f9952d87d93fed66fc13ce1a837f8150ee55b1585434f33eba6f600df9
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31

$ journalctl -b -1 -u ostree-finalize-staged.service
-- Logs begin at Fri 2019-10-04 02:38:54 CDT, end at Thu 2021-05-06 09:06:38 CDT. --
May 06 08:43:19 frito systemd[1]: Finished OSTree Finalize Staged Deployment.
May 06 08:52:14 frito systemd[1]: Stopping OSTree Finalize Staged Deployment...
May 06 08:52:14 frito systemd[14200]: selinux: avc:  netlink recvfrom: error 9
May 06 08:52:14 frito systemd[14200]: selinux: avc:  netlink recvfrom: error 9
May 06 08:52:14 frito ostree[14200]: Finalizing staged deployment
May 06 08:52:35 frito ostree[14200]: Copying /etc changes: 9 modified, 0 removed, 46 added
May 06 08:52:35 frito ostree[14200]: Copying /etc changes: 9 modified, 0 removed, 46 added
May 06 08:52:37 frito ostree[14200]: error: Installing kernel: regfile copy: No space left on device
May 06 08:52:37 frito systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
May 06 08:52:37 frito systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
May 06 08:52:37 frito systemd[1]: Stopped OSTree Finalize Staged Deployment.

$ df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       364M  346M     0 100% /boot

$ pwd
/boot/boot/ostree
[core@frito ostree]$ ls
fedora-coreos-2c3e12b844bef928487b9ea978744f81fe24cab77f51f715d34355f46a6c0d04  fedora-coreos-a5f8c1cd513bb95170dc8e8fa01a1738effd9484feef28e8f74fa685709af280
fedora-coreos-769d9b818d34cde43ad9a242104787151f06cdaed214d437967a03c261c862f5  fedora-coreos-a9e9f9f34c4f1ee9f39e17a75fc97c62034cbae378484823d56cac6107e5834a
fedora-coreos-7bbc78c7f5c49e3bd679517942d74bf1c3024647716b30a01aae83bb4102244b

The text was updated successfully, but these errors were encountered:

cgwalters · 2021-05-06T21:53:28Z

error: Cleaning bootversions: Removing boot/loader.0: unlinkat(entries): Read-only file system

Is the fatal error here. ostree supports /boot being read-only...but could it be that e.g. ext4 is hitting errors and remounting readonly? Something like dmesg | grep "Remounting filesystem read-only" ?

What's the output of systemctl status boot.mount?

cgwalters · 2021-05-06T21:53:56Z

Also the output of jq . < /sysroot/.coreos-aleph-version.json ?

mrguitar · 2021-05-06T22:05:50Z

I don't see a remount or read-only message in dmesg, but it is mounted ro

$ mount |grep boot
/dev/sda1 on /boot type ext4 (ro,nosuid,nodev,relatime,seclabel)

This looks normal:

$  systemctl status boot.mount
● boot.mount - CoreOS Dynamic Mount for /boot
     Loaded: loaded (/run/systemd/generator/boot.mount; generated)
     Active: active (mounted) since Thu 2021-05-06 08:54:40 CDT; 8h ago
      Where: /boot
       What: /dev/sda1
       Docs: https://github.com/coreos/fedora-coreos-config
      Tasks: 0 (limit: 4520)
     Memory: 40.0K
     CGroup: /system.slice/boot.mount

May 06 08:54:40 localhost systemd[1]: Mounting CoreOS Dynamic Mount for /boot...
May 06 08:54:40 localhost systemd[1]: Mounted CoreOS Dynamic Mount for /boot.

$ jq . < /sysroot/.coreos-aleph-version.json 
{
  "build": "30.20191002.0",
  "ref": "fedora/x86_64/coreos/testing",
  "ostree-commit": "326dd9ef07f7ebf5dc862e263ad7ffa33cf58d2055ef48ea8e27b6a9c387ce19",
  "imgid": "fedora-coreos-30.20191002.0-metal.x86_64.raw"
}

mrguitar · 2021-05-06T22:09:17Z

$ systemctl cat boot.mount
# /run/systemd/generator/boot.mount
# Automatically created by coreos-boot-mount-generator
[Unit]
Description=CoreOS Dynamic Mount for /boot
Documentation=https://github.com/coreos/fedora-coreos-config

Before=local-fs.target
Requires=systemd-fsck@dev-disk-by\x2dlabel-boot.service
After=systemd-fsck@dev-disk-by\x2dlabel-boot.service

[Mount]
What=/dev/disk/by-label/boot
Where=/boot
Options=ro,nodev,nosuid

I don't know if the generator sets ro when it's full, but I don't believe /boot should be read-only

dustymabe · 2021-05-07T02:05:07Z

looks like the generator is defaulting to ro:

https://github.com/coreos/fedora-coreos-config/blob/1de21ffa98bb22995e5b059501e1955bf52b562c/overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator#L80

And I see that on my system:

[core@pcengines ~]$ rpm-ostree status 
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Fri 2021-05-07 01:59:57 UTC)
Deployments:
● ostree://fedora:fedora/x86_64/coreos/next
                   Version: 34.20210503.1.0 (2021-05-04T09:22:11Z)
                    Commit: 1b7e0d318f5e78013434236ad17edb8b65297f0e8de3e765562fdedc1a7e52ab
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39

  ostree://fedora:fedora/x86_64/coreos/next
                   Version: 34.20210427.1.0 (2021-04-27T18:45:58Z)
                    Commit: 8eafe2839217fa4feaa30834df4d6025b15d105dbb6814d1b2cf668c7e997bb1
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
[core@pcengines ~]$ 
[core@pcengines ~]$ findmnt /boot
TARGET SOURCE    FSTYPE OPTIONS
/boot  /dev/sda3 ext4   ro,nosuid,nodev,relatime,seclabel

jlebon · 2021-05-07T14:41:02Z

The read-only /boot is expected now with coreos/fedora-coreos-config#659. OSTree knows to remount it in read-write to do modifications.

Hmm, it seems like somehow rpm-ostree is triggering a sysroot cleanup on an OstreeSysroot object without calling ostree_sysroot_set_mount_namespace_in_use and so ostree doesn't try to remount rw before pruning.

jlebon · 2021-05-07T14:52:48Z

Can you also provide the recent-ish output of journalctl -u rpm-ostreed just in case?

mrguitar · 2021-05-07T15:08:17Z

-- Reboot --
May 06 08:52:13 frito systemd[1]: Stopped rpm-ostree System Management Daemon.
May 06 08:52:13 frito systemd[1]: rpm-ostreed.service: Succeeded.
May 06 08:52:13 frito rpm-ostree[7373]: In idle state; will auto-exit in 62 seconds
May 06 08:52:13 frito rpm-ostree[7373]: Process [pid: 12122 uid: 980 unit: (unknown)] disconnected from transaction progress
May 06 08:52:13 frito rpm-ostree[7373]: Unlocked sysroot
May 06 08:52:05 frito systemd[1]: Stopping rpm-ostree System Management Daemon...
May 06 08:52:05 frito rpm-ostree[7373]: Txn FinalizeDeployment on /org/projectatomic/rpmostree1/fedora_coreos successful
May 06 08:52:05 frito rpm-ostree[7373]: Finalized deployment; rebooting into 8563aca63f27acdc80c7104486bbc50435dfa47eff28dc7e9209f2496b166076
May 06 08:52:05 frito rpm-ostree[7373]: Process [pid: 12122 uid: 980 unit: zincati.service] connected to transaction progress
May 06 08:52:05 frito rpm-ostree[7373]: Initiated txn FinalizeDeployment for client(dbus:1.31 unit:zincati.service uid:980): /org/projectatomic/rpmostree1/fedora_coreos
May 06 08:52:05 frito rpm-ostree[7373]: Locked sysroot
May 06 08:51:57 frito rpm-ostree[7373]: In idle state; will auto-exit in 63 seconds
May 06 08:51:57 frito rpm-ostree[7373]: Process [pid: 7370 uid: 980 unit: zincati.service] disconnected from transaction progress
May 06 08:51:57 frito rpm-ostree[7373]: Unlocked sysroot
May 06 08:51:48 frito rpm-ostree[7373]: Txn Deploy on /org/projectatomic/rpmostree1/fedora_coreos successful
May 06 08:51:48 frito rpm-ostree[7373]: libostree pull from 'fedora' for fedora/x86_64/coreos/stable complete
                                        security: GPG: commit 
                                        security: SIGN: disabled http: TLS
                                        non-delta: meta: 2 content: 0
                                        transfer: secs: 2 size: 788 bytes
May 06 08:51:46 frito rpm-ostree[7373]: libostree pull from 'fedora' for 0 refs complete
                                        security: GPG: commit 
                                        security: SIGN: disabled http: TLS
                                        non-delta: meta: 11 content: 0
                                        transfer: secs: 17 size: 6.5 kB
May 06 08:51:28 frito rpm-ostree[7373]: libostree pull from 'fedora' for fedora/x86_64/coreos/stable complete
                                        security: GPG: commit 
                                        security: SIGN: disabled http: TLS
                                        non-delta: meta: 1 content: 0
                                        transfer: secs: 2 size: 592 bytes
May 06 08:51:26 frito rpm-ostree[7373]: Process [pid: 7370 uid: 980 unit: zincati.service] connected to transaction progress
May 06 08:51:26 frito rpm-ostree[7373]: Initiated txn Deploy for client(dbus:1.28 unit:zincati.service uid:980): /org/projectatomic/rpmostree1/fedora_coreos
May 06 08:51:26 frito rpm-ostree[7373]: Locked sysroot
May 06 08:51:22 frito systemd[1]: Started rpm-ostree System Management Daemon.
May 06 08:51:22 frito rpm-ostree[7373]: In idle state; will auto-exit in 61 seconds
May 06 08:51:14 frito rpm-ostree[7373]: Reading config file '/etc/rpm-ostreed.conf'
May 06 08:51:14 frito systemd[7373]: selinux: avc:  netlink recvfrom: error 9
May 06 08:51:14 frito systemd[7373]: selinux: avc:  netlink recvfrom: error 9
May 06 08:51:14 frito systemd[1]: Starting rpm-ostree System Management Daemon...
May 06 08:47:10 frito systemd[1]: rpm-ostreed.service: Succeeded.
May 06 08:47:10 frito rpm-ostree[3515]: In idle state; will auto-exit in 64 seconds
May 06 08:46:08 frito rpm-ostree[3515]: In idle state; will auto-exit in 62 seconds
May 06 08:46:08 frito rpm-ostree[3515]: Process [pid: 3512 uid: 980 unit: zincati.service] disconnected from transaction progress
May 06 08:46:08 frito rpm-ostree[3515]: Unlocked sysroot
May 06 08:45:59 frito rpm-ostree[3515]: Txn Deploy on /org/projectatomic/rpmostree1/fedora_coreos failed: Cleaning bootversions: Removing boot/loader.0: unlinkat(entries): Read-only file system
May 06 08:43:19 frito rpm-ostree[3515]: libostree pull from 'fedora' for fedora/x86_64/coreos/stable complete
                                        security: GPG: commit 
                                        security: SIGN: disabled http: TLS
                                        non-delta: meta: 2 content: 0
                                        transfer: secs: 2 size: 788 bytes
May 06 08:43:16 frito rpm-ostree[3515]: libostree pull from 'fedora' for 0 refs complete
                                        security: GPG: commit 
                                        security: SIGN: disabled http: TLS
                                        non-delta: meta: 11 content: 0
                                        transfer: secs: 18 size: 6.5 kB
May 06 08:42:57 frito rpm-ostree[3515]: libostree pull from 'fedora' for fedora/x86_64/coreos/stable complete
                                        security: GPG: commit 
                                        security: SIGN: disabled http: TLS
                                        non-delta: meta: 1 content: 0
                                        transfer: secs: 3 size: 592 bytes
May 06 08:42:54 frito rpm-ostree[3515]: Process [pid: 3512 uid: 980 unit: zincati.service] connected to transaction progress
May 06 08:42:54 frito rpm-ostree[3515]: Initiated txn Deploy for client(dbus:1.26 unit:zincati.service uid:980): /org/projectatomic/rpmostree1/fedora_coreos
May 06 08:42:54 frito rpm-ostree[3515]: Locked sysroot
May 06 08:42:51 frito systemd[1]: Started rpm-ostree System Management Daemon.
May 06 08:42:51 frito rpm-ostree[3515]: In idle state; will auto-exit in 64 seconds
May 06 08:42:46 frito rpm-ostree[3515]: Reading config file '/etc/rpm-ostreed.conf'
May 06 08:42:45 frito systemd[3515]: selinux: avc:  netlink recvfrom: error 9
May 06 08:42:45 frito systemd[3515]: selinux: avc:  netlink recvfrom: error 9
May 06 08:42:45 frito systemd[1]: Starting rpm-ostree System Management Daemon...
May 06 08:42:20 frito systemd[1]: rpm-ostreed.service: Succeeded.
May 06 08:42:20 frito rpm-ostree[749]: In idle state; will auto-exit in 63 seconds
May 06 08:41:17 frito rpm-ostree[749]: In idle state; will auto-exit in 63 seconds
May 06 08:41:17 frito systemd[1]: Started rpm-ostree System Management Daemon.
May 06 08:41:10 localhost rpm-ostree[749]: Reading config file '/etc/rpm-ostreed.conf'
May 06 08:41:10 localhost systemd[749]: selinux: avc:  netlink recvfrom: error 9
May 06 08:41:10 localhost systemd[749]: selinux: avc:  netlink recvfrom: error 9
May 06 08:41:10 localhost systemd[1]: Starting rpm-ostree System Management Daemon...
-- Reboot --

This is suspicious: selinux: avc: netlink recvfrom: error 9

but this looks like the problem: Txn Deploy on /org/projectatomic/rpmostree1/fedora_coreos failed: Cleaning bootversions: Removing boot/loader.0: unlinkat(entries): Read-only file system

miabbott · 2021-05-11T14:05:36Z

Seeing a similar report downstream in RHCOS - https://bugzilla.redhat.com/show_bug.cgi?id=1959327

miabbott · 2021-05-11T14:08:50Z

systemd[749]: selinux: avc: netlink recvfrom: error 9

Hits on https://bugzilla.redhat.com/show_bug.cgi?id=1883797, which was apparently solved by a reboot?

Might be worth opening a separate issue for that error with the contents of ausearch -i -m avc,user_avc,selinux_err,user_selinux_err

mrguitar · 2021-05-12T22:46:12Z

Remounting /boot as rw doesn't do much besides let me remove the extraneous kernels. Removing those let's zincati & rpm-ostree apply the next update. Unfortunately I have to re-provision need this system because I need it working. I'll close this issue; hopefully this is a freak thing that won't crop up elsewhere.

dustymabe · 2021-05-14T15:38:56Z

I do wish we knew what happened here. If anyone sees something similar, please speak up.

mrguitar · 2021-05-14T16:47:04Z

I did leave the hard drive in tact so I can boot this back up if needed. It's not super convenient to do so, but totally possible.

faust64 · 2022-03-15T21:18:48Z

Upgrading from OCP 4.8 to 4.9, I'm just seeing this on about all nodes I have (except my masters, that are done rebooting)

I0315 20:58:16.806003   87423 rpm-ostree.go:325] Running captured: rpm-ostree status
I0315 20:58:16.857552   87423 daemon.go:853] State: idle
Warning: failed to finalize previous deployment
         error: Installing kernel: regfile copy: No space left on device
         check `journalctl -b -1 -u ostree-finalize-staged.service`
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7ed9cbc1718a68b94df974287141d7c3cb6451326fe5d6fddf91f4acd9b7c7f
              CustomOrigin: Managed by machine-config-operator
                   Version: 48.84.202202142303-0 (2022-02-14T23:07:16Z)

  pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a483b8f61dacaa714c6fb570ba5be023978ce91a60fdced2f410bdee8b1446bc
              CustomOrigin: Managed by machine-config-operator
                   Version: 48.84.202112022303-0 (2021-12-02T23:07:04Z)

...

I0315 21:02:20.230244   87423 daemon.go:806] Current config: rendered-ingress-opco1-9a8c7e54475496cf7e1a97a10f47c408
I0315 21:02:20.230343   87423 daemon.go:807] Desired config: rendered-ingress-opco1-e778fac42bd6244fdf1a3411d05c06a5
I0315 21:02:20.239932   87423 update.go:1874] Disk currentConfig rendered-ingress-opco1-e778fac42bd6244fdf1a3411d05c06a5 overrides node's currentConfig annotation rendered-ingress-opco1-9a8c7e54475496cf7e1a97a10f47c408
I0315 21:02:20.243024   87423 daemon.go:1089] Validating against pending config rendered-ingress-opco1-e778fac42bd6244fdf1a3411d05c06a5
E0315 21:02:20.243203   87423 writer.go:135] Marking Degraded due to: unexpected on-disk state validating against rendered-ingress-opco1-e778fac42bd6244fdf1a3411d05c06a5: expected target osImageURL "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae92a919cb6da4d1a5d832f8bc486ae92e55bf3814ebab94bf4baa4c4bcde85d", have "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7ed9cbc1718a68b94df974287141d7c3cb6451326fe5d6fddf91f4acd9b7c7f"

I can oc debug on those faulty node, and found 3 folders in /boot/ostree, while I was expecting 2:

sh-4.4# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb3       364M  297M   44M  88% /boot
sh-4.4# ls -l ostree/
total 6
drwxr-xr-x. 2 root root 1024 Mar 15 13:54 rhcos-1ad14a13f559779756766f6215e19fc5d8b8922c8f3d00c699cc0bfcb1f661be
drwxr-xr-x. 2 root root 1024 Mar 15 20:16 rhcos-258d513c69a5b29e006dae1dea3506f4da8d18e78b3ee5a39d5f59657b274544
drwxr-xr-x. 2 root root 1024 Mar 15 08:46 rhcos-b98be60347b2ad33d10c3092c4515f118892365267832c55b08d86f0d376b2fe
sh-4.4# journalctl -fu ostree-finalize-staged.service
-- Logs begin at Fri 2022-03-11 16:48:09 UTC. --
Mar 15 20:16:21 maint7-ocp-green-2 systemd[1]: Stopping OSTree Finalize Staged Deployment...
Mar 15 20:16:22 maint7-ocp-green-2 ostree[918706]: Finalizing staged deployment
Mar 15 20:16:24 maint7-ocp-green-2 ostree[918706]: Copying /etc changes: 32 modified, 0 removed, 235 added
Mar 15 20:16:24 maint7-ocp-green-2 ostree[918706]: Copying /etc changes: 32 modified, 0 removed, 235 added
Mar 15 20:16:26 maint7-ocp-green-2 ostree[918706]: error: Installing kernel: regfile copy: No space left on device
Mar 15 20:16:26 maint7-ocp-green-2 systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited status=1
Mar 15 20:16:26 maint7-ocp-green-2 systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Mar 15 20:16:26 maint7-ocp-green-2 systemd[1]: Stopped OSTree Finalize Staged Deployment.
Mar 15 20:16:26 maint7-ocp-green-2 systemd[1]: ostree-finalize-staged.service: Consumed 1.702s CPU time
-- Reboot --

hint, maybe: this is my second upgrade on that cluster today
although strangely enough: masters are OK. Same partition sizes, only two systems

mrguitar added the kind/bug label May 6, 2021

mrguitar closed this as completed May 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

/boot is full and system is boot looping #819

/boot is full and system is boot looping #819

mrguitar commented May 6, 2021 •

edited by dustymabe

Loading

cgwalters commented May 6, 2021

cgwalters commented May 6, 2021

mrguitar commented May 6, 2021 •

edited by dustymabe

Loading

mrguitar commented May 6, 2021 •

edited

Loading

dustymabe commented May 7, 2021

jlebon commented May 7, 2021

jlebon commented May 7, 2021

mrguitar commented May 7, 2021 •

edited

Loading

miabbott commented May 11, 2021

miabbott commented May 11, 2021

mrguitar commented May 12, 2021

dustymabe commented May 14, 2021

mrguitar commented May 14, 2021

faust64 commented Mar 15, 2022 •

edited

Loading

/boot is full and system is boot looping #819

/boot is full and system is boot looping #819

Comments

mrguitar commented May 6, 2021 • edited by dustymabe Loading

cgwalters commented May 6, 2021

cgwalters commented May 6, 2021

mrguitar commented May 6, 2021 • edited by dustymabe Loading

mrguitar commented May 6, 2021 • edited Loading

dustymabe commented May 7, 2021

jlebon commented May 7, 2021

jlebon commented May 7, 2021

mrguitar commented May 7, 2021 • edited Loading

miabbott commented May 11, 2021

miabbott commented May 11, 2021

mrguitar commented May 12, 2021

dustymabe commented May 14, 2021

mrguitar commented May 14, 2021

faust64 commented Mar 15, 2022 • edited Loading

mrguitar commented May 6, 2021 •

edited by dustymabe

Loading

mrguitar commented May 6, 2021 •

edited by dustymabe

Loading

mrguitar commented May 6, 2021 •

edited

Loading

mrguitar commented May 7, 2021 •

edited

Loading

faust64 commented Mar 15, 2022 •

edited

Loading