Upgrade 2023.10.5 -> 2023.11.0 fails on generic x86-64

[uphillbattle]

The problem

SSIA

Upgrade fails and rolls back to 2023.10.5 (takes forever, by the way - database cleanup?)

Supervisor logs:

23-11-02 14:08:28 INFO (MainThread) [supervisor.backups.manager] Backup 64ca3500 starting stage addon_repositories
23-11-02 14:08:28 INFO (MainThread) [supervisor.backups.manager] Backup 64ca3500 starting stage docker_config
23-11-02 14:08:28 INFO (MainThread) [supervisor.backups.manager] Creating new partial backup with slug 64ca3500
23-11-02 14:08:28 INFO (MainThread) [supervisor.backups.manager] Backup 64ca3500 starting stage home_assistant
23-11-02 14:08:28 INFO (MainThread) [supervisor.homeassistant.module] Backing up Home Assistant Core config folder
23-11-02 14:08:39 INFO (MainThread) [supervisor.homeassistant.module] Backup Home Assistant Core config folder done
23-11-02 14:08:39 INFO (MainThread) [supervisor.backups.manager] Backup 64ca3500 starting stage finishing_file
23-11-02 14:08:39 INFO (MainThread) [supervisor.backups.manager] Creating partial backup with slug 64ca3500 completed
23-11-02 14:08:39 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2023.11.0
23-11-02 14:08:39 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/generic-x86-64-homeassistant:2023.10.5 to ghcr.io/home-assistant/generic-x86-64-homeassistant:2023.11.0
23-11-02 14:08:39 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/generic-x86-64-homeassistant with tag 2023.11.0.
23-11-02 14:10:31 INFO (SyncWorker_0) [supervisor.docker.manager] Stopping homeassistant application
23-11-02 14:10:31 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 8
23-11-02 14:10:31 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
23-11-02 14:10:41 INFO (SyncWorker_0) [supervisor.docker.manager] Cleaning homeassistant application
23-11-02 14:10:41 INFO (MainThread) [supervisor.homeassistant.module] Update pulse/client.config: /data/tmp/homeassistant_pulse
23-11-02 14:10:42 INFO (MainThread) [supervisor.docker.homeassistant] Starting Home Assistant ghcr.io/home-assistant/generic-x86-64-homeassistant with version 2023.11.0
23-11-02 14:10:42 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready
23-11-02 14:10:44 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
23-11-02 14:10:44 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
23-11-02 14:10:44 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/jcwillox/hassio-rclone-backup repository
23-11-02 14:10:44 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
23-11-02 14:10:44 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
23-11-02 14:10:44 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
23-11-02 14:10:44 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
23-11-02 14:10:50 INFO (MainThread) [supervisor.store] Loading add-ons from store: 73 all - 0 new - 0 remove
23-11-02 14:10:50 INFO (MainThread) [supervisor.store] Loading add-ons from store: 73 all - 0 new - 0 remove

23-11-02 14:15:43 ERROR (MainThread) [supervisor.homeassistant.core] No API response in 5 minutes, assuming core has had a fatal startup error
23-11-02 14:15:43 CRITICAL (MainThread) [supervisor.homeassistant.core] HomeAssistant update failed -> rollback!
23-11-02 14:15:43 INFO (MainThread) [supervisor.resolution.module] Create new issue update_rollback - core / None
23-11-02 14:15:43 INFO (MainThread) [supervisor.homeassistant.core] A backup of the logfile is stored in /config/home-assistant-rollback.log
23-11-02 14:15:43 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2023.10.5
23-11-02 14:15:43 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/generic-x86-64-homeassistant:2023.11.0 to ghcr.io/home-assistant/generic-x86-64-homeassistant:2023.10.5
23-11-02 14:15:43 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/generic-x86-64-homeassistant with tag 2023.10.5.
23-11-02 14:15:43 INFO (SyncWorker_6) [supervisor.docker.manager] Stopping homeassistant application
23-11-02 14:15:48 INFO (SyncWorker_6) [supervisor.docker.manager] Cleaning homeassistant application
23-11-02 14:15:48 INFO (MainThread) [supervisor.homeassistant.module] Update pulse/client.config: /data/tmp/homeassistant_pulse
23-11-02 14:15:48 INFO (MainThread) [supervisor.docker.homeassistant] Starting Home Assistant ghcr.io/home-assistant/generic-x86-64-homeassistant with version 2023.10.5
23-11-02 14:15:48 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready
23-11-02 14:15:50 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
23-11-02 14:15:50 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
23-11-02 14:15:50 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
23-11-02 14:15:50 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/jcwillox/hassio-rclone-backup repository
23-11-02 14:15:50 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
23-11-02 14:15:50 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
23-11-02 14:15:50 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
23-11-02 14:15:51 INFO (MainThread) [supervisor.store] Loading add-ons from store: 73 all - 0 new - 0 remove
23-11-02 14:15:51 INFO (MainThread) [supervisor.store] Loading add-ons from store: 73 all - 0 new - 0 remove

What version of Home Assistant Core has the issue?

2023.11.0

What was the last working version of Home Assistant Core?

2023.10.5

What type of installation are you running?

Home Assistant OS

Integration causing the issue

core

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[uphillbattle]

Oh, and by the way, the upgrade was smooth without drama on another x86-64 as well as on an RPi4-64, so I don't get it...

[lukaszswitaj]

Where on earth did you discover version 2023.11.1?

[uphillbattle]

Where on earth did you discover version 2023.11.1?

I didn't. A typo - sorry - should of course be 2023.11.0.

[sabbie-github]

Same issue here on RPi 4 64 4gb. Upgrade fails and rolls back to 2023.10.5 (takes forever)

[home-assistant]

Hey there @home-assistant/supervisor, mind taking a look at this issue as it has been labeled with an integration (hassio) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of hassio can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign hassio Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

hassio documentation
hassio source
_{^{(message by IssueLinks)}}

[reaper7]

I tried several times, finally succeeded.
Probably there is some timeout when downloading add-on updates in the background.

[Taraman17]

Same issue for me on RPI3:

23-11-03 13:48:25 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
23-11-03 13:48:25 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.check] System checks complete
23-11-03 13:48:26 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
23-11-03 13:48:27 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
23-11-03 13:48:27 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
23-11-03 13:48:27 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
23-11-03 13:50:14 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
23-11-03 13:51:05 ERROR (MainThread) [supervisor.homeassistant.core] No API response in 5 minutes, assuming core has had a fatal startup error
23-11-03 13:51:05 CRITICAL (MainThread) [supervisor.homeassistant.core] HomeAssistant update failed -> rollback!
23-11-03 13:51:05 INFO (MainThread) [supervisor.resolution.module] Create new issue update_rollback - core / None
23-11-03 13:51:05 INFO (MainThread) [supervisor.homeassistant.core] A backup of the logfile is stored in /config/home-assistant-rollback.log
23-11-03 13:51:05 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2023.10.5
23-11-03 13:51:05 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/raspberrypi3-homeassistant:2023.11.0 to ghcr.io/home-assistant/raspberrypi3-homeassistant:2023.10.5
23-11-03 13:51:05 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/raspberrypi3-homeassistant with tag 2023.10.5.
23-11-03 13:51:06 INFO (SyncWorker_6) [supervisor.docker.manager] Stopping homeassistant application
23-11-03 13:51:12 INFO (SyncWorker_6) [supervisor.docker.manager] Cleaning homeassistant application

[Knoweevil]

I had the same issue using an RPI3 and tried 5 or 6 times making a backup each time. I then tried again without backing up (who needs more than one backup right) and success! I'm assuming any timeout issues were helped by not taking time to backup!

[Taraman17]

For me even without backup no success - 6 tries by now...

[sabbie-github]

It updated without the backup for me on x+1 try, but then failed to boot, got stuck starting services. 3 hard resets and a near 10 minute boot later it seems to run now...

[Lucianosouza99]

Para mim, mesmo sem backup, não tive sucesso - 6 tentativas até agora...

Any solution? So far I haven't been successful

[awedaei]

Any update? I tried everything but it won’t allow me to update

[uphillbattle]

On the 6th or so try yesterday night, and after not having attempted the update for more than 24 hours, and making sure that the backup checkbox was unchecked, my update came through 🥳

I’m guessing, as @Knoweevil mentioned, that it may some kind of timeout which may be set a bit short.

[Taraman17]

With Release 2023.11.1 and backup off still no update possible

[sabbie-github]

Anyone having boot issues? Managed 2023.11 without enabling backup and raspberry pi4 needs unplugging a few times to boot properly. Keeps freezing starting services - never happened before, only since 2023.11 core update

[miguelangel-nubla]

I was having the same issues, also when rebooting a working 2023.11.1
The thing that works for me so far is reverting home assistant operating system to 10.5
ha os update --version 10.5

ha core 2023.11.1 working so far.

[uphillbattle]

On the 6th or so try yesterday night, and after not having attempted the update for more than 24 hours, and making sure that the backup checkbox was unchecked, my update came through 🥳

Finally succeeded in updating 2023.11.0, but further updating to 2023.11.1: Same problem, rolls back to 2023.11.0 after a while.

[Knoweevil]

I'm having the same problem updating to 2023.11.1 despite not creating a backup - I'm going to keep trying but it's becoming a real chore ....

[Knoweevil]

Managed to update to 2023.11.1 after about 10/11 attempts, without creating a backup :-(

[Lucianosouza99]

I still can't update, I tried several times and nothing, do we have a solution? When I run the command ha core check me the message

Error: Can't execute command: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/qemux86-64-homeassistant:latest/json: Not Found ("No such image: ghcr.io/home-assistant/qemux86-64-homeassistant:latest")

[TheRealBenForce]

Also having trouble going 10.5 > 11.1

23-11-05 20:44:15 INFO (MainThread) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/yellow-homeassistant:2023.10.5 to ghcr.io/home-assistant/yellow-homeassistant:2023.11.1 23-11-05 20:44:15 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/yellow-homeassistant with tag 2023.11.1. 23-11-05 20:48:41 WARNING (MainThread) [supervisor.misc.tasks] Watchdog/Application found a problem with observer plugin! 23-11-05 20:48:47 INFO (SyncWorker_7) [supervisor.docker.manager] Stopping hassio_observer application 23-11-05 20:49:24 INFO (SyncWorker_7) [supervisor.docker.manager] Cleaning hassio_observer application 23-11-05 20:49:24 INFO (MainThread) [supervisor.plugins.observer] Starting observer plugin 23-11-05 20:49:30 INFO (MainThread) [supervisor.docker.observer] Starting Observer ghcr.io/home-assistant/aarch64-hassio-observer with version 2023.06.0 - 172.30.32.6 23-11-05 20:49:45 ERROR (MainThread) [supervisor.api.ingress] Stream error with http://172.30.33.2:1337/stable-d7a2b4936af1bfd80cb96f2567af68badc2325e3/static/out/vs/workbench/workbench.web.main.js: Cannot write to closing transport 23-11-05 20:51:07 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/yellow-homeassistant:2023.11.1: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/yellow-homeassistant:2023.11.1/json: Not Found ("No such image: ghcr.io/home-assistant/yellow-homeassistant:2023.11.1") 23-11-05 20:51:07 WARNING (MainThread) [supervisor.homeassistant.core] Updating Home Assistant image failed

Sorry I don't know why the formatting is all jacked up. Here is the important line:
23-11-05 20:51:07 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/yellow-homeassistant:2023.11.1: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/home-assistant/yellow-homeassistant:2023.11.1/json: Not Found ("No such image: ghcr.io/home-assistant/yellow-homeassistant:2023.11.1")

[uphillbattle]

I managed to update from 11.0 to 11.1 on the second attempt; making sure that the "create backup" checkbox was unchecked (made a backup (manually) first).

[agners]

The rollback is being done by the Supervisor. This is not a Core issue, but a Supervisor problem.

The problem has been recognized and addressed, see home-assistant/supervisor#4655. There will be a new Supervisor release soon, once it is out, the update should be working again.