Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

oem-gce.service is broken after upgrading to flatcar 3139.2.0 #714

Closed
ffilippopoulos opened this issue Apr 12, 2022 · 8 comments
Closed

oem-gce.service is broken after upgrading to flatcar 3139.2.0 #714

ffilippopoulos opened this issue Apr 12, 2022 · 8 comments
Assignees
@jepio
Labels
kind/bug Something isn't working platform/GCE

Comments

@ffilippopoulos
Copy link

Description

We have a Kubernetes cluster running on gcp nodes using flatcar os. When we updated nodes from flatcar version 3033.2.4 to 3139.2.0 (latest) we noticed that oem-gce.service fails to start with the following logs:

core@master-k8s-exp-1-839c ~ $ journalctl -u oem-gce.service                                                                                                                                                       
Apr 12 09:29:25 localhost systemd[1]: Starting GCE Linux Agent...                                                                                                                                                  
Apr 12 09:29:26 localhost mkfs.ext4[981]: mke2fs 1.45.5 (07-Jan-2020)
Apr 12 09:29:26 localhost mkfs.ext4[981]: [82B blob data]
Apr 12 09:29:26 localhost mkfs.ext4[981]: Creating filesystem with 262144 4k blocks and 65536 inodes
Apr 12 09:29:26 localhost mkfs.ext4[981]: Filesystem UUID: 9376da71-5201-4916-b603-141a89463da7
Apr 12 09:29:26 localhost mkfs.ext4[981]: Superblock backups stored on blocks:
Apr 12 09:29:26 localhost mkfs.ext4[981]:         32768, 98304, 163840, 229376
Apr 12 09:29:26 localhost mkfs.ext4[981]: [41B blob data]
Apr 12 09:29:26 localhost mkfs.ext4[981]: [38B blob data]
Apr 12 09:29:26 localhost mkfs.ext4[981]: Creating journal (8192 blocks): done
Apr 12 09:29:27 localhost mkfs.ext4[981]: [75B blob data]
Apr 12 09:29:27 localhost umount[1033]: umount: /var/lib/flatcar-oem-gce.img: not mounted.
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd-nspawn[1714]: Spawning container oem-gce on /var/lib/flatcar-oem-gce.img.
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd-nspawn[1714]: Press ^] three times within 1s to kill container.
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd[1]: Started GCE Linux Agent.
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd-nspawn[1714]: + '[' -e /etc/default/instance_configs.cfg.template ']'
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd-nspawn[1714]: + echo -e '[InstanceSetup]\nset_host_keys = false'
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd-nspawn[1714]: + /usr/bin/google_instance_setup
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd-nspawn[1714]: /init.sh: /usr/bin/google_instance_setup: /usr/lib/python-exec/python3.9/python3: bad interpreter: No such file or directory
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd-nspawn[1714]: Container oem-gce failed with error code 126.
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd[1]: oem-gce.service: Main process exited, code=exited, status=126/n/a
Apr 12 09:29:33 master-k8s-exp-1-839c.c.uw-dev.internal systemd[1]: oem-gce.service: Failed with result 'exit-code'

Impact

Since this service is critical for a few gcp features, including setting up host ip routes, a lot of things broke. In particular, we noticed as our external load balancing to the cluster failed.

Environment and steps to reproduce

Latest flatcar on gcp nodes should be enough to observe this behaviour.

Additional information

Flatcar Container Linux by Kinvolk 3139.2.0 (Oklo)
Kernel: 5.15.32-flatcar
Kubernetes: 1.22.5
Container-Runtime: docker://20.10.12
@ffilippopoulos ffilippopoulos added the kind/bug Something isn't working label Apr 12, 2022
@jepio jepio self-assigned this Apr 12, 2022
@jepio
Copy link
Member

jepio commented Apr 12, 2022

Hi @ffilippopoulos,
thanks for reporting this, I stumbled across this and am in the process of righting this. Will fix in all branches.

Can I ask you a favor - would you be able to paste the journalctl -u oem-gce.service logs from a working version (3033.2.4) for me?

@ffilippopoulos
Copy link
Author

Hi @jepio Thanks for the quick response and for looking into this one!

Here are the requested logs:

Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal systemd[1]: Starting GCE Linux Agent...
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]: mke2fs 1.45.5 (07-Jan-2020)
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]: [82B blob data]
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]: Creating filesystem with 262144 4k blocks and 65536 inodes
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]: Filesystem UUID: bd72f8ca-0a79-4050-91df-08f2ab0478a4
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]: Superblock backups stored on blocks:
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]:         32768, 98304, 163840, 229376
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]: [41B blob data]
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]: [38B blob data]
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]: Creating journal (8192 blocks): done
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal mkfs.ext4[964464]: [75B blob data]
Apr 12 10:56:40 master-k8s-4271.c.uw-dev.internal umount[964467]: umount: /var/lib/flatcar-oem-gce.img: not mounted.
Apr 12 10:56:50 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: Spawning container oem-gce on /var/lib/flatcar-oem-gce.img.
Apr 12 10:56:50 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: Press ^] three times within 1s to kill container.
Apr 12 10:56:50 master-k8s-4271.c.uw-dev.internal systemd[1]: Started GCE Linux Agent.
Apr 12 10:56:50 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + '[' -e /etc/default/instance_configs.cfg.template ']'
Apr 12 10:56:50 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + /usr/bin/google_instance_setup
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO Running set_multiqueue.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO Could not set channels for eth0 to 2.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO Setting /proc/irq/27/smp_affinity_list to 0 for device virtio1.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /usr/bin/set_multiqueue: line 101: /proc/irq/27/smp_affinity_list: Read-only file system
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /proc/irq/27/smp_affinity_list: real affinity 0
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO Setting /proc/irq/28/smp_affinity_list to 0 for device virtio1.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /usr/bin/set_multiqueue: line 101: /proc/irq/28/smp_affinity_list: Read-only file system
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /proc/irq/28/smp_affinity_list: real affinity 0
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO Setting /proc/irq/29/smp_affinity_list to 1 for device virtio1.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /usr/bin/set_multiqueue: line 101: /proc/irq/29/smp_affinity_list: Read-only file system
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /proc/irq/29/smp_affinity_list: real affinity 1
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO Setting /proc/irq/30/smp_affinity_list to 1 for device virtio1.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /usr/bin/set_multiqueue: line 101: /proc/irq/30/smp_affinity_list: Read-only file system
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /proc/irq/30/smp_affinity_list: real affinity 1
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /usr/bin/set_multiqueue: line 133: /sys/class/net/eth0/queues/tx-0/xps_cpus: Read-only file system
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO /usr/bin/set_multiqueue: line 133: /sys/class/net/eth0/queues/tx-1/xps_cpus: Read-only file system
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO Queue 0 XPS=1 for /sys/class/net/eth0/queues/tx-0/xps_cpus
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal instance-setup[964697]: INFO Queue 1 XPS=2 for /sys/class/net/eth0/queues/tx-1/xps_cpus
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + /usr/bin/google_metadata_script_runner --script-type startup
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal startup-script[964728]: INFO Starting startup scripts.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal startup-script[964728]: INFO No startup scripts found in metadata.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal startup-script[964728]: INFO Finished running startup scripts.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + trap 'stopping=1 ; kill "${daemon_pids[@]}" || :' SIGTERM
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + daemon_pids=()
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + for d in accounts clock_skew network
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + daemon_pids+=($!)
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + for d in accounts clock_skew network
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + daemon_pids+=($!)
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + for d in accounts clock_skew network
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + daemon_pids+=($!)
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + NOTIFY_SOCKET=/run/systemd/notify
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + /usr/bin/systemd-notify --ready
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + /usr/bin/google_accounts_daemon
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + /usr/bin/google_network_daemon
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + /usr/bin/google_clock_skew_daemon
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: + wait -n 36 37 38
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal google-accounts[964737]: INFO Starting Google Accounts daemon.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal google-clock-skew[964738]: INFO Starting Google Clock Skew daemon.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal google-clock-skew[964738]: INFO Clock drift token has changed: 0.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal google-accounts[964737]: WARNING OS Login not installed.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: hwclock: Cannot access the Hardware Clock via any known method.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal systemd-nspawn[964618]: hwclock: Use the --verbose option to see the details of our search for an access method.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal google-clock-skew[964738]: WARNING Failed to sync system time with hardware clock.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal google-networking[964739]: INFO Starting Google Networking daemon.
Apr 12 10:56:51 master-k8s-4271.c.uw-dev.internal network-setup[964739]: INFO Ethernet interfaces: ['eth0'].
Apr 12 10:56:52 master-k8s-4271.c.uw-dev.internal google-ip-forwarding[964739]: INFO Changing eth0 IPs from

@tormath1 tormath1 moved this from Upcoming / Backlog to In Progress in Flatcar tactical, release planning, and roadmap Apr 12, 2022
@jepio jepio mentioned this issue Apr 12, 2022
Merged
1 task
@cpswan cpswan mentioned this issue Apr 12, 2022
Open
@cpswan
Copy link
Contributor

cpswan commented Apr 12, 2022

I was about to open a ticket reporting gcloud compute ssh asking for a password (that doesn't exist) but it looks like this is the root cause.

At least I can still use the core user.

@jepio
Copy link
Member

jepio commented Apr 12, 2022 •
edited
Loading

@cpswan / @ffilippopoulos

Would one of you be able to test a dev image: https://bucket.release.flatcar-linux.net/flatcar-jenkins/developer/developer/boards/amd64-usr/2022.04.07+dev-flatcar-master-5373/flatcar_production_gce.tar.gz? I'm curious if I fixed the issue with gcloud compute ssh or if more is broken.

The steps are basically: upload this file to google storage, and create an image from it (you can use this as a reference: https://github.com/kinvolk/flatcar-cloud-image-uploader/blob/master/google-cloud-flatcar-image-upload/upload_image.sh#L146-L167).

@cpswan
Copy link
Contributor

cpswan commented Apr 12, 2022

Trying that image now @jepio

@cpswan
Copy link
Contributor

cpswan commented Apr 12, 2022

The dev image seem to be working. oem-gce.service has come up, and I'm able to gcloud compute ssh without specifying the core user.

@ffilippopoulos
Copy link
Author

I can also verify that the dev image is solving the issue for us as well @jepio
oem-gce.service starts fine and our load balancing seems to work as expected.
In case you need the journalctl logs from using the dev image:

pr 13 07:45:34 localhost systemd[1]: Starting GCE Linux Agent...
Apr 13 07:45:35 localhost mkfs.ext4[1070]: mke2fs 1.46.4 (18-Aug-2021)
Apr 13 07:45:35 localhost mkfs.ext4[1070]: [82B blob data]
Apr 13 07:45:35 localhost mkfs.ext4[1070]: Creating filesystem with 262144 4k blocks and 65536 inodes
Apr 13 07:45:35 localhost mkfs.ext4[1070]: Filesystem UUID: 061dbee3-96a0-4eca-a065-2ebf96ba1ffd
Apr 13 07:45:35 localhost mkfs.ext4[1070]: Superblock backups stored on blocks:
Apr 13 07:45:35 localhost mkfs.ext4[1070]:         32768, 98304, 163840, 229376
Apr 13 07:45:35 localhost mkfs.ext4[1070]: [41B blob data]
Apr 13 07:45:35 localhost mkfs.ext4[1070]: [38B blob data]
Apr 13 07:45:35 localhost mkfs.ext4[1070]: Creating journal (8192 blocks): done
Apr 13 07:45:35 localhost mkfs.ext4[1070]: [75B blob data]
Apr 13 07:45:35 localhost umount[1176]: umount: /var/lib/flatcar-oem-gce.img: not mounted.
Apr 13 07:45:45 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: Spawning container oem-gce on /var/lib/flatcar-oem-gce.img.
Apr 13 07:45:45 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: Press ^] three times within 1s to kill container.
Apr 13 07:45:45 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd[1]: Started GCE Linux Agent.
Apr 13 07:45:45 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + '[' -e /etc/default/instance_configs.cfg.template ']'
Apr 13 07:45:45 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + echo -e '[InstanceSetup]\nset_host_keys = false'
Apr 13 07:45:45 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + /usr/bin/google_instance_setup
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO Running google_set_multiqueue.
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO Set channels for eth0 to 2.
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO Setting /proc/irq/31/smp_affinity_list to 0 for device virtio1.
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO /proc/irq/31/smp_affinity_list: real affinity 0
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO Setting /proc/irq/32/smp_affinity_list to 0 for device virtio1.
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO /proc/irq/32/smp_affinity_list: real affinity 0
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO Setting /proc/irq/33/smp_affinity_list to 1 for device virtio1.
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO /proc/irq/33/smp_affinity_list: real affinity 1
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO Setting /proc/irq/34/smp_affinity_list to 1 for device virtio1.
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO /proc/irq/34/smp_affinity_list: real affinity 1
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO Queue 0 XPS=1 for /sys/class/net/eth0/queues/tx-0/xps_cpus
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal instance-setup[2005]: INFO Queue 1 XPS=2 for /sys/class/net/eth0/queues/tx-1/xps_cpus
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + /usr/bin/google_metadata_script_runner --script-type startup
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal startup-script[2036]: INFO Starting startup scripts.
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal startup-script[2036]: INFO No startup scripts found in metadata.
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal startup-script[2036]: INFO Finished running startup scripts.
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + trap 'stopping=1 ; kill "${daemon_pids[@]}" || :' SIGTERM
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + daemon_pids=()
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + for d in accounts clock_skew network
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + daemon_pids+=($!)
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + /usr/bin/google_accounts_daemon
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + for d in accounts clock_skew network
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + daemon_pids+=($!)
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + for d in accounts clock_skew network
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + /usr/bin/google_clock_skew_daemon
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + daemon_pids+=($!)
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + NOTIFY_SOCKET=/run/systemd/notify
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + /usr/bin/systemd-notify --ready
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + /usr/bin/google_network_daemon
Apr 13 07:45:46 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: + wait -n 36 37 38
Apr 13 07:45:47 master-k8s-exp-1-1xr8.c.uw-dev.internal google-clock-skew[2040]: INFO Starting Google Clock Skew daemon.
Apr 13 07:45:47 master-k8s-exp-1-1xr8.c.uw-dev.internal google-clock-skew[2040]: INFO Clock drift token has changed: 0.
Apr 13 07:45:47 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: hwclock: Cannot access the Hardware Clock via any known method.
Apr 13 07:45:47 master-k8s-exp-1-1xr8.c.uw-dev.internal systemd-nspawn[2000]: hwclock: Use the --verbose option to see the details of our search for an access method.
Apr 13 07:45:47 master-k8s-exp-1-1xr8.c.uw-dev.internal google-clock-skew[2040]: WARNING Failed to sync system time with hardware clock.
Apr 13 07:45:47 master-k8s-exp-1-1xr8.c.uw-dev.internal google-networking[2041]: INFO Starting Google Networking daemon.
Apr 13 07:45:47 master-k8s-exp-1-1xr8.c.uw-dev.internal groupadd[2050]: group added to /etc/group: name=google-sudoers, GID=1000
Apr 13 07:45:47 master-k8s-exp-1-1xr8.c.uw-dev.internal groupadd[2050]: group added to /etc/gshadow: name=google-sudoers

@jepio
Copy link
Member

jepio commented Apr 25, 2022 •
edited
Loading

Thanks, merged the fix to all branches:

Will be part of the next release.

@jepio jepio closed this as completed Apr 25, 2022
Repository owner moved this from In Progress to Implemented in Flatcar tactical, release planning, and roadmap Apr 25, 2022
@cpswan cpswan mentioned this issue May 30, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jepio jepio

Labels
kind/bug Something isn't working platform/GCE
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cpswan @jepio @ffilippopoulos