Beta 3549.1.0: missing logs in /var/log/journal

[defo89]

Description

After our QA servers upgraded to new beta 3549.1.0, we noticed that systemd log shipping stopped working.
It looks that logs are not saved to /var/log/journal - they are present only in /run/log/journal

Impact

No logs under /var/log/journal

Environment and steps to reproduce

1. Set-up: 3549.1.0
2. Task: logs

Additional information

3510.1.0:

# find /var/log/journal -print
/var/log/journal
/var/log/journal/31d149ee9da74ddf9e23f9c5aa9d00c8
/var/log/journal/31d149ee9da74ddf9e23f9c5aa9d00c8/user-1524402668.journal
/var/log/journal/31d149ee9da74ddf9e23f9c5aa9d00c8/system.journal
/var/log/journal/remote

3549.1.0:

# find /var/log/journal -print
/var/log/journal
/var/log/journal/remote

[tormath1]

Hello @defo89 and thanks for using beta in QA,

that's interesting - this issue has been caught a few weeks ago (#989) and was supposed to be fixed in beta and alpha last releases but I confirm it does not work as expected on beta:

core@localhost ~ $ journalctl --verify
PASS: /run/log/journal/a3316ef872944f5db33a510d30a8f73e/system.journal
PASS: /run/log/journal/bf755d2675134d1b952a636e2d70c06b/system.journal

while on alpha:

core@localhost ~ $ journalctl --verify
PASS: /run/log/journal/aceb0de1b4da450ca0c94d8eaeb2870f/system.journal
PASS: /var/log/journal/dc7518a9602d4c1e86d2a9854e6bf2ce/system.journal

[tormath1]

@defo89 while we're investigating, there are two workarounds:

• reboot the instance (it only appears on first boot like the initial issue)
• journalctl --flush once booted

[tormath1]

While everything looks good from the git side, the issue appears to be caused by the build system of Flatcar. A PR that should fix the issue and a PR that provides a test have been opened.
In the meantime, it's possible to use one of the suggested workarounds or use the Alpha version.

[tormath1]

Hello @defo89, we just released a new set of maintenance releases including a bugfix on Beta for this issue (https://www.flatcar.org/releases#release-3549.1.1). I just booted a beta-35491.1 and the systemd journal logs are now correctly persisted. I'm closing this issue. Thanks again for using Flatcar Beta!

[defo89]

Thank you @tormath1!