3602.2.0 files overwrite change of behaviour

[ninjasftw]

Description

AWS AMI image based in EU-WEST-2 (ami-0bb8a98c666ad7adf) for release 3602.2.0 appears to have changed the behaviour for writing files to core os settings
Attempting to spin up an instance results in an error in the ingnition-files service

CRITICAL : Ignition failed: failed to create files: failed to create files: error creating /sysroot/etc/flatcar/update.conf: error creating file "/sysroot/etc/flatcar/update.conf": A file exists there already and overwrite is false

I am using Flatcar for a clusters that can't have random restarts so running with REBOOT_STRATEGY=off

variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /etc/flatcar/update.conf
      contents:
        inline: |
                    REBOOT_STRATEGY=off

If I remove that then the AMI boots fine or I can specify overwrite: true

I also however set some custom limits eg

variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /etc/security/limits.conf
      mode: 0644
      append:
        - inline: |
            root soft nofile 1048576
            root hard nofile 1048576

and that is now reporting the same error message

CRITICAL : Ignition failed: failed to create files: failed to create files: error creating /sysroot/etc/security/limits.conf: error creating file "/sysroot/etc/security/limits.conf": A file exists there already and overwrite is false

I can fix it by changing to contents from append and setting overwrite

    - path: /etc/security/limits.conf
      mode: 0644
      overwrite: true
      contents:
        inline: |
            root soft nofile 1048576
            root hard nofile 1048576

Impact

I have work arounds for everything that was failing however it required changes to the build pipelines that were working previously

Environment and steps to reproduce

1. Set-up: Start up a new EC2 instance using release 3602.2.0
2. Task: Create a new EC2 instance with some custom settings involving the overwriting of config files
3. Action(s):
   a. Start up ec2 instance using custom configuration

variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /etc/security/limits.conf
      mode: 0644
      append:
        - inline: |
            root soft nofile 1048576
            root hard nofile 1048576

4. Error: ignition-files service is unable to write the custom configurations and the boot process crashes

CRITICAL : Ignition failed: failed to create files: failed to create files: error creating /sysroot/etc/security/limits.conf: error creating file "/sysroot/etc/security/limits.conf": A file exists there already and overwrite is false

Expected behavior

With previous version 3510.2.8 this applied the custom configurations

Additional information

Please add any information here that does not fit the above format.

[jepio]

Thank you for the report. This will have been caused by switching to using an overlay for /etc, which provides distro defaults for config files. The overlay is created before ignition runs (right @pothos?) instead of config files being created via systemd-tmpfiles after ignition runs.

I think the current behavior is what we want to have though, with users needing to explicitly specify overwrite: true. I do want to apologize for breaking your pipeline. This is a tricky area for us, it is challenging to preserve compatibility with possible user modifications to config files and at the same time shipping updated default configs for users that dont have modifications.

[ninjasftw]

Thanks @jepio
My only concern is that overwrite:true doesn't allow for append mode?

[jepio]

Is that only the case for /etc/security/limits.conf? The issue could be that that file is a symlink by default, and this may actually be a bug now that we're using an overlay.

[ninjasftw]

I only use append mode for /etc/security/limits.conf so i'm not sure sorry
I had an issue with etc/flatcar/update.conf as well but that was just an inline which I could set overwrite: true on

[pothos]

In general there is no real change in behavior because files also existed on the initial rootfs before /etc became an overlay. For particular default files it changes over time whether they are symlinks to read-only locations or writable files in /etc. In the future we want to move away from symlinks towards writable files, but as of now /etc/security/limits.conf is a symlink that needs overwrite to be replaced by a plain file - append won't work.