Upgrade Ignition to version v2.x / spec v3.x, Afterburn to latest upstream

[smalltown]

Current situation
HashiCorp doesn't maintain Terraform provider for Ignitoin v0.x /spec 2.x for a long time, and RedHat W/ community maintain a new Terraform Ignition Provider, that supports Ignition version v2.x / spec v3.x, do you have any plan to upgrade the Ignition contain in the FlatCar?

Impact
In the feature, maybe Flatcar cannot leverage Terraform to provision, hence, user will have to choose other container linux

Ideal future situation
Flatcar user can use the latest HashiCorp Terraform Ignition to provision Flatcar

Implementation options

1. Upgrade Ignition to version v2.x / spec v3.x
2. Or Maintain the legacy Terraform Ignition provider

[amasover]

I would also recommend adding which Ignition version Flatcar supports to the docs, somewhere under the "Provisioning Tools" section. If you do supply the wrong config version, you'll get a helpful error on startup:

[    7.747394] ignition[450]: failed to acquire config: unsupported config version

However, this error message can be easy to miss, especially since many other boot processes will be failing at the same time, for example:

[    7.768758] systemd[1]: Failed to start Ignition (disks).
[    7.813462] systemd[1]: Dependency failed for Ignition (files).
[   16.571755] systemd-resolved[783]: Failed to listen on UDP socket 127.0.0.53:53: Cannot assign requested address
[   16.598174] systemd[1]: Failed to start Network Name Resolution.

[till]

To add some colour: we use the hashicorp provider for ignition and it works.

We build all kinds of systemd units, configure user accounts, set hostname, etc..

Not opposed to updating in general although it means fixing something that isn't broken. What exactly doesn't work or is the concern that Hashicorp doesn't actively maintain the provider?

[adamnovak]

In https://github.com/DataBiosphere/toil we're wanting to use Ignition JSON in AWS user data instead of the old could-config, and we're running into AWS's 16KB user data size limit. The Ignition format at spec version 3.1 got support for compression of file data or a whole config, which would help our use case.

[t-lo]

Ading @Cougar 's input from #326 here since we'll close #326 as a duplicate.

Package name and purpose
ignition got many enhancements and fixes multiple bugs since 0.34.0 (dated back to Dec 21, 2019). Latest release from CoreOS is v2.9.0

Package upstream:

• https://github.com/kinvolk/ignition/releases
• https://github.com/coreos/ignition/releases

Benefits of upgrading this package
One important fix was an ability to use systemd instantiated services, merged to v2.2.0

For example, instead of using

systemd:
  units:
    - name: [email protected]
      enabled: true
    - name: [email protected]
      enabled: true

such hack is needed right now:

storage:
  files:
  - path: /etc/systemd/system-preset/50-instantiated.preset
    filesystem: root
    contents:
      inline: |
        enable [email protected] eno1 ens1f1

Btw, the [email protected] is a custom unit because systemd-networkd-wait-online.service does not wait for all interfaces to become configured - not sure why (RequiredForOnline=yes exists in network configuration).

Additional information
When other base components like Linux kernel and Docker get frequent upgrades, ignition has been forgotten for a long time. As time passes the more complicated the upgrade could be. The same goes for systemd #301.

Or are there any other ideas for ignition roadmap by Kinvolk?

[t-lo]

Update: added Afterburn to the scope.

[pothos]

Ignition is done, Afterburn is next.

Rough list for the tasks needed:

• There are two changes that we need to port over: 212b2f458ef648b9dd064df2f1e85488ee46ed49 (for ignition.platform.id=→flatcar|coreos.oem.id as kernel parameters) and e2e89fcba45a390fe88550506656e67dee9da819 (upstream PR network: Encode information for systemd-networkd-wait-online coreos/afterburn#728)
• For the name of the binary we can set up a symlink (coreos-metadata→afterburn), for the name of the service we can set up an Alias= in coreos-metadata.service/afterburn.service
• Then there is the problem that the build flag for the legacy env var naming is gone. Since have logic to set up the metadata symlink already for the coreos vs flatcar location, maybe we cal also do the conversion in this step through a ExecStartPost invocation of sed instead of patching the source code (note, not only the prefix but also few names changed: AB_AWS→COREOS_EC2, GCP→GCE)
• Our usage from bootengine (the dracut files call the coreos-metadata binary) needs to be checked
• Find out what needs to be done with update-ssh-keys which got dropped: seems we could append the potential output from the authorized_keys.d/afterburn file manually to authorized_keys as action after calling the binary

[pothos]

Here a port of the afterburn kernel parameter patch
0001-util-cmdline-Handle-the-cmdline-flags-as-list-of-sup.patch.txt
It also reintroduces the cl-legacy build flag.

[tormath1]

Afterburn and Ignition are now upgraded and synced with upstream:

• Ignition spec3 is available from 3185.0.0
• Afterburn-5.2.0 should be available in the Alpha release

[pothos]

I took a look at the afterburn usage from bootengine and filed a PR to align the hostname setup and created a follow-up issue for additional functionality #717