add backward compatibility #23

tormath1 · 2021-08-12T14:38:50Z

In this PR, we provide backward compatibility between v1, v2 and v3 of Ignition's config.

We use a forked ign-converter to ensure the conversion (https://github.com/flatcar-linux/ign-converter) - once the config loaded, ignition will check the version:

3.x: it behaves as usual
2.x: it tries to first parse with v2_4 parser (which fallbacks to 1.x if required) then it converts to 3.x

high level changes:

kept support for oem:// file URLs, special handling of the OEM partition, CoreOS/Flatcar Container Linux config locations, translation of v1 and v2 configs to v3

Testing done

run kola tests (some tests support the 3 versions, like the hostname one)
boot an instance with a custom Ignition (v1, v2 or v3) (use systemd.journald.max_level_console=debug console=ttyS0 to get Ignition logs if it bootloops)

Note for reviewers

once merged, we can promote main to be the default branch and backup flatcar-master to spec2x to follow the upstream logic

this commit ID pulls the following PR: * flatcar/ignition#23 it mainly brings V3 support on top of V2 support for Ignition and ensure backward compatibility with existing integration. Signed-off-by: Mathieu Tortuyaux <[email protected]>

pothos · 2021-09-10T12:26:06Z

(info.format == fs.Format || info.label == "OEM") would go here: https://github.com/kinvolk/ignition/blob/main/internal/exec/stages/disks/filesystems.go#L129 as port of 0a5340e and 1c73511

and these two need to be ported: 4b3c0f9 3e27c2e

Edit: the handling of ext4 OEM entries is now done in the translator instead of trying to mount all supported filesystems one by one

this commit ID pulls the following PR: * flatcar/ignition#23 it mainly brings V3 support on top of V2 support for Ignition and ensure backward compatibility with existing integration. Signed-off-by: Mathieu Tortuyaux <[email protected]>

pothos · 2022-02-25T14:36:38Z

The ign-converter uses the files under config/v1 and so on which are gone in this PR. We should either keep them around so that ign-converter can use this branch or we should "archive" the flatcar-master branch to have a place where we maintain the config/v1 files etc.

Edit: Plan is to keep the current flatcar-master state around as spec2 and take over the upstream branch names for our changes

pothos

I think we are good now with the kola tests passing?

Signed-off-by: Mathieu Tortuyaux <[email protected]>

if the version of ignition is 2.x we convert it to 3.1 using ign-converter. it shoud support any 2.x version (or at the least the last 2 releases) Signed-off-by: Mathieu Tortuyaux <[email protected]>

this patch provides backward compatiblity for various config key Signed-off-by: Mathieu Tortuyaux <[email protected]>

we support both CoreOS and Flatcar fw_cfg path to ensure compatiblity Signed-off-by: Mathieu Tortuyaux <[email protected]>

Signed-off-by: Mathieu Tortuyaux <[email protected]>

As soon as the OEM partition's filesystem format changes, all users that have file entries for the OEM partition (e.g., for kernel parameters), need to also switch to the new format, otherwise Ignition will bootloop due to the encountered mismatch error. If users have set the wipeFilesystem option there is no change in behavior and the user would create the desired filesystem on each Ignition run. To spare the users the migration due to an internal detail of the OEM partition, allow the OEM filesystem format to mismatch and just reuse the existing OEM partition. The user can still enable the wipeFilesystem to reformat regardless what content the partition has. Signed-off-by: Mathieu Tortuyaux <[email protected]>

The decision whether to use ignition.config.data or coreos.config.data was based on their presence in ovfenv. If this was missing, coreos.config.data would always be used. Also, if the retrieval of the guestinfo would have an error, the value from ovfenv would not be used even though it was supposed to be a fallback. Thus, refactor the logic to get variables from the ovfenv as fallback while preferring the direct guestinfo variables. With this new function, fix the logic of falling back to coreos.config.data but preferring ignition.config.data. The OVF metadata for CoreOS specified guestinfo.coreos.config.url but that was never used to fetch the Ignition config. Thus, use guestinfo.*.config.url as fallback if no guestinfo.*.config.data variables are set.

version 2 should be able to translate configuration version 1 but the `GetConfigVersion` was not able to detect version 1 configuration since for this particular version the `Version` is an int held into `ignitionVersion` field. Signed-off-by: Mathieu Tortuyaux <[email protected]>

we detect if the config is not an ignition one (script/cloudinit) and we return an ErrEmpty which will make Ignition to ignore this user config. Signed-off-by: Mathieu Tortuyaux <[email protected]>

This reverts commit 0c088d6.

When btrfs is used to fit more content into the partition, mounting fails because ext4 was hardcoded. When mounting ext4 fails, try mounting as btrfs.

The "udevadm settle" command used to wait for udev to process the disk changes and recreate the entries under /dev was still prone to races where udev didn't get notified yet of the final event to wait for. This caused the boot with a btrfs root filesystem created by Ignition to fail almost every time on certain hardware. Issue tagged events and wait for them to be processed by udev. This is actually meanigful in all stages not only for the other parts of the initramfs which may be surprised by sudden device nodes disappearing shortly like the case was with systemd's fsck service but also for the inter-stage dependencies which currently are using the waiter for systemd device units but that doesn't really prevent from races with udev device node recreation. Thus, these changes are complementary to the existing waiter which mainly has the purpose to wait for unmodified devices. For newly created RAIDs we can wait for the new node to be available as udev will not recreate it. Note: This is a port for Ignition 0.35 while for 2.2 this also should be done for LUKS.

tormath1 · 2022-03-07T11:05:22Z

@pothos ok then - I started a new CI this morning using flatcar/mantle#301. If it's good, I'll go ahead and merge this one.
ign-converter dependency has been pinned to v0.1.0 too.

tormath1 self-assigned this Aug 12, 2021

tormath1 force-pushed the tormath1/convert-v2-v3 branch from 1d6538e to 6fbb138 Compare August 12, 2021 15:58

tormath1 force-pushed the tormath1/convert-v2-v3 branch from 5f8164e to 5285546 Compare August 31, 2021 15:31

tormath1 mentioned this pull request Sep 2, 2021

sys-apps/ignition: bump commit ID flatcar-archive/coreos-overlay#1249

Merged

tormath1 force-pushed the tormath1/convert-v2-v3 branch 2 times, most recently from a1a1b20 to ce6fd58 Compare September 21, 2021 10:30

tormath1 mentioned this pull request Nov 5, 2021

Question: How to set a persistent iptables rule flatcar/Flatcar#540

Closed

tormath1 force-pushed the tormath1/convert-v2-v3 branch from 05073ce to 2d1ca66 Compare November 10, 2021 16:12

tormath1 changed the base branch from main to flatcar-master November 10, 2021 16:52

tormath1 changed the base branch from flatcar-master to main November 10, 2021 16:52

tormath1 force-pushed the main branch from 1067979 to 32908ec Compare January 7, 2022 09:03

tormath1 force-pushed the tormath1/convert-v2-v3 branch from 2d1ca66 to 390754d Compare January 7, 2022 15:26

tormath1 force-pushed the tormath1/convert-v2-v3 branch 2 times, most recently from 3a0410f to d8e5d3b Compare February 1, 2022 17:19

tormath1 force-pushed the tormath1/convert-v2-v3 branch 3 times, most recently from 4fe7d2c to 114ff89 Compare February 25, 2022 13:33

tormath1 force-pushed the tormath1/convert-v2-v3 branch from 8bc3d56 to 6de40b2 Compare March 1, 2022 14:15

tormath1 force-pushed the main branch from 32908ec to ce14e51 Compare March 2, 2022 15:21

tormath1 force-pushed the tormath1/convert-v2-v3 branch from 6de40b2 to dcd27a7 Compare March 2, 2022 15:27

pothos approved these changes Mar 7, 2022

View reviewed changes

tormath1 mentioned this pull request Mar 7, 2022

add some ignition/v3 test flatcar/mantle#301

Merged

1 task

Mathieu Tortuyaux and others added 14 commits March 7, 2022 12:02

mod: add ign-converter dependency

47ac5e7

Signed-off-by: Mathieu Tortuyaux <[email protected]>

sum: go mod tidy

899d82d

Signed-off-by: Mathieu Tortuyaux <[email protected]>

vendor: go mod vendor

16e5200

Signed-off-by: Mathieu Tortuyaux <[email protected]>

config/v3_4: convert ignition 2.x to 3.1

5c730b1

if the version of ignition is 2.x we convert it to 3.1 using ign-converter. it shoud support any 2.x version (or at the least the last 2 releases) Signed-off-by: Mathieu Tortuyaux <[email protected]>

internal/prv/cmdline: backport flatcar patche

6581b0c

this patch provides backward compatiblity for various config key Signed-off-by: Mathieu Tortuyaux <[email protected]>

provider/qemu: apply fw_cfg patch

3cc3144

we support both CoreOS and Flatcar fw_cfg path to ensure compatiblity Signed-off-by: Mathieu Tortuyaux <[email protected]>

config/3_4/test: add ignition 2.x test cases

172d2dd

Signed-off-by: Mathieu Tortuyaux <[email protected]>

config/util: add cloud init detection to initial parsing

3823328

we detect if the config is not an ignition one (script/cloudinit) and we return an ErrEmpty which will make Ignition to ignore this user config. Signed-off-by: Mathieu Tortuyaux <[email protected]>

Revert "*: drop OEM URI support"

b06bfb4

This reverts commit 0c088d6.

internal/resource/url: support btrfs as OEM partition filesystem

cdcfe06

When btrfs is used to fit more content into the partition, mounting fails because ext4 was hardcoded. When mounting ext4 fails, try mounting as btrfs.

tormath1 force-pushed the tormath1/convert-v2-v3 branch from dcd27a7 to 858ba5f Compare March 7, 2022 11:03

tormath1 changed the title ~~[wip] add backward compatibility~~ add backward compatibility Mar 7, 2022

tormath1 merged commit e2a7144 into main Mar 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add backward compatibility #23

add backward compatibility #23

tormath1 commented Aug 12, 2021 •

edited by pothos

Loading

pothos commented Sep 10, 2021 •

edited

Loading

pothos commented Feb 25, 2022 •

edited

Loading

pothos left a comment

tormath1 commented Mar 7, 2022

add backward compatibility #23

add backward compatibility #23

Conversation

tormath1 commented Aug 12, 2021 • edited by pothos Loading

Testing done

Note for reviewers

pothos commented Sep 10, 2021 • edited Loading

pothos commented Feb 25, 2022 • edited Loading

pothos left a comment

Choose a reason for hiding this comment

tormath1 commented Mar 7, 2022

tormath1 commented Aug 12, 2021 •

edited by pothos

Loading

pothos commented Sep 10, 2021 •

edited

Loading

pothos commented Feb 25, 2022 •

edited

Loading