Plan to deprecate packer based podvm image builds and switch to mkosi

[bpradipt]

In preparation for deprecating packer based podvm image builds and switching to mkosi based image builds, I'm proposing a restructuring of the current directory layout.

• Consolidate packer based build deps (eg, hcl files, configs, docker files etc) under podvm-packer sub-dir
• Consolidate mkosi based build deps (eg, configs, dockerfiles etc) under podvm-mkosi sub-dir
• Update the github workflows to reflect the new directory structure for packer based builds in CI
• Add/update github workflows to reflect the new directory structure for mkosi based builds in CI

Once this is done, we should decide on a timeline to switch to podvm-mkosi based builds in CI.

@mkulke @stevenhorsman @snir911 @katexochen @surajssd @wainersm and others are you ok with this approach?
Is there any other alternative approach you would like to recommend ?

[stevenhorsman]

We discussed this in the community call and we were happy with the plan, with a couple of specifics/additions

• Due to issues with build ibm cloud images with mkosi, the switch to podvm-mkosi might need to only be for the other cloud providers initially, so we'd keep packer around as deprecated for the IBM Cloud providers. DaLi has created a PR to pin the version, so that should avoid license issues in the short-term
• During the time period that we have CI for both packer based and mkosi based podvms, we might need to re-prioritise the kata-agent caching in order to reduce the overhead of running both builds

[stevenhorsman]

Wow - a year on from here and we've recently made some progress adding mkosi image builds into our CI. We still need to resolve the ability to add GPU support into mkosi before we can proceed much further with the deprecation to avoid breaking some use cases.

[stevenhorsman]

As a note it would be beneficial to add some CI testing for the use cases we want to protect so we can ensure they work on mkosi before the switch.