workflows: Add new mkosi e2e flow #2019
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
stevenhorsman
commented
Aug 28, 2024
•
stevenhorsman
commented
- Combine the s390x and amd64 mkosi workflows together and enhance to create a qcow2 output that is cached with oras, push intermediate steps to GHCR and then pull these cached versions down if available to speed up builds
- Re-work the libvirt e2e flow to add support for mkosi podvm image testing and decouple the architect tests
- Enable EFI optional support for libvirt for x86 mkosi
- Enable cloud-init on the mkosi image for libvirt
stevenhorsman
force-pushed
the
mkosi-workflow-multi-arch
branch
2 times, most recently
from
1765e5c to
71c9664
Compare
|
FYI, I've moved on a lot from this approach to try and build on top of #1965. I'm missed about the best strategy and whether it's worth the complexity to keep separate workflows for the podvm's builder, binaries and image stage, so I've posted on slack to see if others have thoughts. |
stevenhorsman
force-pushed
the
mkosi-workflow-multi-arch
branch
from
0df4c2f to
98275a3
Compare
|
Ok, I've re-worked this to have a single mkosi workflow and not interlink with the separate stages. It's still WIP though and in test! |
stevenhorsman
force-pushed
the
mkosi-workflow-multi-arch
branch
9 times, most recently
from
d9688b1 to
e8d62a2
Compare
stevenhorsman
force-pushed
the
mkosi-workflow-multi-arch
branch
15 times, most recently
from
8399619 to
6bbe745
Compare
stevenhorsman
force-pushed
the
mkosi-workflow-multi-arch
branch
2 times, most recently
from
592c61d to
caefe69
Compare
- Remove seperate s390x workflow - Pin runner version - Add inputs required to plug this into the e2e workflows - Add check for if builder and binaries image already exist - Add s390x mkosi install - bump github actions to latest versions Signed-off-by: stevenhorsman <[email protected]>
- Update image names to match the existing naming scheme - Add debug suffix to the container image name for clarity - Add push/load switch based on the PUSH env Signed-off-by: stevenhorsman <[email protected]>
- Rename the docker provider's podvm Dockerfile to Dockerfile.podvm_docker_provider for more clarity and to enable us to use Dockerfile.podvm for the "main" version of the podvm in future - Add arch awareness, so we can build images for multiple architectures. Signed-off-by: stevenhorsman <[email protected]>
- Add calls to build the podvm-mkosi image in the same places we build the current podvm image - Note: we have decoupled the s390x and amd64 podvm_mkosi image jobs, so that their total e2e workflows can run independently and delays in one step doesn't impact the other. Signed-off-by: stevenhorsman <[email protected]>
- Add input option to select between the debug and non-debug mkosi image build - Initially use the debug build for e2e tests and non-debug for release and image publish Signed-off-by: stevenhorsman <[email protected]>
- Re-use the resources/binaries-tree binaries already fetched in the podvm_mkosi.yaml to build the podvm image for the docker provider. - We also need to clean up the image build artifacts to stop the runner going out of space Signed-off-by: Wainer dos Santos Moschetta <[email protected]> Signed-off-by: stevenhorsman <[email protected]>
Add fedora-like OS support for cross-build-extras Signed-off-by: stevenhorsman <[email protected]>
- Convert the mkosi image to a qcow2 file - Remove the Dockerfile.podvm.fedora file as we want to upload it via oras rather than our own wrapping container image - Run nix develop sudo to avoid permissions error Signed-off-by: stevenhorsman <[email protected]>
- It used to be that the fedora se image was the only image that was built on s390x, but now with the other mkosi builds, it feels better to make the s390x ubuntu podvm image which is arguably set-up incorrectly the special case. When we drop packer support we can remove this code. Signed-off-by: stevenhorsman <[email protected]>
Add runner as an input parameter to e2e_libvirt tests, so we can control it from calling workflow Signed-off-by: stevenhorsman <[email protected]>
Publish the podvm qcow2 files with oras Signed-off-by: stevenhorsman <[email protected]>
In the workflow output the qcow2 oras image location, so that we can use this in the e2e testing step Signed-off-by: stevenhorsman <[email protected]>
In the workflow output the docker provider oci image location, so that we can use this in the e2e testing step Signed-off-by: stevenhorsman <[email protected]>
In the mkosi flow we publish the podvm qcow2 image with oras, so add support to pull this down in e2e testing Signed-off-by: stevenhorsman <[email protected]>
- Added two new decoupled jobs that depends on the amd64 and s390x mkosi podvm builds to test each of those images with libvirt - Added the option of arch specific label triggers in case of specific testing restrictions Signed-off-by: stevenhorsman <[email protected]>
We want to run clean up even if previous steps fail, so try adding `always` to the condition Signed-off-by: stevenhorsman <[email protected]>
Create separate s390x and amd64 CAA builds in the e2e test, so we can reduce the coupling of different architecture's jobs. Also run the s390x dev build natively on the s390x runner, so it doesn't run emulated and take a long time. Signed-off-by: stevenhorsman <[email protected]> fixup multi-arch caa
Given that we want to decouple the libvirt testing and run each architecture independently, having a cached kustomization.yaml doesn't seem to be very helpful, so remove that dependency and instead update the kustomization.yaml in the libvirt e2e step. It might even be worth considering moving this into the e2e framework provisioning itself via an ENVVAR Signed-off-by: stevenhorsman <[email protected]>
Re-work the secure_comms matrix input, so it works for the de-coupled per-arch libvirt test jobs. Signed-off-by: stevenhorsman <[email protected]>
- The packer build doesn't want a custom firmware, so we can't easily keep the default to be ovmf. Instead reverse the logic and set it as blank unless provided. - In order to preserve the existing behaviour set the kustomization.yaml with the old default and update the packer e2e testing to comment it out - Also rename to LIBVIRT_EFI_FIRMWARE to clarify that it's explicitly used for EFI firmware - If the EFI firmware path is set, then set domain firmware to efi and set the device disk to sata & sdb as mkosi needs sata and packer needs IDE. In future once we drop packer we can potentially simplify this. Signed-off-by: stevenhorsman <[email protected]>
The current encrypted image seems to only be amd64 and the kata CI's multi-arch image has a decryption key that is 44 bytes and our code rejects it as it only works with 32 byte keys Signed-off-by: stevenhorsman <[email protected]>
stevenhorsman
force-pushed
the
mkosi-workflow-multi-arch
branch
from
caefe69 to
d9007e8
Compare
|
@wainersm & @mkulke - I think I've address all the comments left and re-tested and the e2e flow: https://github.com/stevenhorsman/cloud-api-adaptor/actions/runs/12137733841 and podvm mkosi builds https://github.com/stevenhorsman/cloud-api-adaptor/actions/runs/12138531982, https://github.com/stevenhorsman/cloud-api-adaptor/actions/runs/12138528028 are still passing, so please feel free to check my responses are satisfactory |
stevenhorsman
changed the title
Just found a quote for this occasion which might be perfectly mistakenly attributed to Oscar Wilde: "Everything is dangerous. If it wasn't so, life wouldn't be worth living". In other words, let's merge it! |
bpradipt
approved these changes
Dec 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.