Generate reports for OEM and base sysext images, allow reports against last nightly

[krnowak]

Needs flatcar/flatcar-build-scripts#149 first. Tested locally.
Needs flatcar/flatcar-build-scripts#153 now.

This does mostly two things:

• Image changes job prints the changes reports for each of the OEM sysext image and base sysexts too.
• Image changes job can print the changes reports against previous nightly.

[pothos]

Would be good to have a Jenkins and GA run for that

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/6970117426

[krnowak]

Would be good to have a Jenkins and GA run for that

For sure. I was waiting for the flatcar-build-scripts stuff to go in. Will kick off them soon.

[krnowak]

Jenkins: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/2585/cldsv

[krnowak]

Jenkins again:

• amd64 against last release: http://jenkins.infra.kinvolk.io:8080/job/container/job/image_changes/2985/
• arm64 against last nightly: http://jenkins.infra.kinvolk.io:8080/job/container/job/image_changes/2986/

GH is running.

[krnowak]

Seems to be working in general. OEM package reports are not working yet, because there are no such reports on bincache - this PR introduces building such reports. On github the OEM reports are useless, because we build only the qemu images, so azure, vmware, ami and others only have error messages about missing files.

Jenkins: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/2709/cldsv/
Github artifacts (search for image-changes-reports): https://github.com/flatcar/scripts/actions/runs/6497561930?pr=1169

[krnowak]

Just one note: On one hand it would be nice to do a hard error when encountering an error but on the other hand this means one wouldn't see the full output. It's ok to leave it as is but I fear it might be broken and people just skip over it because the script gives no final error. Examples are the use of source <(curl…) or local var=$(cmd) that silence errors.

I have split local var=$(cmd) into local var; var=$(cmd) and replaced the use of <(curl…) with curling to a temporary file and sourcing it. I kept <(printf … | sort) as I don't expect it to fail.

[krnowak]

Oh, yeah, backporting is tricky with the sysext stuff… Maybe some hacky if-else cases could prevent further divergence but I if I recall correctly this ship has sailed already for the image change job and it's already not easy to apply patches cleanly on the Stable/LTS branches, or?

I decided to not do the backporting as the advantages of doing it are rather slim - once a version hits a LTS/stable/beta/alpha channel, the image changes job is of limited use - almost nothing changes there comparing to the previous nightly in the channel. The important changes (like kernel version and kernel config changes and the like) are still printed.

Instead I went with a cut-off version of 3795 in jenkins os to decide whether to kick off two image-changes jobs or just one.

[krnowak]

CI: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/2964/cldsv/

[krnowak]

CI has passed. So for versions <3795, we will do the old stuff (image changes against last release, done after the image job, no changes in base sysexts and OEM sysexts), while for newer versions the new stuff (image changes against last release and last nightly, done after vms job, display changes in base sysexts and OEM sysexts).