testing: release 30.20190905.0

[dustymabe]

First, verify that you meet all the prerequisites

Pre-release

Promote testing-devel changes

From the checkout for fedora-coreos-config (replace upstream below with
whichever remote name tracks coreos/):

• git fetch upstream
• git checkout testing
• git reset --hard upstream/testing
• /path/to/fedora-coreos-releng-automation/scripts/promote-config.sh testing-devel
• Sanity check promotion with git show
• Open PR against the testing branch on https://github.com/coreos/fedora-coreos-config
• Ideally have at least one other person check it and approve before merging

Build

• Start a pipeline build (select testing, and fill in version number using the N.YYYYMMDD.P format, pending finalization of Version numbering for OS releases fedora-coreos-tracker#81)
• Wait for the job to finish

Sanity-check the build

Using the the build browser for the testing stream:

• Verify that the parent commit and version match the previous testing release (in the future, we'll want to integrate this check in the release job)
• Run kola on AMI to sanity check it (this will be run automatically on all builds in the future):

kola -p aws run --aws-ami <ami-id> --aws-region us-east-1 --parallel 10 -b fcos

Sign the CHECKSUMS file for releng

This is a stopgap until we do signing through fedora-messaging.

• Download the CHECKSUMS file locally:

aws s3 cp s3://fcos-builds/prod/streams/testing/builds/$VERSION/CHECKSUMS .

• Confirm that the SHA256 of the CHECKSUMS file you just downloaded matches the one from the pipeline Jenkins log output
• Sign it with your key: gpg2 --output CHECKSUMS.sig --detach-sign CHECKSUMS
• Push your signature to the bucket:

aws s3 cp --acl=public-read CHECKSUMS.sig s3://fcos-builds/prod/streams/testing/builds/$VERSION/CHECKSUMS.sig

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Signing artifacts and importing OSTree commit

In the future, the signing part will be integrated in the build job and the OSTree commit import will be integrated in the release job.

• Open an issue on https://pagure.io/releng similar to https://pagure.io/releng/issue/8578 to ask for the artifacts to be signed and OSTree commit to be imported
• Wait for releng to process the request
• Verify that the image artifact signatures are present and have the right ACL, e.g.:

aws s3 ls --recursive s3://fcos-builds/prod/streams/testing/builds/$VERSION/
curl -I https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/$VERSION/x86_64/fedora-coreos-$VERSION-qemu.qcow2.xz.sig

• Verify that the OSTree commit and its signature are present and valid by booting a VM at the previous release (e.g. cosa run -d /path/to/previous.qcow2) and verifying that rpm-ostree upgrade works and rpm-ostree status shows a valid signature.

Run the release job

• Run the release job, filling in for parameters testing and the new version ID
• Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet (instructions for starting a rollout TBD).

Update stream metadata

From a checkout of this repo:

• Run:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/testing/releases.json  -output-file=streams/testing.json -pretty-print

• Commit the changes and open a PR against the repo.
• Wait for the PR to be approved. Ideally, there's another pair of eyes available to have a final look, but otherwise, it's OK to self-approve. In the future, the release job will automatically create a PR, and a syncer will automatically push it to S3.
• Once approved, merge it and push it to S3:

aws s3 cp --acl=public-read streams/testing.json s3://fcos-builds/streams/testing.json --cache-control max-age=60

• Verify the new version shows up on the download page

[dustymabe]

kola tests pass:

[coreos-assembler]$ kola -p aws run --aws-region us-east-1 --parallel 10 -b fcos --aws-ami ami-0cdf885a13ed855fc
=== RUN   coreos.ignition.once
=== RUN   coreos.selinux.enforce
=== RUN   coreos.auth.verify
=== RUN   coreos.ignition.resource.s3
=== RUN   ostree.remote
=== RUN   rhcos.selinux.boolean.persist
=== RUN   ostree.unlock
=== RUN   fcos.basic
=== RUN   ostree.hotfix
=== RUN   coreos.ignition.groups
=== RUN   coreos.ignition.ssh.key
=== RUN   coreos.ignition.sethostname
=== RUN   rpmostree.status
=== RUN   rpmostree.upgrade-rollback
=== RUN   systemd.sysusers.gshadow
=== RUN   rpmostree.install-uninstall
=== RUN   coreos.ignition.resource.local
=== RUN   coreos.selinux.boolean
=== RUN   coreos.ignition.resource.remote
=== RUN   coreos.ignition.security.tls
=== RUN   fcos.python
=== RUN   ostree.unlock/unlock
=== RUN   ostree.unlock/install
=== RUN   ostree.unlock/uninstall
=== RUN   ostree.unlock/discard
--- PASS: coreos.ignition.sethostname (67.74s)
--- PASS: coreos.ignition.ssh.key (67.93s)
--- PASS: coreos.auth.verify (68.01s)
--- PASS: coreos.ignition.groups (69.64s)
--- PASS: coreos.ignition.resource.s3 (71.22s)
--- PASS: fcos.python (75.75s)
--- PASS: coreos.selinux.enforce (125.34s)
=== RUN   rpmostree.upgrade-rollback/upgrade
--- PASS: ostree.unlock (127.48s)
    --- PASS: ostree.unlock/unlock (0.93s)
    --- PASS: ostree.unlock/install (1.61s)
    --- PASS: ostree.unlock/uninstall (1.47s)
    --- PASS: ostree.unlock/discard (55.50s)
--- PASS: coreos.selinux.boolean (127.84s)
=== RUN   ostree.hotfix/unlock
--- PASS: rhcos.selinux.boolean.persist (132.48s)
=== RUN   rpmostree.install-uninstall/install
--- PASS: systemd.sysusers.gshadow (68.46s)
--- PASS: rpmostree.status (66.09s)
=== RUN   ostree.hotfix/install
=== RUN   ostree.hotfix/uninstall
=== RUN   ostree.hotfix/persist
=== RUN   ostree.remote/add
=== RUN   ostree.remote/list
=== RUN   ostree.remote/show-url
=== RUN   ostree.remote/refs
=== RUN   ostree.remote/summary
=== RUN   ostree.remote/delete
=== RUN   ostree.hotfix/rollback
--- PASS: ostree.remote (68.97s)
    --- PASS: ostree.remote/add (0.44s)
    --- PASS: ostree.remote/list (0.38s)
    --- PASS: ostree.remote/show-url (0.39s)
    --- PASS: ostree.remote/refs (1.23s)
    --- PASS: ostree.remote/summary (2.39s)
    --- PASS: ostree.remote/delete (1.17s)
=== RUN   rpmostree.upgrade-rollback/rollback
=== RUN   fcos.basic/PortSSH
=== RUN   fcos.basic/DbusPerms
=== RUN   fcos.basic/ServicesActive
=== RUN   fcos.basic/ReadOnly
=== RUN   fcos.basic/Useradd
=== RUN   fcos.basic/MachineID
--- PASS: coreos.ignition.resource.remote (67.81s)
--- PASS: fcos.basic (82.82s)
    --- PASS: fcos.basic/PortSSH (0.37s)
    --- PASS: fcos.basic/DbusPerms (0.53s)
    --- PASS: fcos.basic/ServicesActive (0.35s)
    --- PASS: fcos.basic/ReadOnly (0.36s)
    --- PASS: fcos.basic/Useradd (0.68s)
    --- PASS: fcos.basic/MachineID (0.38s)
--- PASS: coreos.ignition.resource.local (142.90s)
--- PASS: coreos.ignition.once (116.81s)
=== RUN   rpmostree.install-uninstall/uninstall
--- PASS: ostree.hotfix (182.88s)
    --- PASS: ostree.hotfix/unlock (10.05s)
    --- PASS: ostree.hotfix/install (1.41s)
    --- PASS: ostree.hotfix/uninstall (1.16s)
    --- PASS: ostree.hotfix/persist (47.75s)
    --- PASS: ostree.hotfix/rollback (59.52s)
--- PASS: rpmostree.upgrade-rollback (204.46s)
    --- PASS: rpmostree.upgrade-rollback/upgrade (70.38s)
    --- PASS: rpmostree.upgrade-rollback/rollback (68.56s)
--- PASS: coreos.ignition.security.tls (162.14s)
--- PASS: rpmostree.install-uninstall (265.81s)
    --- PASS: rpmostree.install-uninstall/install (119.51s)
    --- PASS: rpmostree.install-uninstall/uninstall (70.51s)
PASS, output in _kola_temp/aws-2019-09-05-1529-30

[dustymabe]

Verified parent commit:

[coreos-assembler]$ kola -p aws spawn --aws-region us-east-1 -b fcos --aws-ami ami-0cdf885a13ed855fc
[bound] -bash-5.0$ rpm-ostree status 
State: idle
AutomaticUpdates: disabled
Deployments:
* ostree://fedora:fedora/x86_64/coreos/testing
                   Version: 30.20190905.0 (2019-09-05T14:36:50Z)
                    Commit: b4beca154dab3696fd04f32ddab818102caa9247ec3192403adb9aaecc991bd9
              GPGSignature: (unsigned)
[bound] -bash-5.0$ ostree log fedora/x86_64/coreos/testing^
error: No such metadata object a9c8d66d3628d1b9b4c4690777e8b730d08329b4359410cb410a2003296af1ca.commit

a9c8d66d3628d1b9b4c4690777e8b730d08329b4359410cb410a2003296af1ca matches the commit of the 30.20190801.0 release

[dustymabe]

opened releng ticket for release: https://pagure.io/releng/issue/8756

[dustymabe]

* Commit the changes and open a PR against the repo.

PR open here: #13

[dustymabe]

ok everything is done except for:

Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet (instructions for starting a rollout TBD)

This is pending an update from @jlebon and @lucab that will be coming on Monday. What this means is that people doing fresh installs will get the new media from the download page, but existing nodes won't upgrade until we do something on Monday.

[dustymabe]

marking this as done since we finished everything up in #17