stable: new release on 2020-04-21 (31.20200407.3.0)

[dustymabe]

First, verify that you meet all the prerequisites

Name this issue stable: new release on YYYY-MM-DD with today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.3.0).

Pre-release

Promote testing changes to stable

From the checkout for fedora-coreos-config (replace upstream below with
whichever remote name tracks coreos/):

• git fetch upstream
• git checkout stable
• git reset --hard upstream/stable
• /path/to/fedora-coreos-releng-automation/scripts/promote-config.sh testing
• Sanity check promotion with git show
• Open PR against the stable branch on https://github.com/coreos/fedora-coreos-config
• Post a link to the PR as a comment to this issue
• Ideally have at least one other person check it and approve
• Once CI has passed, merge it

Build

• Start a pipeline build (select stable, leave all other defaults)
• Post a link to the job as a comment to this issue
• Wait for the job to finish

Sanity-check the build

Using the the build browser for the stable stream:

• Verify that the parent commit and version match the previous stable release (in the future, we'll want to integrate this check in the release job)
• Check kola AWS run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

• Run the release job, filling in for parameters stable and the new version ID
• Post a link to the job as a comment to this issue
• Wait for job to finish
• Verify that the OSTree commit and its signature are present and valid by booting a VM at the previous release (e.g. cosa run -d /path/to/previous.qcow2) and verifying that rpm-ostree upgrade works and rpm-ostree status shows a valid signature.

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

From a checkout of this repo:

• Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/stable/releases.json  -output-file=streams/stable.json -pretty-print

• Update the updates metadata, editing updates/stable.json:
  • Find the last-known-good release (whose rollout has a start_percentage of 1.0) and set its version to the most recent completed rollout
  • Delete releases with completed rollouts
  • Add a new rollout:
    • Set version field to the new version
    • Set start_epoch field to a future timestamp for the rollout start (e.g. date -d '2019/09/10 14:30UTC' +%s)
    • Set start_percentage field to 0.0
    • Set duration_minutes field to a reasonable rollout window (e.g. 2880 for 48h)
  • Update the last-modified field to current time (e.g. date -u +%Y-%m-%dT%H:%M:%SZ)

A reviewer can validate the start_epoch time by running date -u -d @<EPOCH>. An example of encoding and decoding in one step: date -d '2019/09/10 14:30UTC' +%s | xargs -I{} date -u -d @{}.

• Commit the changes and open a PR against the repo.
• Post a link to the PR as a comment to this issue
• Wait for the PR to be approved.
• Once approved, merge it and push the content to S3:

aws s3 sync --acl public-read --cache-control 'max-age=60' --exclude '*' --include 'streams/*' --include 'updates/*' . s3://fcos-builds

• Verify the new version shows up on the download page
• Verify the incoming edges are showing up in the update graph:

curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=stable&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated and a syncer will push the updated metadata to S3.

[sinnykumari]

Promotion PR - coreos/fedora-coreos-config#359

[sinnykumari]

build job is running at https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/12356/

[dustymabe]

I had to kill that job because we needed to get a fix in first: coreos/fedora-coreos-releng-automation#102

[dustymabe]

new pipeline run: https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/12359/

[dustymabe]

ok the pipeline failed because this was the first ever import into the compose repo for stable:

Exception: Importing failed: Refusing to import non-origin commit into a new branch

I've done the releng thing and manually made that commit so that we should never see that error again. The only stage that runs after ostree import is Kola AWS tests. I manually started those here

[sinnykumari]

Release job - https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/465/

[sinnykumari]

release job failed due to issue coreos/fedora-coreos-releng-automation#103

[sinnykumari]

release job with fix - https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/468/

[sinnykumari]

ostree commit looks good:

$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200407.3.0 (2020-04-21T19:37:39Z)
                    Commit: 89e17cc21b6aa3bea8959d1e6957fda157168d57ba6805d8a36142184edc2901
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200323.3.2 (2020-04-08T14:30:14Z)
                    Commit: 9b322b9cead87ebf80881b75560e481b6e5a26966814b01bf0c12b61fa363708
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

[sinnykumari]

PR for streams and updates metadata - #89

[sinnykumari]

all done, rollout should begin at Wed 22 Apr 2020 03:30:00 PM UTC

[sinnykumari]

One of the test node I had created has successfully rolled out to version 31.20200407.3.0