April 2019 Release Retrospective

[karianna]

Dump your comments and thoughts here.

[tellison]

I'll get the ball rolling, should put some of these into 'did well' and others into 'can improve':

• Any build script / job config gotchas we should improve for next time?
• Should we be privately building and testing vuln patches ahead of the releases?
• Did we agree/implement the tiered platform release well?
• Was the machine / code quiet time ahead of the release the right length?
• Were the steps for agreeing release criteria and steps to publish the release well understood across the teams?
• Where were the bottlenecks in achieving a smooth release: people, machines, process?

[karianna]

• We need a set of release instructions that can survive bus factor
• We need folks that can cross over timezones
• We lacked Mac OS X and Windows test hosts (already resolved for next release)
• OpenJ9's RC policy means an extra spin of all of their binaries at best. Are they being too risk-averse?

[smlambert]

This is what I have observed so far:

Can improve:

• Update top-level build pipelines show child job states accurately (in Jenkins view)
• more test machines for s390x/windows/osx (windows/osx on the way)
• auto-generating test jobs for next release will address issues of Jenkins config that delayed the manual restart of test job when required
• large number of PRs accumulating in openjdk-tests repo during lockdown
• rather than complete lock down of tests repo for 3 days prior, should cherrypick important PRs (Archive openjdk regression tests output including diagnostic files adoptium/aqa-tests#1073 and Update JRE_IMAGE location adoptium/aqa-tests#1077) to merge (as they would have been helpful for release)
• need a way to add extended release testing to build pipeline (but not to nightlies) - right now we launch them manually, though in previous incarnation of build pipelines there was a way to differentiate nightlies from release builds
• we have some AQA mechanisms in place now (gathering SHAs of test repos, listing of disabled test targets, aggregate test results info beta), but still need to produce a file that can be added to the releases repo along with the build metadata and to present it through TRSS view
• finish triaging/cleaning the extended.external targets (as they still show as unstable, 1 of thousands of subtests failing, we should pursue how to address or exclude via outreach to application communities
• Openjdk 11+ build need also build test-image adoptium/temurin-build#248 addressed so we can include openjdk native tests in jdk11+
• finish triaging extended.openjdk test targets (at least to remove any infra/test issues from the set)

Went well:

• pausing nightly builds during release time
• stand-up slack call to directly check in with rest of team

[philippecade]

From the end user point of view what went well:

• adding the banner announcing the releases with estimated delivery time on https://adoptopenjdk.net right on the day of the critical patch update announcement

[lumpfish]

Can improve:

• Main throughput bottleneck for jdk8ub08_openj9-0.14.0 was Windows /osx machines
• AIX release is build only - if testing were added with no additional resources AIX would become the limiting factor
• systemtest seems to spend a large part of the run time running the jlm tests. Can they be tuned to run faster - maybe reduce the monitor process time?
• openj9 selfish view: on the (or a slightly different) build pipeline page it would be good to be able to select the repository forks and branch to use for the build for the eclipse/openj9, eclipse/openj9-omr and opej9-openjdk-jdkx repositories. This would enable much more early testing of open9 or openjdk code before merging it into the main branches.
• Descriptive text against the Jenkins parameters needs improving.
• Web site did not see the new release on github

Went well:

• openj9 code hardened in advance of final code drop from openjdk
• New pipeline submission page (Release / Nightly, Publish name, Scm reference etc. options) does just what we need
• Help from Slack channels regarding build to test result questions
• Triaging / exclude work has provided a 'tests should pass' expectation'

[smlambert]

Encouraging to have read through the list above and have seen many of the items have now been addressed. Since we are now working through the Release retrospective for April 2020, I will close this item.