Release notes, checklist + ansible fixes that are in Galaxy

docs/release/checklist.org

@@ -0,0 +1,141 @@
+#+TITLE: Release checklist
+
+* Clone the release branch
+
+#+begin_src shell
+  git checkout upstream/b0.72
+  git branch -u upsream/b0.72
+#+end_src
+so an update and rebase will get you to the latest commit on the upsream branch.
+
+* Test
+** (Step 1) Build an RPM in the COPR -test repo
+
+Make sure that =agent/VERSION= in the branch reflects the release version (e.g. 0.72.0).
+That will allow the =make= invocation below to create properly named RPMs (e.g. =pbench-agent-pbench-agent-0.72.0-1gXXXXXXX)
+in the appropriate repo (=pbench-0.72-test= in this case)
+#+begin_src shell
+  cd agent/rpm
+  make copr-test
+#+end_src
+
+** (Step 2) Create VMs of various stripes for testing and prep them appropriately
+(e.g. ssh keys need to be installed for ansible to work and repos need to be installed (e.g CRB repos on RHEL-8 and RHEL-9 and EPEL repos).
+
+
+** (Step 3) Run the playbooks to do the prep, repo install, pbench-agent/pbench-systat install, the config of the agent and the firewall rules
+TBD
+
+** (Step 4) Run the smoke tests
+TBD
+
+** (Step 5) Run end-to-end tests
+TBD linpack, uperf, fio: register tools, run the benchmark, clean up the
+tools, push the results.
+
+* Lather, rinse, repeat
+Repeat steps 3-5 in the [[* Test][Test]] section above until there are no
+errors. In general, you /might/ have to rebuild the RPMs in Step 1 if
+any changes to pbench code were made, but you probably do not need to
+remake the VMs in Step 2, although you might need to either uninstall the
+previous RPM or make sure that the build number of the new RPM is
+bumped, so that `dnf` will know to update the installed RPM.
+
+* Commit any changes to the release branch
+
+If there are changes to pbench code, issue PRs for them and make
+sure that they are merged. You might want to do another run through
+the [[* Test][Test]] process before proceeding further.
+
+It may also be necessary to update the ansible roles on Galaxy:
+#+begin_src shell
+  cd agent/ansible
+  make test
+  make publish
+#+end_src
+
+Galaxy will refuse to update unless the version in
+=agent/ansible/collection/galaxy.yml= is bumped (the current version
+as of this writing is 1.0.10).
+
+* Write up release notes - submit for review
+Revise until everybody is OK with the release notest.
+
+The release notes should contain:
+- a short description of the release
+
+- installation notes identifying any user-visible changes to the installation process and the ansible roles
+
+- new commands and major changes to existing commands
+
+- a list of user-visible bug fixes and enhancements to individual programs
+
+- new deprecation notices and retirement notices of previously deprecated items
+
+- an optional section on future directions
+
+- a changelog of all the relevant commits since the last release
+
+The changelog can be obtained using something like =git v0.71.0..HEAD
+--oneline --no-decorate=, but it will need to be manually pruned to
+include only commits for the relevant component(s).
+
+Nobody will be happy with the release notes but do not sweat it too
+much at this point: the release notes will be committed to the branch
+after the release is cut and they can be revised again and again, even
+after the release.
+
+* Make sure there are no pending updates to the release branch.
+The only other change that should go in before the release is cut should be
+the version bump in e.g. =agent/VERSION=. Do this in a separate commit.
+
+* Add a tag and cut the release on GitHub, adding the release notes.
+From the top level of the repo on Github, click on "Releases" on the RHS.
+
+Click on "Draft a new release" and add a new tag.The tag should have
+the form =v0.72.0=. Make sure that it points to the top of the release
+branch, presumably to the commit that bumps the version in the VERSION
+file.
+
+Modify the release notes to include the correct SHA1 from the top
+of the release branch. Do *not* commit the release notes yet.
+Instead cut-and-paste the contents to the text box of the new release
+and add a title (try to be consistent with earlier releases).
+
+Make sure that the "Set as the latest release" box is checked and then
+click on "Publish Release".
+
+* Retest
+Redo the [[* Test][Test]] section above in the updated branch, creating the RPMs
+in the final COPR repo.
+
+** Rebuild RPMs in the COPR repo for the release
+#+begin_src shell
+  cd agent/rpm
+  make copr
+#+end_src
+
+** Redo steps 3-4
+Install the RPMs on the set of VMs and run the smoke tests. If there
+are problems, you will have to repeat the whole process up to this
+point.
+
+** Build tagged containers
+
+#+begin_src shell
+  cd agent/containers/images
+  make everything
+  make push
+  make push-latest
+#+end_src
+
+** Send announcement out
+
+See the boilerplate =docs/release/sample-announcement.txt= in this
+directory for an example.
+
+** Submit post-release PR to the release branch
+The commit should contain the updated release notes. If problems are
+found later on, the release notes can be updated with more commits and
+copied the same way as before into the text box of the release area on
+Github.

docs/release/sample-announcement.txt

@@ -0,0 +1,168 @@
+* Sample announcement for reference
+
+
+Subject: Release of version 0.71 of the pbench-agent is now available
+To: <recipients>
+
+
+We are pleased to announce the release of version 0.71 of
+pbench-agent. There are some gotchas and known problems that are
+described first: please read them carefully because you are almost
+certainly going to run into one or more of them.
+
+<GOTCHAS>
+- The notion of a "default" tool set is being deprecated and will be
+removed in the upcoming Pbench Agent v1.0 release. To replace it, the
+Pbench Agent is introducing a few named tool sets. See "Default Tool
+Set is Deprecated; Named tool sets introduced" in the Release Notes.
+
+- All tools registered prior to installing v0.71.0 must be
+re-registered; tools registered locally, or remotely, on a host with
+v0.69 or earlier version of the pbench-agent will be ignored. See
+"Tool registration kept local to the host where registration happens"
+in the Release Notes.
+
+- Note that pbench no longer installs tools: any missing tools will
+only be found and reported when you start a run.
+
+<GOTCHAS/>
+
+<KNOWN PROBLEMS>
+In addition, there are some known problems with specific distros:
+
+- The (currently unreleased) RHEL 9.1 version has very spotty coverage
+  in its Subscription Manager repos. Installation of pbench-agent on
+  it fails with missing basic packages (e.g. bzip2). There are
+  probably unofficial repos that could be used instead: this is under
+  investigation.
+
+- Installation on Fedora 36 succeeds but you will not be able to do
+  anything with it: Python 3.10 changed the way that pip install modules
+  (it inserts a "local" element in the path), so the PYTHONPATH settings
+  are wrong and modules cannot be found. This is a larger problem and we
+  are working on that, but the upshot is that you won't be able to use
+  Fedora 36 with this release.
+
+- You should upgrade the pbench Ansible roles with
+
+    ansible-galaxy collection install --force pbench.agent
+
+  If your version of ansible-galaxy does not support --force, delete
+  the existing roles and install:
+
+    rm -rf $HOME/.ansible/collections/ansible_collections/pbench/agent
+    ansible-galaxy collection install pbench.agent
+
+  The current version is 1.0.7. It contains a fix for the CentOS-Stream-8
+  distro: without that fix, you are not going to be able to use the playbooks
+  to install on that distro.
+
+- If you are upgrading from v0.69, there is one additional problem:
+  the v0.69 installation probably installed the `pbench-perl-JSON-XS'
+  RPM, our own build of perl-JSON-XS which was missing from all RHEL
+  plaforms.  It (and many other perl packages) is now available from
+  standard repos (EPEL on RHEL7, CRB (Codeready Builder) repo on
+  RHEL8/9, PowerTools repo on Centos-stream-8/9). However, the
+  installed `pbench-perl-JSON-XS' on v0.69 conflicts with the official
+  `perl-JSON-XS', so the RPM installation fails. The workaround is to
+  erase the `pbench-perl-JSON-XS' RPM before updating to v0.71.0:
+
+      $ yum shell
+      > erase pbench-perl-JSON-XS
+      > update pbench-agent
+
+  That's the best way to do it on a single system, but if you have a
+  lot of systems that you are installing to, it's probably easier to
+  use your inventory file to do the erasing with an ad-hoc Ansible
+  command and then install normally:
+
+      ansible -i /path/to/Inventory/file servers -m command -a 'yum erase -y pbench-perl-JSON-XS'
+      ansible-playbook -i /path/to/Inventory/file pbench-agent-install.yml
+
+- Installing on RHEL 8.4 and 8.5 fails because of the ansible-core
+  dependency.  There are more problems with the ansible package on
+  RHEL78.4 and 8.5 however: see
+
+      https://access.redhat.com/discussions/6962395
+
+  If you cannot upgrade to RHEL 8.6, let us know: we might be able to
+  help.
+
+</KNOWN PROBLEMS>
+
+
+<ANNOUNCEMENT>
+The RPMs are available in the main COPR pbench repo:
+
+    https://copr.fedorainfracloud.org/coprs/ndokos/pbench/
+
+We strongly recommend that you use Ansible playbooks to install pbench-agent.
+See below for the link to the Installation Guide.
+
+You are going to need an inventory file. Here's a template: just list
+the hosts where you want to install pbench-agent in the [servers]
+section; nothing else needs to be touched:
+
+[servers]
+hostA
+hostB
+...
+
+[servers:vars]
+pbench_repo_name = pbench
+fedoraproject_username = ndokos
+
+pbench_key_url =  http://git.app.eng.bos.redhat.com/git/perf-dept.git/plain/bench/pbench/agent/{{ pbench_configuration_environment }}/ssh
+pbench_config_url = http://git.app.eng.bos.redhat.com/git/perf-dept.git/plain/bench/pbench/agent/{{ pbench_configuration_environment }}/config
+
+After installation, the RPM version reported by `dnf list installed
+pbench-agent' should be:
+
+    pbench-agent.noarch         0.71.0-3g85910732a          @copr-pbench-ndokos
+
+If you had previously installed a test version with a "greater"
+version number, you might need to erase it first with `dnf erase
+pbench-agent' before this version can be installed: we were a bit
+cavalier with the numbering and `dnf' will not install this version if
+the version number doesn't allow it.
+
+Containers
+----------
+
+Containers have been built based on the RPMs above and are available from
+
+    https://quay.io
+
+
+Release Notes
+-------------
+
+The release notes for this release are available at
+
+    https://github.com/distributed-system-analysis/pbench/releases/
+
+There are a lot of changes in this release, so we recommend that you
+read the release notes carefully.
+
+
+Installation Guide
+------------------
+
+The pbench-agent installation guide is available internally at
+
+    http://pbench.perf.lab.eng.bos.redhat.com
+
+
+Problems
+--------
+
+Let us know of any problems you encounter. The best place is the
+issue tracker:
+
+    https://github.com/distributed-system-analysis/pbench/issues
+
+although for urgent problems the GChat pbench room can be used.
+<ANNOUNCEMENT/>
+
+--
+The pbench team