From 159c9a095b52f2c36ba2ca0cec6091104cd153a4 Mon Sep 17 00:00:00 2001 From: andyzhangx Date: Sat, 16 Dec 2023 02:56:41 +0000 Subject: [PATCH] Squashed 'release-tools/' changes from 670bb0ef..b54c1ba4 b54c1ba4 Merge pull request #246 from xing-yang/go_1.21 5436c81e Change go version to 1.21.5 267b40e9 Merge pull request #244 from carlory/sig-storage b42e5a2d nominate self (carlory) as kubernetes-csi reviewer a17f536f Merge pull request #210 from sunnylovestiramisu/sidecar 011033de Use set -x instead of die 5deaf667 Add wrapper script for sidecar release f8c8cc4c Merge pull request #237 from msau42/prow b36b5bfd Merge pull request #240 from dannawang0221/upgrade-go-version adfddcc9 Merge pull request #243 from pohly/git-subtree-pull-fix c4650889 pull-test.sh: avoid "git subtree pull" error 7b175a1e Update csi-test version to v5.2.0 987c90cc Update go version to 1.21 to match k/k 2c625d41 Add script to generate patch release notes f9d5b9c0 Merge pull request #236 from mowangdk/feature/bump_csi-driver-host-path_version b01fd537 Bump csi-driver-host-path version up to v1.12.0 984feece Merge pull request #234 from siddhikhapare/csi-tools 1f7e6059 fixed broken links of testgrid dashboard de2fba88 Merge pull request #233 from andyzhangx/andyzhangx-patch-1 cee895e1 remove windows 20H2 build since it's EOL long time ago git-subtree-dir: release-tools git-subtree-split: b54c1ba49469d4d5d1b5d75285e8868ffe3d328f --- KUBERNETES_CSI_OWNERS_ALIASES | 1 + SIDECAR_RELEASE_PROCESS.md | 90 +++++++++++----------- generate_patch_release_notes.sh | 106 ++++++++++++++++++++++++++ go-modules-update.sh | 129 ++++++++++++++++++++++++++++++++ prow.sh | 8 +- pull-test.sh | 1 + 6 files changed, 283 insertions(+), 52 deletions(-) create mode 100755 generate_patch_release_notes.sh create mode 100755 go-modules-update.sh diff --git a/KUBERNETES_CSI_OWNERS_ALIASES b/KUBERNETES_CSI_OWNERS_ALIASES index 04a07bfb..6e46cf29 100644 --- a/KUBERNETES_CSI_OWNERS_ALIASES +++ b/KUBERNETES_CSI_OWNERS_ALIASES @@ -18,6 +18,7 @@ aliases: # when they are temporarily unable to review PRs. kubernetes-csi-reviewers: - andyzhangx + - carlory - chrishenzie - ggriffiths - gnufied diff --git a/SIDECAR_RELEASE_PROCESS.md b/SIDECAR_RELEASE_PROCESS.md index 0e3d516a..4f5d033c 100644 --- a/SIDECAR_RELEASE_PROCESS.md +++ b/SIDECAR_RELEASE_PROCESS.md @@ -17,7 +17,7 @@ The release manager must: Whenever a new Kubernetes minor version is released, our kubernetes-csi CI jobs must be updated. -[Our CI jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) have the +[Our CI jobs](https://testgrid.k8s.io/sig-storage-csi-ci) have the naming convention `-on-`. 1. Jobs should be actively monitored to find and fix failures in sidecars and @@ -46,54 +46,48 @@ naming convention `-on-`. ## Release Process 1. Identify all issues and ongoing PRs that should go into the release, and drive them to resolution. -1. Download the latest version of the - [K8s release notes generator](https://github.com/kubernetes/release/tree/HEAD/cmd/release-notes) -1. Create a - [Github personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) - with `repo:public_repo` access -1. Generate release notes for the release. Replace arguments with the relevant - information. - * Clean up old cached information (also needed if you are generating release - notes for multiple repos) - ```bash - rm -rf /tmp/k8s-repo - ``` - * For new minor releases on master: - ```bash - GITHUB_TOKEN= release-notes \ - --discover=mergebase-to-latest \ - --org=kubernetes-csi \ - --repo=external-provisioner \ - --required-author="" \ - --markdown-links \ - --output out.md - ``` - * For new patch releases on a release branch: - ```bash - GITHUB_TOKEN= release-notes \ - --discover=patch-to-latest \ - --branch=release-1.1 \ - --org=kubernetes-csi \ - --repo=external-provisioner \ - --required-author="" \ - --markdown-links \ - --output out.md - ``` -1. Compare the generated output to the new commits for the release to check if - any notable change missed a release note. -1. Reword release notes as needed. Make sure to check notes for breaking - changes and deprecations. -1. If release is a new major/minor version, create a new `CHANGELOG-..md` - file. Otherwise, add the release notes to the top of the existing CHANGELOG - file for that minor version. -1. Submit a PR for the CHANGELOG changes. -1. Submit a PR for README changes, in particular, Compatibility, Feature status, - and any other sections that may need updating. +1. Update dependencies for sidecars via + [go-modules-update.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/HEAD/release-tools/go-modules-update.sh), + and get PRs approved and merged. 1. Check that all [canary CI - jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) are passing, + jobs](https://testgrid.k8s.io/sig-storage-csi-ci) are passing, and that test coverage is adequate for the changes that are going into the release. 1. Check that the post-\-push-images builds are succeeding. - [Example](https://k8s-testgrid.appspot.com/sig-storage-image-build#post-external-snapshotter-push-images) + [Example](https://testgrid.k8s.io/sig-storage-image-build#post-external-snapshotter-push-images) +1. Generate release notes. + 1. Download the latest version of the [K8s release notes generator](https://github.com/kubernetes/release/tree/HEAD/cmd/release-notes) + 1. Create a + [Github personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) + with `repo:public_repo` access + 1. For patch release, use the script generate_patch_release_notes.sh. Read the instructions at the top of the + script. The script also creates PRs for each branch. + 1. For new minor releases, follow these steps and replace arguments with the relevant + information. + * Clean up old cached information (also needed if you are generating release + notes for multiple repos) + ```bash + rm -rf /tmp/k8s-repo + ``` + * For new minor releases on master: + ```bash + GITHUB_TOKEN= release-notes \ + --discover=mergebase-to-latest \ + --org=kubernetes-csi \ + --repo=external-provisioner \ + --required-author="" \ + --markdown-links \ + --output out.md + ``` + 1. Compare the generated output to the new commits for the release to check if + any notable change missed a release note. + 1. Reword release notes as needed, ideally in the original PRs so that the + release notes can be regnerated. Make sure to check notes for breaking + changes and deprecations. + 1. If release is a new major/minor version, create a new `CHANGELOG-..md` + file. + 1. Submit a PR for the CHANGELOG changes. +1. Submit a PR for README changes, in particular, Compatibility, Feature status, + and any other sections that may need updating. 1. Make sure that no new PRs have merged in the meantime, and no PRs are in flight and soon to be merged. 1. Create a new release following a previous release as a template. Be sure to select the correct @@ -101,7 +95,7 @@ naming convention `-on-`. [external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new) 1. If release was a new major/minor version, create a new `release-` branch at that commit. -1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build). +1. Check [image build status](https://testgrid.k8s.io/sig-storage-image-build). 1. Promote images from k8s-staging-sig-storage to registry.k8s.io/sig-storage. From the [k8s image repo](https://github.com/kubernetes/k8s.io/tree/HEAD/registry.k8s.io/images/k8s-staging-sig-storage), @@ -120,7 +114,7 @@ naming convention `-on-`. The following jobs are triggered after tagging to produce the corresponding image(s): -https://k8s-testgrid.appspot.com/sig-storage-image-build +https://testgrid.k8s.io/sig-storage-image-build Clicking on a failed build job opens that job in https://prow.k8s.io. Next to the job title is a rerun icon (circle with arrow). Clicking it opens a popup diff --git a/generate_patch_release_notes.sh b/generate_patch_release_notes.sh new file mode 100755 index 00000000..606c24d2 --- /dev/null +++ b/generate_patch_release_notes.sh @@ -0,0 +1,106 @@ +#!/bin/bash + +# Copyright 2023 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Usage: generate_patch_release_notes.sh +# +# Generates and creates PRs for kubernetes-csi patch releases. +# +# Required environment variables +# CSI_RELEASE_TOKEN: Github token needed for generating release notes +# GITHUB_USER: Github username to create PRs with +# +# Instructions: +# 1. Login with "gh auth login" +# 2. Copy this script to the kubernetes-csi directory (one directory above the +# repos) +# 3. Update the repos and versions in the $releases array +# 4. Set environment variables +# 5. Run script from the kubernetes-csi directory +# +# Caveats: +# - This script doesn't handle regenerating and updating existing PRs yet. +# It might work if you comment out the PR creation line + +set -e +set -x + +releases=( +# "external-attacher 4.4.1" +# "external-provisioner 3.6.1" +# "external-snapshotter 6.2.3" +) + +function gen_patch_relnotes() { + rm out.md || true + rm -rf /tmp/k8s-repo || true + GITHUB_TOKEN="$CSI_RELEASE_TOKEN" \ + release-notes --discover=patch-to-latest --branch="$2" \ + --org=kubernetes-csi --repo="$1" \ + --required-author="" --markdown-links --output out.md +} + +for rel in "${releases[@]}"; do + read -r repo version <<< "$rel" + + # Parse minor version + minorPattern="(^[[:digit:]]+\.[[:digit:]]+)\." + [[ "$version" =~ $minorPattern ]] + minor="${BASH_REMATCH[1]}" + + echo "$repo" "$version" "$minor" + + pushd "$repo/CHANGELOG" + + git fetch upstream + + # Create branch + branch="changelog-release-$minor" + git checkout master + git branch -D "$branch" || true + git checkout --track "upstream/release-$minor" -b "$branch" + + # Generate release notes + gen_patch_relnotes "$repo" "release-$minor" + cat > tmp.md <> tmp.md + echo >> tmp.md + + file="CHANGELOG-$minor.md" + cat "$file" >> tmp.md + mv tmp.md "$file" + + git add -u + git commit -m "Add changelog for $version" + git push -f origin "$branch" + + # Create PR +prbody=$(cat </dev/null)" ]; then + git checkout master && git branch -d "module-update-$i" + fi + git checkout -B "module-update-$i" "origin/$i" + rm -rf .git/MERGE* + if ! git subtree pull --squash --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master; then + # Sometimes "--squash" leads to merge conflicts. Because we know that "release-tools" + # is an unmodified copy of csi-release-tools, we can automatically resolve that + # by replacing it completely. + if [ -e .git/MERGE_MSG ] && [ -e .git/FETCH_HEAD ] && grep -q "^# Conflict" .git/MERGE_MSG; then + rm -rf release-tools + mkdir release-tools + git archive FETCH_HEAD | tar -C release-tools -xf - + git add release-tools + git commit --file=.git/MERGE_MSG + else + exit 1 + fi + fi + RETRY=0 + while ! ./release-tools/go-get-kubernetes.sh -p "$v" && RETRY < $MAX_RETRY + do + RETRY=$((RETRY+1)) + go mod tidy && go mod vendor && go mod tidy + done + go mod tidy && go mod vendor && go mod tidy + git add --all + git commit -m "Update dependency go modules for k8s v$v" + git remote set-url origin "https://github.com/$username/$repo.git" + make test + git push origin "module-update-$i" --force + # Create PR +prbody=$(cat < as # environment variable, then it must write a suitable test driver configuration # into that file in addition to installing the driver. -configvar CSI_PROW_DRIVER_VERSION "v1.11.0" "CSI driver version" +configvar CSI_PROW_DRIVER_VERSION "v1.12.0" "CSI driver version" configvar CSI_PROW_DRIVER_REPO https://github.com/kubernetes-csi/csi-driver-host-path "CSI driver repo" configvar CSI_PROW_DEPLOYMENT "" "deployment" configvar CSI_PROW_DEPLOYMENT_SUFFIX "" "additional suffix in kubernetes-x.yy[suffix].yaml files" @@ -240,7 +240,7 @@ configvar CSI_PROW_SIDECAR_E2E_IMPORT_PATH "none" "CSI Sidecar E2E package" # of the cluster. The alternative would have been to (cross-)compile csi-sanity # and install it inside the cluster, which is not necessarily easier. configvar CSI_PROW_SANITY_REPO https://github.com/kubernetes-csi/csi-test "csi-test repo" -configvar CSI_PROW_SANITY_VERSION v5.0.0 "csi-test version" +configvar CSI_PROW_SANITY_VERSION v5.2.0 "csi-test version" configvar CSI_PROW_SANITY_PACKAGE_PATH github.com/kubernetes-csi/csi-test "csi-test package" configvar CSI_PROW_SANITY_SERVICE "hostpath-service" "Kubernetes TCP service name that exposes csi.sock" configvar CSI_PROW_SANITY_POD "csi-hostpathplugin-0" "Kubernetes pod with CSI driver" diff --git a/pull-test.sh b/pull-test.sh index b019c177..5ea051ec 100755 --- a/pull-test.sh +++ b/pull-test.sh @@ -25,6 +25,7 @@ CSI_RELEASE_TOOLS_DIR="$(pwd)" # Update the other repo. cd "$PULL_TEST_REPO_DIR" +git reset --hard # Shouldn't be necessary, but somehow is to avoid "fatal: working tree has modifications. Cannot add." (https://stackoverflow.com/questions/3623351/git-subtree-pull-says-that-the-working-tree-has-modifications-but-git-status-sa) git subtree pull --squash --prefix=release-tools "$CSI_RELEASE_TOOLS_DIR" master git log -n2