From 0042abf593d2a665608bc7dce1ebcf1e55e76112 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Sun, 16 May 2021 06:56:04 -0400 Subject: [PATCH 1/2] Add Release README and Scripts to create and release tarballs --- arrow/README.md | 31 +-- dev/release/README.md | 240 ++++++++++++++++++ dev/release/check-rat-report.py | 59 +++++ dev/release/create-tarball.sh | 123 +++++++++ dev/release/release-tarball.sh | 72 ++++++ .../release/update_change_log.sh | 15 +- 6 files changed, 510 insertions(+), 30 deletions(-) create mode 100644 dev/release/README.md create mode 100644 dev/release/check-rat-report.py create mode 100755 dev/release/create-tarball.sh create mode 100755 dev/release/release-tarball.sh rename change_log.sh => dev/release/update_change_log.sh (75%) diff --git a/arrow/README.md b/arrow/README.md index 674c3fc6c8b2..7c54da04a106 100644 --- a/arrow/README.md +++ b/arrow/README.md @@ -174,33 +174,6 @@ specific JIRA issues and reference them in these code comments. For example: // This is not sound because .... see https://issues.apache.org/jira/browse/ARROW-nnnnn ``` -# Publishing to crates.io +# Releases and publishing to crates.io -An Arrow committer can publish this crate after an official project release has -been made to crates.io using the following instructions. - -Follow [these -instructions](https://doc.rust-lang.org/cargo/reference/publishing.html) to -create an account and login to crates.io before asking to be added as an owner -of the [arrow crate](https://crates.io/crates/arrow). - -Checkout the tag for the version to be released. For example: - -```bash -git checkout apache-arrow-0.11.0 -``` - -If the Cargo.toml in this tag already contains `version = "0.11.0"` (as it -should) then the crate can be published with the following command: - -```bash -cargo publish -``` - -If the Cargo.toml does not have the correct version then it will be necessary -to modify it manually. Since there is now a modified file locally that is not -committed to GitHub it will be necessary to use the following command. - -```bash -cargo publish --allow-dirty -``` +Please see the [release](../dev/release/README.md) for details on how to create arrow releases diff --git a/dev/release/README.md b/dev/release/README.md new file mode 100644 index 000000000000..2f8f7b3e00d6 --- /dev/null +++ b/dev/release/README.md @@ -0,0 +1,240 @@ + + +# Release Process + +## Branching + +We would maintain two branches: `active_release` and `master`. +* All new PRs are created and merged against `master` +* All versions are created from the `active_release` branch +* Once merged to master, changes are "cherry-picked" (via a hopefully soon to be automated process), to the `active_release` branch based on the judgement of the original PR author and maintainers. + +* We do not merge breaking api changes, as defined in [Rust RFC 1105](https://github.com/rust-lang/rfcs/blob/master/text/1105-api-evolution.md) to the `active_release` + +Please see the [original proposal](https://docs.google.com/document/d/1tMQ67iu8XyGGZuj--h9WQYB9inCk6c2sL_4xMTwENGc/edit?ts=60961758) document the rational of this change. + +## Release Branching +We aim to release every other week from the `active_release` branch. + +Every other Monday, a maintainer proposes a minor (e.g. `4.1.0` to `4.2.0`) or patch (e.g `4.1.0` to `4.1.1`) release, depending on changes to the `active_release` in the previous 2 weeks, following the process beloe. + +If this release is approved by at least three PMC members, a new version from that tarball is released to crates.io later in the week. + +Apache Arrow in general does synchronized major releases every three months. The Rust implementation aims to do its major releases in the same time frame. + +# Release Mechanics + +This directory contains the scripts used to manage an Apache Arrow Release. + +# Process Overview +As part of the Apache governance model, official releases consist of +signed source tarballs approved by the PMC. + +We then use the code in the approved source tarball to release to +crates.io, the Rust ecosystem's package manager. + +## Branching + + +# Release Preparation + +# Change Log + +We create a `CHANGELOG.md` so our users know what has been changed between releases. + +The CHANGELOG is created automatically using +[change_log.sh](https://github.com/apache/arrow-rs/blob/master/change_log.sh) + +This script creates a changelog using github issues and the +labels associated with them. + + + + +# Mechanics of creating a release + +## Prepare the release branch and tags + +First, ensure that `active_release` contains the content of the desired release. For minor and patch releases, no additional steps are needed. + +To prepare for *a major release*, change `active release` to point at the latest `master` with commands such as: + +``` +git checkout active_release +git fetch apache +git reset --hard apache/master +git push -f +``` + +### Update CHANGELOG.md + Version + +Now prepare a PR to update `CHANGELOG.md` and versions on `active_release` branch to reflect the planned release. + +See [#298](https://github.com/apache/arrow-rs/pull/298) for an example. + +Here are the commands used to prepare the 4.1.0 release: + +```bash +git checkout active_release +git pull +git checkout -b make-release + +# manully edit ./dev/release/update_change_log.sh to reflect the release version +# create the changelog +CHANGELOG_GITHUB_TOKEN= ./dev/release/update_change_log.sh +# review change log / edit issues and labels if needed, rerun +git commit -a -m 'Create changelog' + +# update versions +sed -i '' -e 's/5.0.0-SNAPSHOT/4.1.0/g' `find . -name 'Cargo.toml'` +git commit -a -m 'Update version' +``` + +Note that when reviewing the change log, rather than editing the +`CHANGELOG.md`, it is preferred to update the issues and their labels +(e.g. add `invalid` label to exclude them from release notes) + + +## Prepare release candidate tarball + +(Note you need to be a committer to run these scripts as they upload to the apache svn distribution servers) + +### Create git tag for the release: + +While the official release artifact is a signed tarball, we also tag the commit it was created for convenience and code archaeology. + +Using a string such as `4.0.1` as the ``, create and push the tag thusly: + +```shell +git fetch apache +git tag apache/active_release +# push tag to apache +git push apache +``` + +### Pick an Release Candidate (RC) number + +Pick numbers in sequential order, with `0` for `rc1`, `1` for `rc1`, etc. + +### Create, sign, and upload tarball + +Run the `create-tarball.sh` with the `` tag and `` and you found in previous steps: + +```shell +./dev/release/create-tarball.sh 4.1.0 2 +``` + +This script + +1. creates and uploads a release candidate tarball to the [arrow +dev](https://dist.apache.org/repos/dist/dev/arrow) location on the +apache distribution svn server + +2. provide you an email template to +send to dev@arrow.apache.org for release voting. + + +### Vote on Release Candidate tarball + +Send the email output from the script to dev@arrow.apache.org. The email should look like + +``` +To: dev@arrow.apache.org +Subject: [VOTE][RUST] Release Apache Arrow + +Hi, + +I would like to propose a release of Apache Arrow Rust +Implementation, version 4.1.0. + +This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1] + +The proposed release tarball and signatures are hosted at [2]. +The changelog is located at [3]. + +Please download, verify checksums and signatures, run the unit tests, +and vote on the release. + +The vote will be open for at least 72 hours. + +[ ] +1 Release this as Apache Arrow Rust +[ ] +0 +[ ] -1 Do not release this as Apache Arrow Rust because... + +[1]: https://github.com/apache/arrow-rs/tree/a5dd428f57e62db20a945e8b1895de91405958c4 +[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-4.1.0 +[3]: https://github.com/apache/arrow-rs/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md +``` + +For the release to become "official" it needs at least three PMC members to vote +1 on it. + + +#### Verifying Release Candidates + +There is a script in this repository which can be used to help `dev/release/verify-release-candidate.sh` assist the verification process. Run it like: + +``` +./dev/release/verify-release-candidate.sh 4.1.0 2 +``` + + +#### If the release is not approved + +If the release is not approved, fix whatever the problem is and try again with the next RC number + + +### If the release is approved, + +Move tarball to the release location in SVN, e.g. https://dist.apache.org/repos/dist/release/arrow/arrow-4.1.0/, using the `release-tarball.sh` script: + +```shell +./dev/release/release-tarball.sh 4.1.0 2 +``` + +### Publish on Crates.io + +Only approved releases of the tarball should be published to +crates.io, in order to conform to Apache Software Foundation +governance standards. + +An Arrow committer can publish this crate after an official project release has +been made to crates.io using the following instructions. + +Follow [these +instructions](https://doc.rust-lang.org/cargo/reference/publishing.html) to +create an account and login to crates.io before asking to be added as an owner +of the [arrow crate](https://crates.io/crates/arrow). + +Download and unpack the official release tarball + +If the Cargo.toml in this tag already contains `version = "0.11.0"` (as it +should) then the crate can be published with the following command: + +```shell +cargo publish +``` + +If the Cargo.toml does not have the correct version then it will be necessary +to modify it manually. Since there is now a modified file locally that is not +committed to GitHub it will be necessary to use the following command. + +```shell +cargo publish --allow-dirty +``` diff --git a/dev/release/check-rat-report.py b/dev/release/check-rat-report.py new file mode 100644 index 000000000000..e30d72bddd7f --- /dev/null +++ b/dev/release/check-rat-report.py @@ -0,0 +1,59 @@ +#!/usr/bin/python +############################################################################## +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +############################################################################## +import fnmatch +import re +import sys +import xml.etree.ElementTree as ET + +if len(sys.argv) != 3: + sys.stderr.write("Usage: %s exclude_globs.lst rat_report.xml\n" % + sys.argv[0]) + sys.exit(1) + +exclude_globs_filename = sys.argv[1] +xml_filename = sys.argv[2] + +globs = [line.strip() for line in open(exclude_globs_filename, "r")] + +tree = ET.parse(xml_filename) +root = tree.getroot() +resources = root.findall('resource') + +all_ok = True +for r in resources: + approvals = r.findall('license-approval') + if not approvals or approvals[0].attrib['name'] == 'true': + continue + clean_name = re.sub('^[^/]+/', '', r.attrib['name']) + excluded = False + for g in globs: + if fnmatch.fnmatch(clean_name, g): + excluded = True + break + if not excluded: + sys.stdout.write("NOT APPROVED: %s (%s): %s\n" % ( + clean_name, r.attrib['name'], approvals[0].attrib['name'])) + all_ok = False + +if not all_ok: + sys.exit(1) + +print('OK') +sys.exit(0) diff --git a/dev/release/create-tarball.sh b/dev/release/create-tarball.sh new file mode 100755 index 000000000000..ab3e1d2d2c48 --- /dev/null +++ b/dev/release/create-tarball.sh @@ -0,0 +1,123 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + + +# This script creates a signed tarball in +# dev/dist/apache-arrow-rs--.tar.gz and uploads it to +# the "dev" area of the dist.apache.arrow repository and prepares an +# email for sending to the dev@arrow.apache.org list for a formal +# vote. +# +# See release/README.md for full release instructions +# +# Requirements: +# +# 1. gpg setup for signing and have uploaded your public +# signature to https://pgp.mit.edu/ +# +# 2. Logged into the apache svn server with the appropriate +# credentials +# +# +# Based in part on 02-source.sh from apache/arrow +# + +set -e + +SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SOURCE_TOP_DIR="$(cd "${SOURCE_DIR}/../../" && pwd)" + +if [ "$#" -ne 2 ]; then + echo "Usage: $0 " + echo "ex. $0 4.1.0 2" + exit +fi + +tag=$1 +rc=$2 + +release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag}) + +release=apache-arrow-rs-${tag} +distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc} +tarname=${release}.tar.gz +tarball=${distdir}/${tarname} +url="https://dist.apache.org/repos/dist/dev/arrow/${release}-rc${rc}" + +echo "Attempting to create ${tarball} from tag ${tag}" + + +if [ -z "$release_hash" ]; then + echo "Cannot continue: unknown git tag: $tag" +fi + +echo "Draft email for dev@arrow.apache.org mailing list" +echo "" +echo "---------------------------------------------------------" +cat < containing the files in git at $release_hash +# the files in the tarball are prefixed with {tag} (e.g. 4.0.1) +mkdir -p ${distdir} +(cd "${SOURCE_TOP_DIR}" && git archive ${release_hash} --prefix ${release}/ | gzip > ${tarball}) + +echo "Running rat license checker on ${tarball}" +${SOURCE_DIR}/run-rat.sh ${tarball} + +echo "Signing tarball and creating checksums" +gpg --armor --output ${tarball}.asc --detach-sig ${tarball} +# create signing with relative path of tarball +# so that they can be verified with a command such as +# shasum --check apache-arrow-rs-4.1.0-rc2.tar.gz.sha512 +(cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256 +(cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512 + +echo "Uploading to apache dist/dev to ${url}" +svn co --depth=empty https://dist.apache.org/repos/dist/dev/arrow ${SOURCE_TOP_DIR}/dev/dist +svn add ${distdir} +svn ci -m "Apache Arrow Rust ${tag} ${rc}" ${distdir} diff --git a/dev/release/release-tarball.sh b/dev/release/release-tarball.sh new file mode 100755 index 000000000000..74fcad710120 --- /dev/null +++ b/dev/release/release-tarball.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# This script copies a tarball from the "dev" area of the +# dist.apache.arrow repository to the "release" area +# +# This script should only be run after the release has been approved +# by the arrow PMC committee. +# +# See release/README.md for full release instructions +# +# Based in part on post-01-upload.sh from apache/arrow + + +set -e +set -u + +if [ "$#" -ne 2 ]; then + echo "Usage: $0 " + echo "ex. $0 4.1.0 2" + exit +fi + +version=$1 +rc=$2 + +tmp_dir=tmp-apache-arrow-dist + +echo "Recreate temporary directory: ${tmp_dir}" +rm -rf ${tmp_dir} +mkdir -p ${tmp_dir} + +echo "Clone dev dist repository" +svn \ + co \ + https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-${version}-rc${rc} \ + ${tmp_dir}/dev + +echo "Clone release dist repository" +svn co https://dist.apache.org/repos/dist/release/arrow ${tmp_dir}/release + +echo "Copy ${version}-rc${rc} to release working copy" +release_version=arrow-${version} +mkdir -p ${tmp_dir}/release/${release_version} +cp -r ${tmp_dir}/dev/* ${tmp_dir}/release/${release_version}/ +svn add ${tmp_dir}/release/${release_version} + +echo "Commit release" +svn ci -m "Apache Arrow Rust ${version}" ${tmp_dir}/release + +echo "Clean up" +rm -rf ${tmp_dir} + +echo "Success! The release is available here:" +echo " https://dist.apache.org/repos/dist/release/arrow/${release_version}" diff --git a/change_log.sh b/dev/release/update_change_log.sh similarity index 75% rename from change_log.sh rename to dev/release/update_change_log.sh index 3d9b57c674d8..a11f96698369 100755 --- a/change_log.sh +++ b/dev/release/update_change_log.sh @@ -18,9 +18,22 @@ # under the License. # +# invokes the changelog generator with the config located in +# arrow-rs/.github_changelog_generator +# +# Usage: +# CHANGELOG_GITHUB_TOKEN= ./update_change_log.sh + +set -e + +SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SOURCE_TOP_DIR="$(cd "${SOURCE_DIR}/../../" && pwd)" + +pushd ${SOURCE_TOP_DIR} docker run -it --rm -e CHANGELOG_GITHUB_TOKEN=$CHANGELOG_GITHUB_TOKEN -v "$(pwd)":/usr/local/src/your-app githubchangeloggenerator/github-changelog-generator \ --user apache \ --project arrow-rs \ --since-commit 2021-04-20 \ - --future-release 4.0.1 + --future-release 4.1.0 + sed -i "s/\\\n/\n\n/" CHANGELOG.md From 7c3e6aa744a5278c04608db00ee5c9efa89e0808 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Mon, 24 May 2021 14:01:37 -0400 Subject: [PATCH 2/2] Suggestions from Andy Grove --- dev/release/README.md | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index 2f8f7b3e00d6..9581a6314669 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -224,17 +224,13 @@ of the [arrow crate](https://crates.io/crates/arrow). Download and unpack the official release tarball -If the Cargo.toml in this tag already contains `version = "0.11.0"` (as it -should) then the crate can be published with the following command: +Verify that the Cargo.toml in the tarball contains the correct version +(e.g. `version = "0.11.0"`) and then publish the crate with the +following commands ```shell -cargo publish -``` - -If the Cargo.toml does not have the correct version then it will be necessary -to modify it manually. Since there is now a modified file locally that is not -committed to GitHub it will be necessary to use the following command. - -```shell -cargo publish --allow-dirty +(cd arrow && cargo publish) +(cd arrow_flight && cargo publish) +(cd parquet && cargo publish) +(cd parquet_derive && cargo publish) ```