Skip to content

Commit

Permalink
Add Release README and Scripts to create and release tarballs
Browse files Browse the repository at this point in the history
  • Loading branch information
@alamb
alamb committed May 22, 2021
1 parent 71c2159 commit 6149d62
Show file tree
Hide file tree
Showing 6 changed files with 510 additions and 30 deletions.
31 changes: 2 additions & 29 deletions arrow/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -174,33 +174,6 @@ specific JIRA issues and reference them in these code comments. For example:
// This is not sound because .... see https://issues.apache.org/jira/browse/ARROW-nnnnn
```

# Publishing to crates.io
# Releases and publishing to crates.io

An Arrow committer can publish this crate after an official project release has
been made to crates.io using the following instructions.

Follow [these
instructions](https://doc.rust-lang.org/cargo/reference/publishing.html) to
create an account and login to crates.io before asking to be added as an owner
of the [arrow crate](https://crates.io/crates/arrow).

Checkout the tag for the version to be released. For example:

```bash
git checkout apache-arrow-0.11.0
```

If the Cargo.toml in this tag already contains `version = "0.11.0"` (as it
should) then the crate can be published with the following command:

```bash
cargo publish
```

If the Cargo.toml does not have the correct version then it will be necessary
to modify it manually. Since there is now a modified file locally that is not
committed to GitHub it will be necessary to use the following command.

```bash
cargo publish --allow-dirty
```
Please see the [release](../dev/release/README.md) for details on how to create arrow releases
240 changes: 240 additions & 0 deletions dev/release/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,240 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Release Process

## Branching

We would maintain two branches: `active_release` and `master`.
* All new PRs are created and merged against `master`
* All versions are created from the `active_release` branch
* Once merged to master, changes are "cherry-picked" (via a hopefully soon to be automated process), to the `active_release` branch based on the judgement of the original PR author and maintainers.

* We do not merge breaking api changes, as defined in [Rust RFC 1105](https://github.com/rust-lang/rfcs/blob/master/text/1105-api-evolution.md) to the `active_release`

Please see the [original proposal](https://docs.google.com/document/d/1tMQ67iu8XyGGZuj--h9WQYB9inCk6c2sL_4xMTwENGc/edit?ts=60961758) document the rational of this change.

## Release Branching
We aim to release every other week from the `active_release` branch.

Every other Monday, a maintainer proposes a minor (e.g. `4.1.0` to `4.2.0`) or patch (e.g `4.1.0` to `4.1.1`) release, depending on changes to the `active_release` in the previous 2 weeks, following the process beloe.

If this release is approved by at least three PMC members, a new version from that tarball is released to crates.io later in the week.

Apache Arrow in general does synchronized major releases every three months. The Rust implementation aims to do its major releases in the same time frame.

# Release Mechanics

This directory contains the scripts used to manage an Apache Arrow Release.

# Process Overview
As part of the Apache governance model, official releases consist of
signed source tarballs approved by the PMC.

We then use the code in the approved source tarball to release to
crates.io, the Rust ecosystem's package manager.

## Branching


# Release Preparation

# Change Log

We create a `CHANGELOG.md` so our users know what has been changed between releases.

The CHANGELOG is created automatically using
[change_log.sh](https://github.com/apache/arrow-rs/blob/master/change_log.sh)

This script creates a changelog using github issues and the
labels associated with them.




# Mechanics of creating a release

## Prepare the release branch and tags

First, ensure that `active_release` contains the content of the desired release. For minor and patch releases, no additional steps are needed.

To prepare for *a major release*, change `active release` to point at the latest `master` with commands such as:

```
git checkout active_release
git fetch apache
git reset --hard apache/master
git push -f
```

### Update CHANGELOG.md + Version

Now prepare a PR to update `CHANGELOG.md` and versions on `active_release` branch to reflect the planned release.

See [#298](https://github.com/apache/arrow-rs/pull/298) for an example.

Here are the commands used to prepare the 4.1.0 release:

```bash
git checkout active_release
git pull
git checkout -b make-release

# manully edit ./dev/release/update_change_log.sh to reflect the release version
# create the changelog
CHANGELOG_GITHUB_TOKEN=<TOKEN> ./dev/release/update_change_log.sh
# review change log / edit issues and labels if needed, rerun
git commit -a -m 'Create changelog'

# update versions
sed -i '' -e 's/5.0.0-SNAPSHOT/4.1.0/g' `find . -name 'Cargo.toml'`
git commit -a -m 'Update version'
```

Note that when reviewing the change log, rather than editing the
`CHANGELOG.md`, it is preferred to update the issues and their labels
(e.g. add `invalid` label to exclude them from release notes)


## Prepare release candidate tarball

(Note you need to be a committer to run these scripts as they upload to the apache svn distribution servers)

### Create git tag for the release:

While the official release artifact is a signed tarball, we also tag the commit it was created for convenience and code archaeology.

Using a string such as `4.0.1` as the `<version>`, create and push the tag thusly:

```shell
git fetch apache
git tag <version> apache/active_release
# push tag to apache
git push apache <version>
```

### Pick an Release Candidate (RC) number

Pick numbers in sequential order, with `0` for `rc1`, `1` for `rc1`, etc.

### Create, sign, and upload tarball

Run the `create-tarball.sh` with the `<version>` tag and `<rc>` and you found in previous steps:

```shell
./dev/release/create-tarball.sh 4.1.0 2
```

This script

1. creates and uploads a release candidate tarball to the [arrow
dev](https://dist.apache.org/repos/dist/dev/arrow) location on the
apache distribution svn server

2. provide you an email template to
send to [email protected] for release voting.


### Vote on Release Candidate tarball

Send the email output from the script to [email protected]. The email should look like

```
To: [email protected]
Subject: [VOTE][RUST] Release Apache Arrow
Hi,
I would like to propose a release of Apache Arrow Rust
Implementation, version 4.1.0.
This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1]
The proposed release tarball and signatures are hosted at [2].
The changelog is located at [3].
Please download, verify checksums and signatures, run the unit tests,
and vote on the release.
The vote will be open for at least 72 hours.
[ ] +1 Release this as Apache Arrow Rust
[ ] +0
[ ] -1 Do not release this as Apache Arrow Rust because...
[1]: https://github.com/apache/arrow-rs/tree/a5dd428f57e62db20a945e8b1895de91405958c4
[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-4.1.0
[3]: https://github.com/apache/arrow-rs/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md
```

For the release to become "official" it needs at least three PMC members to vote +1 on it.


#### Verifying Release Candidates

There is a script in this repository which can be used to help `dev/release/verify-release-candidate.sh` assist the verification process. Run it like:

```
./dev/release/verify-release-candidate.sh 4.1.0 2
```


#### If the release is not approved

If the release is not approved, fix whatever the problem is and try again with the next RC number


### If the release is approved,

Move tarball to the release location in SVN, e.g. https://dist.apache.org/repos/dist/release/arrow/arrow-4.1.0/, using the `release-tarball.sh` script:

```shell
./dev/release/release-tarball.sh 4.1.0 2
```

### Publish on Crates.io

Only approved releases of the tarball should be published to
crates.io, in order to conform to Apache Software Foundation
governance standards.

An Arrow committer can publish this crate after an official project release has
been made to crates.io using the following instructions.

Follow [these
instructions](https://doc.rust-lang.org/cargo/reference/publishing.html) to
create an account and login to crates.io before asking to be added as an owner
of the [arrow crate](https://crates.io/crates/arrow).

Download and unpack the official release tarball

If the Cargo.toml in this tag already contains `version = "0.11.0"` (as it
should) then the crate can be published with the following command:

```shell
cargo publish
```

If the Cargo.toml does not have the correct version then it will be necessary
to modify it manually. Since there is now a modified file locally that is not
committed to GitHub it will be necessary to use the following command.

```shell
cargo publish --allow-dirty
```
59 changes: 59 additions & 0 deletions dev/release/check-rat-report.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/usr/bin/python
##############################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
##############################################################################
import fnmatch
import re
import sys
import xml.etree.ElementTree as ET

if len(sys.argv) != 3:
sys.stderr.write("Usage: %s exclude_globs.lst rat_report.xml\n" %
sys.argv[0])
sys.exit(1)

exclude_globs_filename = sys.argv[1]
xml_filename = sys.argv[2]

globs = [line.strip() for line in open(exclude_globs_filename, "r")]

tree = ET.parse(xml_filename)
root = tree.getroot()
resources = root.findall('resource')

all_ok = True
for r in resources:
approvals = r.findall('license-approval')
if not approvals or approvals[0].attrib['name'] == 'true':
continue
clean_name = re.sub('^[^/]+/', '', r.attrib['name'])
excluded = False
for g in globs:
if fnmatch.fnmatch(clean_name, g):
excluded = True
break
if not excluded:
sys.stdout.write("NOT APPROVED: %s (%s): %s\n" % (
clean_name, r.attrib['name'], approvals[0].attrib['name']))
all_ok = False

if not all_ok:
sys.exit(1)

print('OK')
sys.exit(0)
Loading

0 comments on commit 6149d62

Please sign in to comment.