Make the release workflow more resilient (#4728)

## Summary

Currently, it is possible to create a tag and then have the release
fail, which is a problem since we can't edit the tag
(#4468). This change the
release process so that the tag is created inside the release workflow.
This leaves as a failure mode that we have published to pypi but then
creating the tag or GitHub release doesn't work, but in this case we can
restart and the pypi upload is just skipped because we use the skip
existing option.

The release workflow is started by a workflow dispatch with the tag
instead of creating the tag yourself. You can start the release workflow
without a tag to do a dry run which does not publish an artifacts. You
can optionally add a git sha to the workflow run and it will verify that
the release runs on the mentioned commit.

This also adds docs on how to release and a small style improvement for
the maturin integration.

## Test Plan

Testing is hard since we can't do real releases, i've tested a minimized
workflow in a separate dummy repository.

.github/workflows/ci.yaml

@@ -220,11 +220,10 @@ jobs:
       - name: "Build wheels"
         uses: PyO3/maturin-action@v1
         with:
-          manylinux: auto
           args: --out dist
       - name: "Test wheel"
         run: |
-          pip install dist/${{ env.PACKAGE_NAME }}-*.whl --force-reinstall
+          pip install --force-reinstall --find-links dist ${{ env.PACKAGE_NAME }}
           ruff --help
           python -m ruff --help
       - name: "Remove wheels from cache"

.github/workflows/release.yaml

@@ -2,8 +2,17 @@ name: "[ruff] Release"
 
 on:
   workflow_dispatch:
-  release:
-    types: [ published ]
+    inputs:
+      tag:
+        description: "The version to tag, without the leading 'v'. If omitted, will initiate a dry run skipping uploading artifact."
+        type: string
+      sha:
+        description: "Optionally, the full sha of the commit to be released"
+        type: string
+  push:
+    paths:
+      # When we change pyproject.toml, we want to ensure that the maturin builds still work
+      - pyproject.toml
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -394,7 +403,8 @@ jobs:
       - linux-cross
       - musllinux
       - musllinux-cross
-    if: "startsWith(github.ref, 'refs/tags/')"
+    # If you don't set an input it's a dry run skipping uploading artifact
+    if: ${{ inputs.tag }}
     environment:
       name: release
     permissions:
@@ -403,11 +413,34 @@ jobs:
       # For GitHub release publishing
       contents: write
     steps:
+      - name: Consistency check tag
+        run: |
+          version=$(grep "version = " pyproject.toml | sed -e 's/version = "\(.*\)"/\1/g')
+          if [ "${{ inputs.tag }}" != "${version}" ]; then
+            echo "The input tag does not match the version from pyproject.toml:" >&2
+            echo "${{ inputs.tag }}" >&2
+            echo "${version}" >&2
+            exit 1
+          else
+            echo "Releasing ${version}"
+          fi
+      - name: Consistency check sha
+        if: ${{ inputs.sha }}
+        run: |
+          git_sha=$(git rev-parse HEAD)
+          if [ "${{ inputs.sha }}" != "${git_sha}" ]; then
+            echo "The specified sha does not match the git checkout" >&2
+            echo "${{ inputs.sha }}" >&2
+            echo "${git_sha}" >&2
+            exit 1
+          else
+            echo "Releasing ${git_sha}"
+          fi
       - uses: actions/download-artifact@v3
         with:
           name: wheels
           path: wheels
-      - name: "Publish to PyPi"
+      - name: Publish to PyPi
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
           skip-existing: true
@@ -417,10 +450,20 @@ jobs:
         with:
           name: binaries
           path: binaries
+      - name: git tag
+        run: |
+          git config user.email "[email protected]"
+          git config user.name "Ruff Release CI"
+          git tag -m "v${{ inputs.tag }}" "v${{ inputs.tag }}"
+          # If there is duplicate tag, this will fail. The publish to pypi action will have been a noop (due to skip
+          # existing), so we make a non-destructive exit here
+          git push --tags
       - name: "Publish to GitHub"
         uses: softprops/action-gh-release@v1
         with:
+          draft: true
           files: binaries/*
+          tag_name: v${{ inputs.tag }}
 
   # After the release has been published, we update downstream repositories
   # This is separate because if this fails the release is still fine, we just need to do some manual workflow triggers

CONTRIBUTING.md

@@ -271,6 +271,28 @@ them to [PyPI](https://pypi.org/project/ruff/).
 Ruff follows the [semver](https://semver.org/) versioning standard. However, as pre-1.0 software,
 even patch releases may contain [non-backwards-compatible changes](https://semver.org/#spec-item-4).
 
+### Creating a new release
+
+1. Update the version with `rg 0.0.269 --files-with-matches | xargs sed -i 's/0.0.269/0.0.270/g'`
+1. Update `BREAKING_CHANGES.md`
+1. Create a PR with the version and `BREAKING_CHANGES.md` updated
+1. Merge the PR
+1. Run the release workflow with the version number (without starting `v`) as input. Make sure
+   main has your merged PR as last commit
+1. The release workflow will do the following:
+   1. Build all the assets. If this fails (even though we tested in step 4), we haven’t tagged or
+      uploaded anything, you can restart after pushing a fix
+   1. Upload to pypi
+   1. Create and push the git tag (from pyproject.toml). We create the git tag only here
+      because we can't change it ([#4468](https://github.com/charliermarsh/ruff/issues/4468)), so
+      we want to make sure everything up to and including publishing to pypi worked.
+   1. Attach artifacts to draft GitHub release
+   1. Trigger downstream repositories. This can fail without causing fallout, it is possible (if
+      inconvenient) to trigger the downstream jobs manually
+1. Create release notes in GitHub UI and promote from draft to proper release(<https://github.com/charliermarsh/ruff/releases/new>)
+1. If needed, [update the schemastore](https://github.com/charliermarsh/ruff/blob/main/scripts/update_schemastore.py)
+1. If needed, update ruff-lsp and ruff-vscode
+
 ## Ecosystem CI
 
 GitHub Actions will run your changes against a number of real-world projects from GitHub and