Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Comment with dependency diff #120

Merged
merged 44 commits into from
Sep 17, 2024
Merged
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
2072102
Raw copy of the script from Day One Android
wzieba Sep 13, 2024
8c3785f
Accept configuration, owner and repo parameters
wzieba Sep 13, 2024
f9f6019
Support Kotlin script (`*.gradle.kts`) files as well
wzieba Sep 13, 2024
b1d21df
Support Dependencies Catalog (`*.toml`) files
wzieba Sep 13, 2024
b9fc9f3
Use fastlane action for commenting on a PR
wzieba Sep 13, 2024
8a79cae
Provide module argument
wzieba Sep 13, 2024
1c0d130
Remove dependencyTreeDiffCommentToGitHub execution if no .gradle file…
wzieba Sep 13, 2024
65a0f10
Fix checking for arguments
wzieba Sep 13, 2024
453286e
Use -E for file pattern matching
wzieba Sep 13, 2024
48a72d9
Fix typo when providing reuse-identifier
wzieba Sep 13, 2024
1d4c242
Fix running fastlane action
wzieba Sep 13, 2024
8a7c97d
Add support for buildEnvironment diff
wzieba Sep 13, 2024
05b0c1e
Use template to generate comment body
wzieba Sep 13, 2024
0eb94ec
Actually create $DIFF_BUILD_ENV_FILE. Check if diff files exist befor…
wzieba Sep 13, 2024
8fecda9
Apply fixes to how comment body is created
wzieba Sep 13, 2024
9b54d2a
Don't call `diff` tool
wzieba Sep 13, 2024
8ecae63
Improve formatting of the comment messages
wzieba Sep 13, 2024
1b4f2d8
Log status and diff before checkout
wzieba Sep 15, 2024
470dc33
Sometimes, a diff might be generated (e.g. Gemfile.lock after gem ins…
wzieba Sep 15, 2024
06908dc
Use file to keep comment body
wzieba Sep 15, 2024
87bbed0
Another approach to fix Argument list too long
wzieba Sep 15, 2024
48964e6
Verify if the problem is size of the comment
wzieba Sep 15, 2024
f7a7dfc
Use comment_on_pr script instead of fastlane action
wzieba Sep 15, 2024
f9ed024
Pass comment file to comment_on_pr
wzieba Sep 15, 2024
e6036b3
Possible fix for Argument list too long in comment_on_pr
wzieba Sep 15, 2024
5ce8db0
Pass json_payload_file as a file
wzieba Sep 15, 2024
d3b312d
Pass raw file to jq
wzieba Sep 15, 2024
6b9b0d8
Cover against GitHub comment max size
wzieba Sep 16, 2024
3fabdf1
Check if adding the new content will exceed the maximum allowed size
wzieba Sep 16, 2024
1f47440
Improve markdown formatting
wzieba Sep 16, 2024
c13194f
Remove quotation marks from configuration name
wzieba Sep 16, 2024
a907fe1
Don't require providing owner and repo names
wzieba Sep 16, 2024
fbefec7
Add a changelog entry
wzieba Sep 16, 2024
a5c13b6
Remove forgotten debug code
wzieba Sep 17, 2024
32fc243
Improve clarity with json_payload_file creation
wzieba Sep 17, 2024
0e36309
Add user manual for the comment_with_dependency_diff
wzieba Sep 17, 2024
c00a998
Protect against lack of BUILDKITE_PULL_REQUEST
wzieba Sep 17, 2024
20681ce
Use $BUILDKITE_PULL_REQUEST_BASE_BRANCH instead of fetching base bran…
wzieba Sep 17, 2024
e12d73b
Use uppercase for baseBranch
wzieba Sep 17, 2024
823cf96
Move dependency tree checksum
wzieba Sep 17, 2024
613fcab
Account the warnings message + header size when calculating possible …
wzieba Sep 17, 2024
6696f47
Add a link to "content exceeds" message
wzieba Sep 17, 2024
a60c013
Include header in "content exceeds" case
wzieba Sep 17, 2024
242bf1e
Merge branch 'trunk' into comment_with_dependency_diff
wzieba Sep 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ _None._

### New Features

_None._
- Add a script for commenting dependencies diff on PRs in Android projects [#120]

### Bug Fixes

Expand Down
15 changes: 12 additions & 3 deletions bin/comment_on_pr
Original file line number Diff line number Diff line change
Expand Up @@ -126,20 +126,29 @@ $comment_body_id
$arg_pr_comment
EOF
)
json_payload=$(jq -c --null-input --arg body "$comment_body" '{body: $body}')

# Write the comment body to a temporary file to fix possible "Argument list too long" error
comment_body_file=$(mktemp)
echo "$comment_body" > "$comment_body_file"

json_payload_file=$(mktemp)
jq -c --null-input \
--rawfile body "$comment_body_file" \
'{ body: $body }' \
> "$json_payload_file"

if [[ -n "$existing_comment_id" && "$opt_if_exist" == "update" ]]; then
echo "Update the existing comment: $existing_comment_id"
github_api "$issues_endpoint/comments/$existing_comment_id" \
--fail-with-body -X PATCH \
-H "Content-Type: application/json" \
-d "${json_payload}"
--data-binary @"$json_payload_file"
elif [[ -n "$arg_pr_comment" ]]; then
echo "Post a new comment"
github_api "$issues_endpoint/${BUILDKITE_PULL_REQUEST}/comments" \
--fail-with-body -X POST \
-H "Content-Type: application/json" \
-d "${json_payload}"
--data-binary @"$json_payload_file"
else
# No comment body was given in CLI, so no new comment to post
echo "No new comment to post"
Expand Down
181 changes: 181 additions & 0 deletions bin/comment_with_dependency_diff
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
#!/bin/bash

# This script is used in a Buildkite CI pipeline to analyze and comment on dependency changes in a
# GitHub pull request (PR). It compares the Gradle project dependencies and build environment between the
# base branch and the head branch to identify changes caused by updates to Gradle files.
#
# Usage:
# ./comment_with_dependency_diff <MODULE> <CONFIGURATION>
#
# Parameters:
# <MODULE> : The Gradle module whose dependencies will be checked.
# <CONFIGURATION> : The Gradle configuration to inspect for dependencies (e.g., "runtimeClasspath").
#
# Steps:
# 1. The script fetches the base branch of the PR and merges it into the head branch.
# 2. It generates the dependency and build environment reports for both the base and head branches.
# 3. The script compares these reports to detect any changes in the project's dependencies.
# 4. If changes are found, the script creates a comment on the GitHub PR with the results.
# 5. If no changes are detected, the script exits without posting any comment.
#
# Notes:
# - The GitHub token must be set via the environment variable `GITHUB_TOKEN` to authenticate API requests.
# - The comment on the PR will include a summary of changes and a detailed list in expandable sections.
# - A jar file `dependency-tree-diff.jar` is downloaded and executed to perform the dependency diff.
#
# Example:
# ./comment_with_dependency_diff app release
#
# This will analyze the `app` module with the `release` configuration and post the result to the PR.

set -euo pipefail
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📚 The more we add utility scripts like those to this plugin, the more I realize we should have made a better job as providing help / documentation at what each of those do. Especially so that people not as familiar with our CI (e.g. devs that are interested in using some of those helper scripts in their pipelines themselves but have no idea what each utility does) can better understand the purpose of each.

For example, the comment at the start of comment_on_pr is a good example of having some code-comment documentation on what the utility does and how it's expected to be used

I wish we took that habit sooner and applied it to all of our other bin/* utilities in this repo, making them all start with a code-comment explaining their role / context in which those helpers would be used and could help, documenting what the script does and its expected parameters and inputs (e.g. "this script should only run after ./gradlew … has been called and generated the build artifacts", etc).
We should probably make a pass at all of them at some point.

In the meantime, wdyt about us already starting moving towards this good habit and add some intro doc comments in any new helpers we introduce, starting from this one?


MODULE="${1:-}"
CONFIGURATION="${2:-}"

if [ -z "$MODULE" ] || [ -z "$CONFIGURATION" ]; then
echo "Not enough arguments provided. Usage: ./comment_with_dependency_diff <MODULE> <CONFIGURATION>"
exit 1
fi

if [ "${BUILDKITE_PULL_REQUEST:-false}" == "false" ]; then
echo "This script should only be called on pull requests. Be sure to set \`if: build.pull_request.id != null\` on the step invoking it in your Buildkite pipeline"
exit 2
fi

DIFF_DEPENDENCIES_FOLDER="./build/reports/diff"
BASE_BRANCH_DEPENDENCIES_FILE="$DIFF_DEPENDENCIES_FOLDER/base_branch_dependencies.txt"
HEAD_BRANCH_DEPENDENCIES_FILE="$DIFF_DEPENDENCIES_FOLDER/head_branch_dependencies.txt"
BASE_BRANCH_BUILD_ENV_FILE="$DIFF_DEPENDENCIES_FOLDER/base_branch_build_env.txt"
HEAD_BRANCH_BUILD_ENV_FILE="$DIFF_DEPENDENCIES_FOLDER/head_branch_build_env.txt"
DIFF_DEPENDENCIES_FILE="$DIFF_DEPENDENCIES_FOLDER/diff_dependencies.txt"
DIFF_BUILD_ENV_FILE="$DIFF_DEPENDENCIES_FOLDER/diff_build_env.txt"

DEPENDENCY_TREE_VERSION="1.2.1"
DEPENDENCY_TREE_CHECKSUM="39c50f22cd4211a3e52afec43120b7fdf90f9890"

git config --global user.email "[email protected]"
git config --global user.name "Dependency Tree Diff Tool"

echo "--> Starting the check"
BASE_BRANCH=$BUILDKITE_PULL_REQUEST_BASE_BRANCH
echo "--> Base branch is $BASE_BRANCH"

git merge "origin/$BASE_BRANCH" --no-edit

if (git diff --name-status "origin/$BASE_BRANCH" | grep -E "\.gradle(\.kts)?|\.toml" --quiet); then
echo "Gradle files have been changed. Looking for caused dependency changes"
else
echo "Gradle files haven't been changed. There is no need to run the diff"
exit 0
fi

mkdir -p "$DIFF_DEPENDENCIES_FOLDER"

echo "--> Generating dependencies to the file $HEAD_BRANCH_DEPENDENCIES_FILE"
./gradlew :"$MODULE":dependencies --configuration "$CONFIGURATION" >$HEAD_BRANCH_DEPENDENCIES_FILE

echo "--> Generating build environment dependencies for the head branch"
./gradlew buildEnvironment >$HEAD_BRANCH_BUILD_ENV_FILE

echo "--> Generating dependencies to the file $BASE_BRANCH_DEPENDENCIES_FILE"

if ! git diff --quiet; then
echo "Changes detected. Stashing and continuing with the script"
git diff
git stash
fi

git checkout "$BASE_BRANCH"
./gradlew :"$MODULE":dependencies --configuration "$CONFIGURATION" >$BASE_BRANCH_DEPENDENCIES_FILE

echo "--> Generating build environment dependencies for the base branch"
./gradlew buildEnvironment >$BASE_BRANCH_BUILD_ENV_FILE

git checkout "-"

echo "--> Downloading dependency-tree-diff.jar"
curl -v -L "https://github.com/JakeWharton/dependency-tree-diff/releases/download/$DEPENDENCY_TREE_VERSION/dependency-tree-diff.jar" -o dependency-tree-diff.jar
sha=$(sha1sum dependency-tree-diff.jar | cut -d' ' -f1)
if [[ $sha != "$DEPENDENCY_TREE_CHECKSUM" ]]; then
echo "dependency-tree-diff.jar file has unexpected sha1"
exit 1
fi
chmod +x dependency-tree-diff.jar

echo "--> Running dependency-tree-diff.jar"
./dependency-tree-diff.jar $BASE_BRANCH_DEPENDENCIES_FILE $HEAD_BRANCH_DEPENDENCIES_FILE >$DIFF_DEPENDENCIES_FILE

./dependency-tree-diff.jar $BASE_BRANCH_BUILD_ENV_FILE $HEAD_BRANCH_BUILD_ENV_FILE >$DIFF_BUILD_ENV_FILE

generate_project_dependencies_comment_body() {
cat <<EOF

The following changes in project dependencies were detected (configuration \`$CONFIGURATION\`):

<details><summary>expand</summary>

\`\`\`diff
$(cat "$DIFF_DEPENDENCIES_FILE")
\`\`\`

</details>


EOF
}

generate_build_dependencies_comment_body() {
cat <<EOF

The following changes in the build classpath were detected:

<details><summary>expand</summary>

\`\`\`diff
$(cat "$DIFF_BUILD_ENV_FILE")
\`\`\`

</details>

EOF
}

if [ -n "$DIFF_DEPENDENCIES_FILE" ] || [ -n "$DIFF_BUILD_ENV_FILE" ]; then
echo "--> Commenting result to GitHub"

# Create or clear the comment body file
COMMENT_FILE="$DIFF_DEPENDENCIES_FOLDER/comment_body.txt"
true > $COMMENT_FILE # 'true' used to clear the file as a no-op

append_content() {
local header="$1"
local content="$2"
local github_comment_max_size=65536
local warning_message="Content exceeds $github_comment_max_size characters. [Navigate to Buildkite build artifacts](${BUILDKITE_BUILD_URL}#${BUILDKITE_JOB_ID}) to see details."

local current_size
current_size=$(wc -c < "$COMMENT_FILE")

# Calculate if there's enough space for the content and header
if (( current_size + ${#header} + ${#content} > github_comment_max_size - ${#warning_message} )); then
printf "%s\n" "$warning_message" > $COMMENT_FILE
return
fi

printf "\n%s\n" "$header" >> $COMMENT_FILE
printf "%s\n" "$content" >> $COMMENT_FILE
}

if [ -s "$DIFF_DEPENDENCIES_FILE" ]; then
append_content "## Project dependencies changes" "$(generate_project_dependencies_comment_body)"
fi

if [ -s "$DIFF_BUILD_ENV_FILE" ]; then
append_content "## Build environment changes" "$(generate_build_dependencies_comment_body)"
fi

comment_on_pr --id "dependency-diff" --if-exist update "$COMMENT_FILE"
else
comment_on_pr --id "dependency-diff" --if-exist delete
fi