Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ci): Expand cached state disks before running tests #4962

Merged
merged 6 commits into from
Aug 28, 2022
Merged

fix(ci): Expand cached state disks before running tests #4962

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
89818e8
Expand cached state disks before running tests
teor2345 Aug 27, 2022
60101a4
Install partition management tool
teor2345 Aug 27, 2022
d50900a
There isn't actually a partition on the cached state image
teor2345 Aug 27, 2022
fa218fe
Make e2fsck non-interactive
teor2345 Aug 27, 2022
0359f07
Limit the length of image names to 63 characters
teor2345 Aug 27, 2022
e098efa
Ignore possibly long branch names when matching images, just match th…
teor2345 Aug 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 45 additions & 16 deletions .github/workflows/deploy-gcp-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,9 @@ jobs:
--zone ${{ env.ZONE }}
sleep 60

# Create a docker volume with the new disk we just created.
#
# SSH into the just created VM, and create a docker volume with the newly created disk.
- name: Create ${{ inputs.test_id }} Docker volume
run: |
gcloud compute ssh \
Expand All @@ -157,7 +160,7 @@ jobs:
--ssh-flag="-o ServerAliveInterval=5" \
--command \
"\
sudo mkfs.ext4 /dev/sdb \
sudo mkfs.ext4 -v /dev/sdb \
&& \
docker volume create --driver local --opt type=ext4 --opt device=/dev/sdb \
${{ inputs.test_id }}-${{ env.GITHUB_SHA_SHORT }} \
Expand Down Expand Up @@ -285,11 +288,16 @@ jobs:
DISK_PREFIX=${{ inputs.zebra_state_dir || inputs.disk_prefix }}
fi

# Try to find an image generated from this branch and commit
# Fields are listed in the "Create image from state disk" step
COMMIT_DISK_PREFIX="${DISK_PREFIX}-${GITHUB_REF_SLUG_URL}-${GITHUB_SHA_SHORT}-v${LOCAL_STATE_VERSION}-${NETWORK}-${{ inputs.disk_suffix }}"
# Try to find an image generated from a previous step or run of this commit.
# Fields are listed in the "Create image from state disk" step.
#
# We can't match the full branch name here,
# because it might have been shortened for the image.
#
# The probability of two matching short commit hashes within the same month is very low.
COMMIT_DISK_PREFIX="${DISK_PREFIX}-.+-${{ env.GITHUB_SHA_SHORT }}-v${LOCAL_STATE_VERSION}-${NETWORK}-${{ inputs.disk_suffix }}"
COMMIT_CACHED_DISK_NAME=$(gcloud compute images list --filter="name~${COMMIT_DISK_PREFIX}" --format="value(NAME)" --sort-by=~creationTimestamp --limit=1)
echo "${GITHUB_REF_SLUG_URL}-${GITHUB_SHA_SHORT} Disk: $COMMIT_CACHED_DISK_NAME"
echo "${GITHUB_REF_SLUG_URL}-${{ env.GITHUB_SHA_SHORT }} Disk: $COMMIT_CACHED_DISK_NAME"
if [[ -n "$COMMIT_CACHED_DISK_NAME" ]]; then
echo "Description: $(gcloud compute images describe $COMMIT_CACHED_DISK_NAME --format='value(DESCRIPTION)')"
fi
Expand Down Expand Up @@ -351,7 +359,10 @@ jobs:

# Create a docker volume with the selected cached state.
#
# SSH into the just created VM, and create a docker volume with the recently attached disk.
# SSH into the just created VM, expand the partition and filesystem to fill the entire disk,
# then create a docker volume with the recently attached disk.
# (The cached state and disk are usually the same size,
# but the cached state can be smaller if we just increased the disk size.)
- name: Create ${{ inputs.test_id }} Docker volume
run: |
gcloud compute ssh \
Expand All @@ -361,6 +372,10 @@ jobs:
--ssh-flag="-o ServerAliveInterval=5" \
--command \
"\
sudo e2fsck -v -f -p /dev/sdb \
&& \
sudo resize2fs -p /dev/sdb \
&& \
docker volume create --driver local --opt type=ext4 --opt device=/dev/sdb \
${{ inputs.test_id }}-${{ env.GITHUB_SHA_SHORT }} \
"
Expand Down Expand Up @@ -928,14 +943,22 @@ jobs:
with:
short-length: 7

# Performs formatting on disk name components.
#
# Disk images in GCP are required to be in lowercase, but the blockchain network
# uses sentence case, so we need to downcase ${{ inputs.network }}
# uses sentence case, so we need to downcase ${{ inputs.network }}.
#
# Passes ${{ inputs.network }} to subsequent steps using $NETWORK env variable
- name: Downcase network name for disks
# Disk image names in GCP are limited to 63 characters, so we need to limit
# branch names to 13 characters.
#
# Passes ${{ inputs.network }} to subsequent steps using $NETWORK env variable.
# Passes ${{ env.GITHUB_REF_SLUG_URL }} to subsequent steps using $SHORT_GITHUB_REF env variable.
- name: Format network name and branch name for disks
run: |
NETWORK_CAPS=${{ inputs.network }}
echo "NETWORK=${NETWORK_CAPS,,}" >> $GITHUB_ENV
LONG_GITHUB_REF=${{ env.GITHUB_REF_SLUG_URL }}
echo "SHORT_GITHUB_REF=${LONG_GITHUB_REF:0:13}" >> $GITHUB_ENV

# Setup gcloud CLI
- name: Authenticate to Google Cloud
Expand Down Expand Up @@ -980,7 +1003,7 @@ jobs:
SYNC_HEIGHT=$(echo $DOCKER_LOGS | grep -oE '${{ inputs.height_grep_text }}\([0-9]+\)' | grep -oE '[0-9]+' | tail -1 || [[ $? == 1 ]])
echo "SYNC_HEIGHT=$SYNC_HEIGHT" >> $GITHUB_ENV

# Sets the $UPDATE_SUFFIX env var to "-update" if using cached state,
# Sets the $UPDATE_SUFFIX env var to "-u" if using cached state,
# and the empty string otherwise.
#
# Also sets a unique date and time suffix $TIME_SUFFIX.
Expand All @@ -989,26 +1012,32 @@ jobs:
UPDATE_SUFFIX=""

if [[ "${{ inputs.needs_zebra_state }}" == "true" ]]; then
UPDATE_SUFFIX="-update"
UPDATE_SUFFIX="-u"
fi

TIME_SUFFIX=$(date '+%Y-%m-%d-%H-%M-%S' --utc)
# We're going to delete old images after a month, so we don't need the year here
TIME_SUFFIX=$(date '+%m%d%H%M%S' --utc)

echo "UPDATE_SUFFIX=$UPDATE_SUFFIX" >> $GITHUB_ENV
echo "TIME_SUFFIX=$TIME_SUFFIX" >> $GITHUB_ENV

# Create an image from disk that will be used for following/other tests
# Create an image from disk that will be used for following/other tests.
#
# This image can contain:
# - Zebra cached state
# - Zebra + lightwalletd cached state
# Which cached state is being saved to the disk is defined by ${{ inputs.disk_prefix }}
# Which cached state is being saved to the disk is defined by ${{ inputs.disk_prefix }}.
#
# The image name must be unique, and be 63 characters or less.
# The timestamp makes images from the same commit unique,
# as long as they don't finish in the same second.
#
# Force the image creation (--force) as the disk is still attached even though is not being
# used by the container
# used by the container.
- name: Create image from state disk
run: |
gcloud compute images create \
"${{ inputs.disk_prefix }}-${{ env.GITHUB_REF_SLUG_URL }}-${{ env.GITHUB_SHA_SHORT }}-v${{ env.STATE_VERSION }}-${{ env.NETWORK }}-${{ inputs.disk_suffix }}$UPDATE_SUFFIX-$TIME_SUFFIX" \
"${{ inputs.disk_prefix }}-${SHORT_GITHUB_REF}-${{ env.GITHUB_SHA_SHORT }}-v${{ env.STATE_VERSION }}-${{ env.NETWORK }}-${{ inputs.disk_suffix }}${UPDATE_SUFFIX}-${TIME_SUFFIX}" \
--force \
--source-disk=${{ inputs.test_id }}-${{ env.GITHUB_SHA_SHORT }} \
--source-disk-zone=${{ env.ZONE }} \
Expand Down