Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testnet-automation: Don't clobber testnet prefix #12527

Closed
wants to merge 1 commit into from
Closed

testnet-automation: Don't clobber testnet prefix #12527

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions system-test/testnet-automation.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ $(eval echo "$@")"
)

execution_step "Deleting Testnet"
"${REPO_ROOT}"/net/"${CLOUD_PROVIDER}".sh delete -p "${TESTNET_TAG}"
"${REPO_ROOT}"/net/"${CLOUD_PROVIDER}".sh delete -p "${testnetPrefix}"

}
trap 'cleanup_testnet $BASH_COMMAND' EXIT
Expand All @@ -63,7 +63,7 @@ function launch_testnet() {
-d pd-ssd \
-n "$NUMBER_OF_VALIDATOR_NODES" -c "$NUMBER_OF_CLIENT_NODES" \
$maybeCustomMachineType "$VALIDATOR_NODE_MACHINE_TYPE" $maybeEnableGpu \
-p "$TESTNET_TAG" $maybeCreateAllowBootFailures $maybePublicIpAddresses \
-p "$testnetPrefix" $maybeCreateAllowBootFailures $maybePublicIpAddresses \
${TESTNET_CLOUD_ZONES[@]/#/"-z "} \
--self-destruct-hours 0 \
${ADDITIONAL_FLAGS[@]/#/" "}
Expand All @@ -74,7 +74,7 @@ function launch_testnet() {
"${REPO_ROOT}"/net/ec2.sh create \
-n "$NUMBER_OF_VALIDATOR_NODES" -c "$NUMBER_OF_CLIENT_NODES" \
$maybeCustomMachineType "$VALIDATOR_NODE_MACHINE_TYPE" $maybeEnableGpu \
-p "$TESTNET_TAG" $maybeCreateAllowBootFailures $maybePublicIpAddresses \
-p "$testnetPrefix" $maybeCreateAllowBootFailures $maybePublicIpAddresses \
${TESTNET_CLOUD_ZONES[@]/#/"-z "} \
${ADDITIONAL_FLAGS[@]/#/" "}
;;
Expand All @@ -84,7 +84,7 @@ function launch_testnet() {
"${REPO_ROOT}"/net/azure.sh create \
-n "$NUMBER_OF_VALIDATOR_NODES" -c "$NUMBER_OF_CLIENT_NODES" \
$maybeCustomMachineType "$VALIDATOR_NODE_MACHINE_TYPE" $maybeEnableGpu \
-p "$TESTNET_TAG" $maybeCreateAllowBootFailures $maybePublicIpAddresses \
-p "$testnetPrefix" $maybeCreateAllowBootFailures $maybePublicIpAddresses \
${TESTNET_CLOUD_ZONES[@]/#/"-z "} \
${ADDITIONAL_FLAGS[@]/#/" "}
;;
Expand All @@ -94,7 +94,7 @@ function launch_testnet() {
# shellcheck disable=SC2086
"${REPO_ROOT}"/net/colo.sh create \
-n "$NUMBER_OF_VALIDATOR_NODES" -c "$NUMBER_OF_CLIENT_NODES" $maybeEnableGpu \
-p "$TESTNET_TAG" $maybePublicIpAddresses --dedicated \
-p "$testnetPrefix" $maybePublicIpAddresses --dedicated \
${ADDITIONAL_FLAGS[@]/#/" "}
;;
*)
Expand Down Expand Up @@ -223,6 +223,9 @@ if [[ -z $NUMBER_OF_VALIDATOR_NODES ]]; then
exit 1
fi

dbName="$TESTNET_TAG"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So won't clobber the testnets, but you will have overlapping writes to the same db in this case if two hosts are running parallel networks with the same TESTNET_TAG

Copy link
Contributor

@danpaul000 danpaul000 Sep 29, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll need to update init-metrics.sh and the call to it here

solana/system-test/testnet-automation.sh

Line 106 in 13ff6b2

"${REPO_ROOT}"/net/init-metrics.sh -e
to support the dynamic creation of new db's for this to work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah... yeah... are we worried about having a mess of DBs?

Would be nice if we could have just the one, then generate a UUID for each test run that's tagged to metrics, but I suspect that's a lot of plumbing elsewhere

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could set custom retention policies for dynamically-created DBs so they don't stick around for too long. The slackbot already creates a custom URL for each test that's run so it displays the db in question only for the time during which the test was running. If something simple like that works for you, that's fine. The bigger problem will come if/when you are having nodes from unrelated testnets writing to the same db at the same time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'll look into that this evening and report back

testnetPrefix="${TESTNET_TAG}-$(hostname)"

startGpuMode="off"
if [[ -z $ENABLE_GPU ]]; then
ENABLE_GPU=false
Expand All @@ -242,7 +245,7 @@ if [[ -z $SOLANA_METRICS_CONFIG ]]; then
echo SOLANA_METRICS_PARTIAL_CONFIG not defined
exit 1
fi
export SOLANA_METRICS_CONFIG="db=$TESTNET_TAG,host=$INFLUX_HOST,$SOLANA_METRICS_PARTIAL_CONFIG"
export SOLANA_METRICS_CONFIG="db=$dbName,host=$INFLUX_HOST,$SOLANA_METRICS_PARTIAL_CONFIG"
fi
echo "SOLANA_METRICS_CONFIG: $SOLANA_METRICS_CONFIG"

Expand Down