Skip to content

Commit

Permalink
[SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-depen…
Browse files Browse the repository at this point in the history
…dencies.sh; fix version switching

This patch aims to fix another potential source of flakiness in the `dev/test-dependencies.sh` script.

pwendell's original patch and my version used `$(date +%s | tail -c6)` to generate a suffix to use when installing temporary Spark versions into the local Maven cache, but this value only changes once per second and thus is highly collision-prone when concurrent builds launch on AMPLab Jenkins. In order to reduce the potential for conflicts, this patch updates the script to call Python's random number generator instead.

I also fixed a bug in how we captured the original project version; the bug was causing the exit handler code to fail.

Author: Josh Rosen <[email protected]>

Closes apache#10558 from JoshRosen/build-dep-tests-round-3.
  • Loading branch information
JoshRosen committed Jan 10, 2016
1 parent e1ab044 commit f8b4fdb
Showing 1 changed file with 13 additions and 3 deletions.
16 changes: 13 additions & 3 deletions dev/test-dependencies.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,19 @@ HADOOP_PROFILES=(
# the old version. We need to do this because the `dependency:build-classpath` task needs to
# resolve Spark's internal submodule dependencies.

# See http://stackoverflow.com/a/3545363 for an explanation of this one-liner:
OLD_VERSION=$($MVN help:evaluate -Dexpression=project.version|grep -Ev '(^\[|Download\w+:)')
TEMP_VERSION="spark-$(date +%s | tail -c6)"
# From http://stackoverflow.com/a/26514030
set +e
OLD_VERSION=$($MVN -q \
-Dexec.executable="echo" \
-Dexec.args='${project.version}' \
--non-recursive \
org.codehaus.mojo:exec-maven-plugin:1.3.1:exec)
if [ $? != 0 ]; then
echo -e "Error while getting version string from Maven:\n$OLD_VERSION"
exit 1
fi
set -e
TEMP_VERSION="spark-$(python -S -c "import random; print(random.randrange(100000, 999999))")"

function reset_version {
# Delete the temporary POMs that we wrote to the local Maven repo:
Expand Down

0 comments on commit f8b4fdb

Please sign in to comment.