[SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching

[JoshRosen]

This patch aims to fix another potential source of flakiness in the dev/test-dependencies.sh script.

@pwendell's original patch and my version used $(date +%s | tail -c6) to generate a suffix to use when installing temporary Spark versions into the local Maven cache, but this value only changes once per second and thus is highly collision-prone when concurrent builds launch on AMPLab Jenkins. In order to reduce the potential for conflicts, this patch updates the script to call Python's random number generator instead.

I also fixed a bug in how we captured the original project version; the bug was causing the exit handler code to fail.

[JoshRosen]

/cc @rxin

[SparkQA]

Test build #48589 has finished for PR 10558 at commit 8e86e9c.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Test failure was due to Python 3.

[SparkQA]

Test build #48591 has finished for PR 10558 at commit 77a23bf.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48594 has finished for PR 10558 at commit 0a6b120.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2298 has finished for PR 10558 at commit 0a6b120.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

lgtm

On Saturday, January 2, 2016, Apache Spark QA [email protected]
wrote:

Test build #2298 has finished
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2298/consoleFull
for PR 10558 at commit 0a6b120
0a6b120
.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

—
Reply to this email directly or view it on GitHub
#10558 (comment).

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #48595 has finished for PR 10558 at commit 0a6b120.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Aha! It looks like the code for resetting the version has a problem:

+ build/mvn --force -q versions:set '-DnewVersion=

OLD_VERSION isn't being set properly:

+ OLD_VERSION='
[WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency information available

[WARNING] Failed to retrieve plugin descriptor for org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be resolved: Could not find artifact org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in central (https://repo1.maven.org/maven2)
2.0.0-SNAPSHOT'

[JoshRosen]

Pushed a fix for the version issue, so I'm going to run this a few more times then will merge if it's passing.

[SparkQA]

Test build #48596 has finished for PR 10558 at commit ae3d7a3.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #48599 has finished for PR 10558 at commit a2d59e5.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #48615 has finished for PR 10558 at commit a2d59e5.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #48640 has finished for PR 10558 at commit a2d59e5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Merging now.