From 2feab4ef4f949a4904a58949334269e07bf7d6f0 Mon Sep 17 00:00:00 2001 From: Thomas Graves Date: Tue, 25 Aug 2020 23:05:35 +0900 Subject: [PATCH] [SPARK-32287][TESTS][FOLLOWUP] Add debugging for flaky ExecutorAllocationManagerSuite ### What changes were proposed in this pull request? fixing flaky test in ExecutorAllocationManagerSuite. The issue is that there is a timing issue when we do a reset as to when the numExecutorsToAddPerResourceProfileId gets reset. The fix is to just always set those values back to 1 when we call reset(). ### Why are the changes needed? fixing flaky test in ExecutorAllocationManagerSuite ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? ran the unit test via this PR a bunch of times and the fix seems to be working. Closes #29508 from tgravescs/debugExecAllocTest. Authored-by: Thomas Graves Signed-off-by: HyukjinKwon --- .../scala/org/apache/spark/ExecutorAllocationManager.scala | 3 +++ 1 file changed, 3 insertions(+) diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala index d27ee784ed5a2..c2989314262c5 100644 --- a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala +++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala @@ -279,6 +279,9 @@ private[spark] class ExecutorAllocationManager( numExecutorsTargetPerResourceProfileId.keys.foreach { rpId => numExecutorsTargetPerResourceProfileId(rpId) = initialNumExecutors } + numExecutorsToAddPerResourceProfileId.keys.foreach { rpId => + numExecutorsToAddPerResourceProfileId(rpId) = 1 + } executorMonitor.reset() }