SPARK-1337: Application web UI garbage collects newest stages #320
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Looks good provided it passes tests. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
LGTM |
| stages.trimEnd(toRemove) | ||
| } | ||
| stages.trimStart(toRemove) | ||
| } |
There was a problem hiding this comment.
ahhh darn - will fix
There was a problem hiding this comment.
while you are fixing that, do u mind changing the foreach to remove the extra { ? it is killing my eyes :(
|
@rxin okay updated... look okay now? |
|
lgtm |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Merged this and backported to 0.9. The backport was nontrivial (mostly just backporting the tests) because a lot of the code around the listeners changed. |
Simple fix... Author: Patrick Wendell <[email protected]> Closes #320 from pwendell/stage-clean-up and squashes the following commits: 29be62e [Patrick Wendell] SPARK-1337: Application web UI garbage collects newest stages instead old ones (cherry picked from commit ee6e9e7) Conflicts: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
Remove erroneous FAILED state for killed tasks. Currently, when tasks are killed, the Executor first sends a status update for the task with a "KILLED" state, and then sends a second status update with a "FAILED" state saying that the task failed due to an exception. The second FAILED state is misleading/unncessary, and occurs due to a NonLocalReturnControl Exception that gets thrown due to the way we kill tasks. This commit eliminates that problem. I'm not at all sure that this is the best way to fix this problem, so alternate suggestions welcome. @rxin guessing you're the right person to look at this.
Remove erroneous FAILED state for killed tasks. Currently, when tasks are killed, the Executor first sends a status update for the task with a "KILLED" state, and then sends a second status update with a "FAILED" state saying that the task failed due to an exception. The second FAILED state is misleading/unncessary, and occurs due to a NonLocalReturnControl Exception that gets thrown due to the way we kill tasks. This commit eliminates that problem. I'm not at all sure that this is the best way to fix this problem, so alternate suggestions welcome. @rxin guessing you're the right person to look at this. (cherry picked from commit 0475ca8) Signed-off-by: Reynold Xin <[email protected]>
Simple fix... Author: Patrick Wendell <[email protected]> Closes apache#320 from pwendell/stage-clean-up and squashes the following commits: 29be62e [Patrick Wendell] SPARK-1337: Application web UI garbage collects newest stages instead old ones
Trigger osb-checker job manually
Simple fix...