Skip to content

Commit

Permalink
[SPARK-23147][UI] Fix task page table IndexOutOfBound Exception
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

Stage's task page table will throw an exception when there's no complete tasks. Furthermore, because the `dataSize` doesn't take running tasks into account, so sometimes UI cannot show the running tasks. Besides table will only be displayed when first task is finished according to the default sortColumn("index").

![screen shot 2018-01-18 at 8 50 08 pm](https://user-images.githubusercontent.com/850797/35100052-470b4cae-fc95-11e7-96a2-ad9636e732b3.png)

To reproduce this issue, user could try `sc.parallelize(1 to 20, 20).map { i => Thread.sleep(10000); i }.collect()` or `sc.parallelize(1 to 20, 20).map { i => Thread.sleep((20 - i) * 1000); i }.collect` to reproduce the above issue.

Here propose a solution to fix it. Not sure if it is a right fix, please help to review.

## How was this patch tested?

Manual test.

Author: jerryshao <[email protected]>

Closes #20315 from jerryshao/SPARK-23147.
  • Loading branch information
jerryshao authored and Marcelo Vanzin committed Jan 18, 2018
1 parent 5063b74 commit cf7ee17
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -676,7 +676,7 @@ private[ui] class TaskDataSource(

private var _tasksToShow: Seq[TaskData] = null

override def dataSize: Int = stage.numCompleteTasks + stage.numFailedTasks + stage.numKilledTasks
override def dataSize: Int = stage.numTasks

override def sliceData(from: Int, to: Int): Seq[TaskData] = {
if (_tasksToShow == null) {
Expand Down

0 comments on commit cf7ee17

Please sign in to comment.