Skip to content

Commit

Permalink
Don't use coalesce for gathering all data to a single partition, as i…
Browse files Browse the repository at this point in the history
…t does not work correctly with mutable rows.
  • Loading branch information
marmbrus authored and liancheng committed Apr 11, 2014
1 parent e36cdd0 commit 1965123
Showing 1 changed file with 8 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,14 @@ case class Exchange(newPartitioning: Partitioning, child: SparkPlan) extends Una
shuffled.map(_._1)

case SinglePartition =>
child.execute().coalesce(1, shuffle = true)
val rdd = child.execute().mapPartitions { iter =>
val mutablePair = new MutablePair[Null, Row]()
iter.map(r => mutablePair.update(null, r))
}
val partitioner = new HashPartitioner(1)
val shuffled = new ShuffledRDD[Null, Row, MutablePair[Null, Row]](rdd, partitioner)
shuffled.setSerializer(new SparkSqlSerializer(new SparkConf(false)))
shuffled.map(_._2)

case _ => sys.error(s"Exchange not implemented for $newPartitioning")
// TODO: Handle BroadcastPartitioning.
Expand Down

0 comments on commit 1965123

Please sign in to comment.