Skip to content

Commit

Permalink
Merge pull request alteryx#236 from pwendell/shuffle-docs
Browse files Browse the repository at this point in the history
Adding disclaimer for shuffle file consolidation
  • Loading branch information
pwendell committed Dec 7, 2013
2 parents e5d5728 + b9451ac commit 1b38f5f
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -331,7 +331,7 @@ Apart from these, the following properties are also available, and may be useful
<td>spark.shuffle.consolidateFiles</td>
<td>false</td>
<td>
If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance if you run shuffles with large numbers of reduce tasks.
If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks. It is reccomended to set this to "true" when using ext4 or xfs filesystems. On ext3, this option might degrade performance on machines with many (>8) cores due to filesystem limitations.
</td>
</tr>
<tr>
Expand Down

0 comments on commit 1b38f5f

Please sign in to comment.