Spark SQL merge small files to big files Update InsertIntoHiveTable.scala

[wuzhilon]

Merge hive small files into large files, support orc and text data table storage format

What changes were proposed in this pull request?

we have more spark SQL partitions tables ,table partition have more small files。Causing the cluster hdfs a lot of pressure, we use this feature to merge small files, to the cluster down to 1/10 hdfs pressure
##test this function
you can use SQL: INSERT INTO TABLE AS SELECT XXXX

[AmplabJenkins]

Can one of the admins verify this patch?

[gatorsmile]

Can you just repartition your data before writing to the file?

[wuzhilon]

I am trying to, now the problem is unable to know the number of data MB, I can only get the number of data, and then re-coarse-grained.

[HyukjinKwon]

@wuzhilon, could you explain why it is problematic if we just repartition? I didn't understand

the problem is unable to know the number of data MB, I can only get the number of data, and then re-coarse-grained.

I think the approach here is quite poorly implemented and we should close. This at least looks listing files which is quite costly on S3.