[SPARK-14557][SQL] Reading textfile (created though CTAS) doesn't work

[kasjain]

What changes were proposed in this pull request?

These changes fix the below broken functionality and does a small performance improvement.

1. Reading the CSV table (created through CTAS query) doesn't work when the pathFilter is provided.
   A bug in HadoopFileReader. Currently, when the pathFilter is provided, it passes the list of filtered files to the "setInputPaths" which wrongly sets the string of incorrectly escaped comma separated files in an array-sequence of size one. This should have been a sequence of size equal to the number of files obtained after filtering. Hence the exception mentioned in the bug.
   ------ FileInputFormat.setInputPaths(jobConf, Seq[Path](new Path%28path%29): _*)

2. Secondly, in this flow, filtering is triggered twice for each file. Once in hadoopTableReader.applyFilterIfApplicable and then again in FileInputFormat.singleThreadedListStatus. This is costly and redundant.

To solve both the issues above, we can just pass the directory path itself even when the pathFilter is enabled.

How was this patch tested?

Integration tests, manual tests

[andrewor14]

ok to test

[SparkQA]

Test build #55741 has finished for PR 12356 at commit 48a598e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

@liancheng @yhuai?

[saucam]

I think we can eliminate applyFilterIfNeeded method as well.

[SparkQA]

Test build #55796 has finished for PR 12356 at commit 6bd529c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[kasjain]

Can any of the admin verify the above fix?

[kasjain]

Resolved the merge conflicts for easy merging

[SparkQA]

Test build #56660 has finished for PR 12356 at commit ca9a160.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

cc @marmbrus

[kasjain]

Resolved the merge conflicts for easy merging

[SparkQA]

Test build #57504 has finished for PR 12356 at commit 5ba453b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Is it possible to write unit tests for this?

[marmbrus]

ok to test

[SparkQA]

Test build #59146 has finished for PR 12356 at commit 5ba453b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[kasjain]

Sure. Let me add the CTAS query in the test suite

[gatorsmile]

@kasjain Could you add a test case? Does it still fail in the latest master?

[gatorsmile]

We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!