[SPARK-6067][SQL]delete the file with the same name before create the path for HiveFileFormatUtils.getHiveRecordWriter #4980
Conversation
|
I think this patch should test by the Reporter of SPARK-6067 |
|
ok to test |
|
/cc @liancheng |
|
Test build #29636 has finished for PR 4980 at commit
|
| val path = new Path(workPath, getOutputName) | ||
| val fs = path.getFileSystem(conf.value) | ||
| // this judgement is added for SPARK-6067 | ||
| if (fs.exists(path) && path.toUri.toString.indexOf("part-") > 0) { |
There was a problem hiding this comment.
Maybe check for part- before getting a file system or checking if the file exists. This saves RPCs.
Did you mean > 0 in the sense of contains? that's not quite correct if so, since the condition should be >= 0. But then contains would be simpler anyway.
Finally, duplicat -> duplicate
|
thank you @srowen |
|
Test build #30804 has finished for PR 4980 at commit
|
|
Test build #31055 has started for PR 4980 at commit |
baishuo
changed the title
| val path = new Path(workPath, getOutputName) | ||
| val fs = path.getFileSystem(conf.value) | ||
| // this judgement is added for SPARK-6067 | ||
| if (fs.exists(path) && path.toUri.toString.contains("part-")) { |
There was a problem hiding this comment.
As @srowen mentioned, please check "part-" first, and then check existence, to save NameNode RPC.
|
Test build #31228 has finished for PR 4980 at commit
|
|
can this patch be merged？:) |
|
@liancheng can this patch be merged? Is it still relevant? |
|
hi @andrewor14 @liancheng , I think this is a useful patch :) |
|
Seems that this issue has already fixed by #6833? |
|
Shall we close this PR then? |
|
no problem. close it :) @andrewor14 @liancheng |
this is commit for https://issues.apache.org/jira/browse/SPARK-6067