[SPARK-12987][SQL]Fixing the name resolution in drop column #11009
Conversation
|
ok to test |
| @@ -1257,7 +1257,8 @@ class DataFrame private[sql]( | |||
| def drop(colNames: String*): DataFrame = { | |||
| val resolver = sqlContext.analyzer.resolver | |||
| val remainingCols = | |||
| schema.filter(f => colNames.forall(n => !resolver(f.name, n))).map(f => Column(f.name)) | |||
| schema.filter(f => colNames.forall(n => !resolver(f.name, n))) | |||
There was a problem hiding this comment.
super nit: i would consider breaking before the filter also if you have to break for the map. otherwise the filter looks special when really you are just chaining a bunch of operations.
|
Test build #50515 has finished for PR 11009 at commit
|
|
is it duplicated with #10943? |
|
The two jiras seem to be identical; but the fixes are different. |
|
@cloud-fan Yeah.. 10943 also fixes the same issue. @yzhou2001 Actually the original fix was sort of similar to fix in this PR except that it also considered the drop(Column) interface. I have been trying to improve the fix based on suggestion from Wenchen :-) |
|
@marmbrus @cloud-fan @dilipbiswal @yzhou2001 : it seems this is the duplicate of [SPARK-12988][SQL] Can't drop columns that contain dots #10943, I will close this PR. thanks. |
|
Actually, I'd really like to include this in 1.6.1 and this is a nice small fix. Could you reopen? |
|
@marmbrus @cloud-fan @dilipbiswal @yzhou2001 : I have change the code based on Michael's comments, can you help review it again? Not sure why the first test failed, I run the sql test locally, it passed. [info] Run completed in 3 minutes, 23 seconds. |
|
Test build #50686 has finished for PR 11009 at commit
|
|
Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket. |
@marmbrus @cloud-fan @thomas @jayadevan : Hello All: Can you help review this code fix?
This problem is coming from drop column, after we drop off the old column, construct the new dataframe from the remaining columns.
the new dataframe is using the Schema information to construct the column name, in this case, the name is string ‘a.c’. Since it is from Schema, we should take the name as it is, we should not do any parsing on the name.