[SPARK-12257][SQL] Non partitioned insert into a partitioned Hive table doesn't fail #10254
Conversation
|
ok to test |
|
Test build #47494 has finished for PR 10254 at commit
|
| @@ -155,6 +155,11 @@ case class InsertIntoHiveTable( | |||
| val partitionColumns = fileSinkConf.getTableInfo.getProperties.getProperty("partition_columns") | |||
| val partitionColumnNames = Option(partitionColumns).map(_.split("/")).orNull | |||
|
|
|||
| // Validate that partition values are specified for partition columns. | |||
| if (partitionColumnNames != null && partitionColumnNames.size > 0 && partitionSpec.size == 0) { | |||
| throw new SparkException(ErrorMsg.NEED_PARTITION_ERROR.getMsg) | |||
There was a problem hiding this comment.
AnalysisException for anything that is thrown due to an invalid query.
There was a problem hiding this comment.
@marmbrus Thanks. Actually right after the code block i changed, there are a few places where we raise SparkException. So i thought there may be a reason for it and followed it. :-). I will change all those places as well.
|
Test build #47553 has finished for PR 10254 at commit
|
|
@marmbrus Michael, have implemented your comments. Can you please take a look when you get a chance ? |
|
@yhuai could you look at this? |
|
Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Since most of the code in this part has been rewritten, I'm going to push a commit to close it. Feel free to reopen it or create a new one. |
Added a check and raise error on missing partition values.
Currently the check is very similar to hive.