[SPARK-24366][SQL] Improving of error messages for type converting #21410
Conversation
|
Test build #91034 has finished for PR 21410 at commit
|
|
This is an excellent start and a worthy improvement. Is there a way to identify where in the schema the issue is occurring? For example, when you have a schema with many nested fields, the failing value is helpful but the breadcrumb trail, e.g., |
|
jenkins, retest this, please |
|
Test build #91062 has finished for PR 21410 at commit
|
We can catch the exceptions on each level of schema tree traversal, and show sub-trees in each catch. For example:
I guess in the case of arrays and maps, you want to see indexes and keys. Could you provide concrete example with values and a schema (array, struct, map), and what kind of info the error should contain. Just in case, I would propose to make such improvements in a separate PR. |
|
@gatorsmile Could you look at the PR, please. The changes should help us in trouble shooting of customer's issues. |
| @@ -309,6 +322,9 @@ object CatalystTypeConverters { | |||
| case d: JavaBigDecimal => Decimal(d) | |||
| case d: JavaBigInteger => Decimal(d) | |||
| case d: Decimal => d | |||
| case other => throw new IllegalArgumentException( | |||
| s"The value (${other.toString}) of the type (${other.getClass.getCanonicalName}) " | |||
| + s"cannot be converted to ${dataType.simpleString}") | |||
There was a problem hiding this comment.
Let us use catalogString here?
There was a problem hiding this comment.
All simpleStrings are replaced by catalogStrings
|
LGTM except one minor comment. |
|
Test build #91151 has finished for PR 21410 at commit
|
|
Thanks! Merged to master. |
What changes were proposed in this pull request?
Currently, users are getting the following error messages on type conversions:
The message doesn't give any clues to the users where in the schema the error happened. In this PR, I would like to improve the error message like:
How was this patch tested?
Added tests for converting of wrong values to
struct,map,array,stringanddecimal.