-
Notifications
You must be signed in to change notification settings - Fork 28.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-22973][SQL] Fix incorrect results of Casting Map to String #20166
Conversation
Test build #85726 has finished for PR 20166 at commit
|
Test build #85727 has finished for PR 20166 at commit
|
if (map.numElements > 0) { | ||
val keyToUTF8String = castToString(kt) | ||
val valueToUTF8String = castToString(vt) | ||
builder.append(keyToUTF8String(map.keyArray().get(0, kt)).asInstanceOf[UTF8String]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
map.keyArray()
and map.valueArray
appear many times, we can create 2 local variables for them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok
LGTM except one minor comment |
BTW, the current
If we cast them before prints, we could get more simpler forms like;
I'm not sure through, is this acceptable? (Probably, we might need to add a option to keep the old behaviour) |
Test build #85751 has finished for PR 20166 at commit
|
I think this is a bug, in
Which means we do want to show strings like Anyway let's fix in another PR, I'm merging this PR first |
merging to master/2.3! |
## What changes were proposed in this pull request? This pr fixed the issue when casting maps into strings; ``` scala> Seq(Map(1 -> "a", 2 -> "b")).toDF("a").write.saveAsTable("t") scala> sql("SELECT cast(a as String) FROM t").show(false) +----------------------------------------------------------------+ |a | +----------------------------------------------------------------+ |org.apache.spark.sql.catalyst.expressions.UnsafeMapData38bdd75d| +----------------------------------------------------------------+ ``` This pr modified the result into; ``` +----------------+ |a | +----------------+ |[1 -> a, 2 -> b]| +----------------+ ``` ## How was this patch tested? Added tests in `CastSuite`. Author: Takeshi Yamamuro <[email protected]> Closes #20166 from maropu/SPARK-22973. (cherry picked from commit 18e9414) Signed-off-by: Wenchen Fan <[email protected]>
ok, I'll fix struct in a next following pr first. |
What changes were proposed in this pull request?
This pr fixed the issue when casting maps into strings;
This pr modified the result into;
How was this patch tested?
Added tests in
CastSuite
.