-
Notifications
You must be signed in to change notification settings - Fork 358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change default write mode to overwrite to be consistent with pandas #1209
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1209 +/- ##
==========================================
- Coverage 95.18% 93.82% -1.37%
==========================================
Files 35 35
Lines 7201 7201
==========================================
- Hits 6854 6756 -98
- Misses 347 445 +98
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I noticed the other but related thing.
self.to_spark().write.saveAsTable(name=name, format=format, mode=mode, | ||
partitionBy=partition_cols, **options) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh,, should this lines be self.to_spark().write.mode(mode).saveAsTable(...)
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And ditto for other to_xxx
related to Spark IO?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Eh, do you mean for consistency? saveAsTable
in Spark seems having mode
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just made the changes back.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nvm, I thought we should explicitly specify mode
with write.mode(mode)
, but it's not needed.
c4952e4
to
7062bb8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Thanks! merging. |
pandas always overwrites. Currently, only JSON and CSV overwrites in Koalas. This PR proposes to consistently overwrite for now.