Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add read_json and let to_json use spark.write.json #753

Merged
merged 2 commits into from
Sep 16, 2019

Conversation

HyukjinKwon
Copy link
Member

This PR proposes to use spark.write.json API to enable distributed computation when path is specified. If path is not specified, it just calls pandas' to_json as was.

Also, this PR adds read_json API.

@codecov-io
Copy link

codecov-io commented Sep 6, 2019

Codecov Report

Merging #753 into master will increase coverage by 0.01%.
The diff coverage is 81.81%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #753      +/-   ##
==========================================
+ Coverage   93.91%   93.92%   +0.01%     
==========================================
  Files          32       32              
  Lines        5669     5700      +31     
==========================================
+ Hits         5324     5354      +30     
- Misses        345      346       +1
Impacted Files Coverage Δ
databricks/koalas/namespace.py 81.12% <100%> (-0.88%) ⬇️
databricks/koalas/generic.py 95.91% <77.77%> (-0.09%) ⬇️
databricks/koalas/missing/series.py 100% <0%> (ø) ⬆️
databricks/koalas/plot.py 95.02% <0%> (+0.11%) ⬆️
databricks/koalas/frame.py 96.38% <0%> (+0.14%) ⬆️
databricks/koalas/groupby.py 90.13% <0%> (+0.22%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b12990a...600232b. Read the comment docs.

@softagram-bot
Copy link

Softagram Impact Report for pull/753 (head commit: 600232b)

⭐ Change Overview

Showing the changed files, dependency changes and the impact - click for full size
(Open in Softagram Desktop for full details)

📄 Full report

Impact Report explained. Give feedback on this report to [email protected]

@HyukjinKwon HyukjinKwon merged commit ff5d84c into databricks:master Sep 16, 2019
@HyukjinKwon
Copy link
Member Author

Let me merge this. although it's a breaking change, we should use spark's API whenever possible.

@HyukjinKwon HyukjinKwon deleted the to_json branch November 6, 2019 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants