DataFrame.cache()

[shril]

This PR proposes DataFrame.cache() API.

Resolves #333

[codecov-io]

Codecov Report

Merging #397 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #397      +/-   ##
==========================================
+ Coverage   93.12%   93.15%   +0.02%     
==========================================
  Files          28       28              
  Lines        3448     3462      +14     
==========================================
+ Hits         3211     3225      +14     
  Misses        237      237

Impacted Files	Coverage Δ
databricks/koalas/frame.py	94.86% <100%> (+0.1%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05b55e4...0b3b9ee. Read the comment docs.

[shril]

@HyukjinKwon, I have changed the implementation from using contextmanager to __enter__() and __exit()__ functions.

The reason for the change of implementation is that df = df.cache() was not working in try-finally block. The block used to yield the DataFrame so it could only be used like this -

with df.cache() as cached_df:
    # do_something

But in the current implementation, I am returning the DataFrame, so we can run further operations on it.
I hope this solves the problem. Please have a look.

[shril]

@HyukjinKwon, @ueshin can you please look into my PR?

[ueshin]

I'd prefer not to expose __enter__ and __exit__ in DataFrame directly since we might want to use them in the different case in the future.

How about introducing _CachedDataFrame extending DataFrame and supply __enter__, __exit__, and unpersist?

class DataFrame(...):

    ...

    def cache(self):
        return _CachedDataFrame(self._sdf, ...)

    ...

class _CachedDataFrame(DataFrame):

    def __init__(self, sdf, ...):
        self._cached = sdf.cache()
        super(_CachedDataFrame, self).__init__(self._cached, ...)

    def __enter__(self):
        return self

    def __exit__(self, ...):
        self.unpersist()

    def unpersist(self):
        if self._cached.is_cached:
            self._cached.unpersist()

I might miss something, though.

WDYT? cc @HyukjinKwon

[shril]

Hi @ueshin,
I have made the changes you asked for.
Exposing the DataFrame was not a good decision on my part. Thank you for correcting it.

Please review my changes. I am happy to help regarding any further issues.

[ueshin]

Thanks! I left some comments and otherwise LGTM.
I'd like to hear opinions from @HyukjinKwon, so I'd leave it to him.

[shril]

Hi @ueshin,
I have made the changes you asked for.
Please have a look.

[softagram-bot]

Softagram Impact Report for pull/397 (head commit: d9ef2c7)

⭐ Change Overview

(Open in Softagram Desktop for full details)

📄 Full report

• Permalink: Full report for pull/397

Give feedback on this report to [email protected]

[shril]

cc: @HyukjinKwon, can you please look into this?

[HyukjinKwon]

Sorry for late response. yup, I am taking a look now :).

[HyukjinKwon]

Merged. Thank you @shril and @ueshin