Implement Index.dropna & MultiIndex.dropna

[itholic]

There is functionality pandas.Index.dropna (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Index.dropna.html#pandas.Index.dropna)

support it also for Koalas same as pandas.

>>> df = ks.DataFrame([[1, 2], [4, 5], [7, 8]],
...                   index=['cobra', 'viper', None],
...                   columns=['max_speed', 'shield'])
>>> df
       max_speed  shield
cobra          1       2
viper          4       5
NaN            7       8

>>> df.index.dropna()
Index(['cobra', 'viper'], dtype='object')

>>> df.to_pandas().index.dropna()
Index(['cobra', 'viper'], dtype='object')

[itholic]

While i'm here, i'm going to make this possible to support MultiIndex also.
Above resolved:

>>> midx = pd.MultiIndex([['lama', 'cow', 'falcon'],
...                       [None, 'weight', 'length']],
...                      [[0, 1, 1, 1, 1, 1, 2, 2, 2],
...                       [0, 1, 1, 0, 1, 2, 1, 1, 2]])
>>> s = ks.Series([45, 200, 1.2, 30, 250, 1.5, 320, 1, None],
...               index=midx)
>>> s
lama    NaN        45.0
cow     weight    200.0
        weight      1.2
        NaN        30.0
        weight    250.0
        length      1.5
falcon  weight    320.0
        weight      1.0
        length      NaN
Name: 0, dtype: float64

>>> s.index.dropna()
MultiIndex([(   'cow', 'weight'),
            (   'cow', 'weight'),
            (   'cow', 'weight'),
            (   'cow', 'length'),
            ('falcon', 'weight'),
            ('falcon', 'weight'),
            ('falcon', 'length')],
           )

>>> s.to_pandas().index.dropna()
MultiIndex([(   'cow', 'weight'),
            (   'cow', 'weight'),
            (   'cow', 'weight'),
            (   'cow', 'length'),
            ('falcon', 'weight'),
            ('falcon', 'weight'),
            ('falcon', 'length')],
           )

[codecov-io]

Codecov Report

Merging #938 into master will increase coverage by 0.68%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master    #938      +/-   ##
=========================================
+ Coverage   94.52%   95.2%   +0.68%     
=========================================
  Files          34      34              
  Lines        6466    6868     +402     
=========================================
+ Hits         6112    6539     +427     
+ Misses        354     329      -25

Impacted Files	Coverage Δ
databricks/koalas/missing/indexes.py	100% <ø> (ø)	⬆️
databricks/koalas/indexes.py	96.17% <100%> (-0.28%)	⬇️
databricks/koalas/usage_logging/__init__.py	97.27% <0%> (ø)	⬆️
databricks/koalas/usage_logging/usage_logger.py	100% <0%> (ø)	⬆️
databricks/koalas/__init__.py	87.23% <0%> (+1.86%)	⬆️
databricks/conftest.py	98% <0%> (+0.12%)	⬆️
databricks/koalas/utils.py	98.17% <0%> (+0.14%)	⬆️
databricks/koalas/frame.py	96.74% <0%> (+0.72%)	⬆️
databricks/koalas/groupby.py	91.39% <0%> (ø)	⬆️
databricks/koalas/namespace.py	87.34% <0%> (+0.51%)	⬆️
... and 14 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f53aeb...b58b49e. Read the comment docs.

[HyukjinKwon]

Looks good otherwise.

[HyukjinKwon]

I think we can do as below:

sdf = kdf._internal.sdf.select(index_scols).dropna()
internal = kdf._internal.copy(sdf=sdf, column_index=[], ...)
kdf = DataFrame(internal)
return Index(kdf) if type(self) == Index else MultiIndex(kdf)

[itholic]

thanks for the comment!
i fixed with using _InternalFrame rather than using copy since i think we can't copy internal because sdf has only index columns while kdf has also data columns

[softagram-bot]

Softagram Impact Report for pull/938 (head commit: b58b49e)

⭐ Change Overview

(Open in Softagram Desktop for full details)

📄 Full report

• Permalink: Full report for pull/938

Impact Report explained. Give feedback on this report to [email protected]