Fix Index._with_new_scol to update its anchor. #1334
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1334 +/- ##
=========================================
Coverage ? 95.22%
=========================================
Files ? 34
Lines ? 7430
Branches ? 0
=========================================
Hits ? 7075
Misses ? 355
Partials ? 0
Continue to review full report at Codecov.
|
| internal = _InternalFrame( | ||
| spark_frame=sdf, index_map=OrderedDict(zip(sdf.columns, self._internal.index_names)) | ||
| ) | ||
| return DataFrame(internal).index |
There was a problem hiding this comment.
Hm .. I think it will trigger the operations on different DataFrames among the operations in Index .. e.g.)
import databricks.koalas as ks
index = ks.range(100).index
index + index + indexThere looks another issue in compute.ops_on_diff_frames ... operations against the indexes from different frames are not supported ..
There was a problem hiding this comment.
Good point.
The way we manage index Spark columns is different from Series, so we need further refactoring in _InternalFrame, e.g., we should manage them as Spark columns instead of Spark column names. The way in Series is not good enough yet, though.
Actually creating new DataFrame as an anchor is better than now compared to pandas behavior:
>>> pdf = pd.DataFrame({'a':[1,2,3]})
>>> pidx = pdf.index
>>> pidx.name = 'i'
>>> pdf
a
i
0 1
1 2
2 3
>>> (pidx + pidx).name = 'ii'
>>> pdf
a
i
0 1
1 2
2 3When operating pidx + pidx, the connection to the original pdf seems dropped.
Whereas:
>>> kdf = ks.DataFrame({'a':[1,2,3]})
>>> kidx = kdf.index
>>> kidx.name = 'i'
>>> kdf
a
i
0 1
1 2
2 3
>>> (kidx + kidx).name = 'ii'
>>> kdf
a
ii
0 1
1 2
2 3It updates index name even after kidx + kidx.
As discussed at #1328 (comment), we can consolidate the
Index.repeat()andMultiIndex.repeat()by updating its anchor with a new scol.Also resolves #1190 and other functions which were mistakenly using
index_scolsinIndex, whose usage is valid now.