Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with ks.merge to Series #818

Merged
merged 7 commits into from
Sep 24, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 22 additions & 21 deletions databricks/koalas/frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -5093,33 +5093,34 @@ def merge(self, right: 'DataFrame', how: str = 'inner',
Parameters
----------
right: Object to merge with.
DataFrame or named Series
how: Type of merge to be performed.
{'left', 'right', 'outer', 'inner'}, default 'inner'

left: use only keys from left frame, similar to a SQL left outer join; preserve key
order.
right: use only keys from right frame, similar to a SQL right outer join; preserve key
order.
outer: use union of keys from both frames, similar to a SQL full outer join; sort keys
lexicographically.
inner: use intersection of keys from both frames, similar to a SQL inner join;
preserve the order of the left keys.
on: Column or index level names to join on. These must be found in both DataFrames. If on
is None and not merging on indexes then this defaults to the intersection of the
left: use only keys from left frame, similar to a SQL left outer join;
preserve key order.
right: use only keys from right frame, similar to a SQL right outer join;
preserve key order.
outer: use union of keys from both frames, similar to a SQL full outer
join; sort keys lexicographically.
inner: use intersection of keys from both frames, similar to a SQL inner
join; preserve the order of the left keys.
on: Column or index level names to join on.
These must be found in both DataFrames. If on is None and not merging on
indexes then this defaults to the intersection of the
columns in both DataFrames.
left_on: Column or index level names to join on in the left DataFrame. Can also
be an array or list of arrays of the length of the left DataFrame.
left_on: Column or index level names to join on in the left DataFrame.
Can also be an array or list of arrays of the length of the left DataFrame.
These arrays are treated as if they are columns.
right_on: Column or index level names to join on in the right DataFrame. Can also
be an array or list of arrays of the length of the right DataFrame.
right_on: Column or index level names to join on in the right DataFrame.
Can also be an array or list of arrays of the length of the right DataFrame.
These arrays are treated as if they are columns.
left_index: Use the index from the left DataFrame as the join key(s). If it is a
MultiIndex, the number of keys in the other DataFrame (either the index or a number of
columns) must match the number of levels.
right_index: Use the index from the right DataFrame as the join key. Same caveats as
left_index.
suffixes: Suffix to apply to overlapping column names in the left and right side,
respectively.
left_index: Use the index from the left DataFrame as the join key(s).
If it is a MultiIndex, the number of keys in the other DataFrame (either the
index or a number of columns) must match the number of levels.
right_index: Use the index from the right DataFrame as the join key.
Same caveats as left_index.
suffixes: Suffix to apply to overlapping column names in the left and right side, respectively.

Returns
-------
Expand Down
35 changes: 21 additions & 14 deletions databricks/koalas/tests/test_dataframe.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

from datetime import date, datetime
import inspect
from distutils.version import LooseVersion

import numpy as np
import pandas as pd
Expand Down Expand Up @@ -767,21 +768,23 @@ def check(op, right_kdf=right_kdf, right_pdf=right_pdf):
suffixes=['_left', '_right']))

# Test Series on the right
check(lambda left, right: left.merge(right), right_ks, right_ps)
check(lambda left, right: left.merge(right, left_on='x', right_on='x'),
right_ks, right_ps)
check(lambda left, right: left.set_index('x').merge(right, left_index=True, right_on='x'),
right_ks, right_ps)

# Test join types with Series
for how in ['inner', 'left', 'right', 'outer']:
check(lambda left, right: left.merge(right, how=how), right_ks, right_ps)
check(lambda left, right: left.merge(right, left_on='x', right_on='x', how=how),
# pd.DataFrame.merge with Series is implemented since version 0.24.0
if LooseVersion(pd.__version__) > LooseVersion("0.24.2"):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: shall we use 0.24.0 if pandas supports since then?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.

check(lambda left, right: left.merge(right), right_ks, right_ps)
check(lambda left, right: left.merge(right, left_on='x', right_on='x'),
right_ks, right_ps)
check(lambda left, right: left.set_index('x').merge(right, left_index=True, right_on='x'),
right_ks, right_ps)

# suffix
check(lambda left, right: left.merge(right, suffixes=['_left', '_right'], how='outer',
left_index=True, right_index=True), right_ks, right_ps)
# Test join types with Series
for how in ['inner', 'left', 'right', 'outer']:
check(lambda left, right: left.merge(right, how=how), right_ks, right_ps)
check(lambda left, right: left.merge(right, left_on='x', right_on='x', how=how),
right_ks, right_ps)

# suffix with Series
check(lambda left, right: left.merge(right, suffixes=['_left', '_right'], how='outer',
left_index=True, right_index=True), right_ks, right_ps)

def test_merge_retains_indices(self):
left_pdf = pd.DataFrame({'A': [0, 1]})
Expand Down Expand Up @@ -1029,11 +1032,15 @@ def test_join(self):

self.assert_eq(join_pdf, join_kdf)

# join with duplicated columns in Series and DataFrame
# join with duplicated columns in Series
with self.assertRaisesRegex(ValueError,
"columns overlap but no suffix specified"):
kdf1.join(ks1, how='outer')
# join with duplicated columns in DataFrame
with self.assertRaisesRegex(ValueError,
"columns overlap but no suffix specified"):
kdf1.join(kdf2, how='outer')

# check `on` parameter
join_pdf = pdf1.join(pdf2.set_index('key'), on='key', lsuffix='_left', rsuffix='_right')
join_pdf.sort_values(by=list(join_pdf.columns), inplace=True)
Expand Down