Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interval index and interval_range #7182

Merged
merged 116 commits into from
Apr 2, 2021
Merged
Show file tree
Hide file tree
Changes from 87 commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
fd6fb9c
interval dtype and tests
marlenezw Dec 11, 2020
c653ac5
changelog
marlenezw Dec 11, 2020
9549f61
Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into i…
marlenezw Dec 14, 2020
4a16266
updated closed parameter and tests.
marlenezw Dec 14, 2020
f843042
adding both and neither parameters.
marlenezw Dec 14, 2020
5e61f1a
updates to accomodate interval dataframes.
marlenezw Dec 16, 2020
82ef308
Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into i…
marlenezw Dec 16, 2020
80d0c40
removing comments and resolving interval_dtype issues.
marlenezw Dec 16, 2020
446e849
Update python/cudf/cudf/core/column/column.py
marlenezw Dec 16, 2020
6b3a9f0
resolving merge conflicts
marlenezw Jan 20, 2021
43ee680
Merge branch 'interval_dtype' of https://github.com/marlenezw/cudf in…
marlenezw Jan 20, 2021
ed2edb1
fixing style issues
marlenezw Jan 20, 2021
11ad3bc
Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into i…
marlenezw Jan 21, 2021
18a82ae
code for intervalindex and interval range
marlenezw Jan 21, 2021
4255b83
fixing style issues.
marlenezw Jan 21, 2021
8da544b
Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into i…
marlenezw Jan 21, 2021
bc6b5b3
Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into i…
marlenezw Jan 22, 2021
1a15565
updates to interval_range
marlenezw Jan 25, 2021
88d3390
fixing merge conflicts.
marlenezw Jan 25, 2021
19135c9
fixing merge conflicts
marlenezw Jan 25, 2021
a6fd058
updates to interval_range after tests.
marlenezw Jan 26, 2021
86e4a20
Merge branch 'branch-0.18' of https://github.com/rapidsai/cudf into i…
marlenezw Jan 26, 2021
48d00a4
Update python/cudf/cudf/core/column/column.py
marlenezw Jan 26, 2021
daeef06
Merge branch 'intervalIndex' of https://github.com/marlenezw/cudf int…
marlenezw Jan 26, 2021
d4ee46a
more tests and changes to range_interval
marlenezw Jan 27, 2021
d88fdb4
added more tests and comments for clarity.
marlenezw Jan 28, 2021
777cdb5
fixing style changes.
marlenezw Jan 28, 2021
a5d192b
fixing merge conflicts
marlenezw Feb 1, 2021
d56ae9a
style changes
marlenezw Feb 1, 2021
e5baeae
updates for branch-0.19
marlenezw Feb 1, 2021
f24f893
adding type annotations.
marlenezw Feb 1, 2021
9914c3a
initial changes
marlenezw Feb 2, 2021
1375e70
changes based on comments.
marlenezw Feb 16, 2021
a6e2cbf
fixing code that caused csv test to fail.
marlenezw Feb 16, 2021
578c54c
style changes.
marlenezw Feb 16, 2021
7d6f40e
fixing merge conflicts.
marlenezw Feb 16, 2021
cca756d
removing changes to changelog.
marlenezw Feb 17, 2021
eb6855b
base changes.
marlenezw Feb 17, 2021
1606292
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Feb 17, 2021
c8e528c
addressing chnages to fame.py and interval.py
marlenezw Feb 17, 2021
0cfe8f9
fixing mypy styling issue.
marlenezw Feb 17, 2021
dba282a
fixing failing index test.
marlenezw Feb 17, 2021
76955cc
Update python/cudf/cudf/core/column/interval.py
marlenezw Feb 18, 2021
4e9ee08
updating from interval dtype changes.
marlenezw Feb 18, 2021
ab788cf
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Feb 18, 2021
7a360d0
adding periods.
marlenezw Feb 18, 2021
444919c
updates to add new param periods and more tests
marlenezw Feb 23, 2021
be4a77a
fixing merge conflicts
marlenezw Feb 23, 2021
4cf229a
fixing mypy style issues.
marlenezw Feb 25, 2021
d16ec1a
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Feb 25, 2021
524709b
fixing mypy failures.
marlenezw Feb 25, 2021
31c9b75
changes to index.py.
marlenezw Feb 25, 2021
1401c11
making code dryer.
marlenezw Feb 25, 2021
9ffa7d2
style changes.
marlenezw Feb 25, 2021
b3f707c
style issue.
marlenezw Feb 25, 2021
45cde0a
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Feb 26, 2021
16d73f6
cleaning up code.
marlenezw Feb 26, 2021
6dc86ec
sytyle.
marlenezw Feb 26, 2021
6764e0b
fixes for mypy.
marlenezw Feb 26, 2021
223f366
cleaning up code.
marlenezw Feb 26, 2021
725847f
changing interval_range to function
marlenezw Mar 2, 2021
2be072d
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 2, 2021
f3245c2
making sure style is ok.
marlenezw Mar 2, 2021
8420555
fixing mypy style issue.
marlenezw Mar 2, 2021
49de389
updates to code.
marlenezw Mar 3, 2021
606bfe1
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 3, 2021
4ea34c3
changes to interval_range and intervalindex
marlenezw Mar 3, 2021
af4ea1d
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 4, 2021
c00ca33
changes to for tests to pass.
marlenezw Mar 5, 2021
819d04a
fixing merge conflicts
marlenezw Mar 5, 2021
6412ab4
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 8, 2021
f6189cf
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 9, 2021
6fd7eb9
cleaning up docs and adding from_breaks method in intervalindex.
marlenezw Mar 9, 2021
92be49f
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 9, 2021
7085def
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 10, 2021
6d062c9
Update python/cudf/cudf/core/index.py
marlenezw Mar 25, 2021
8ec8716
Update python/cudf/cudf/core/index.py
marlenezw Mar 25, 2021
906d82c
Update python/cudf/cudf/core/index.py
marlenezw Mar 25, 2021
37e988b
Update python/cudf/cudf/core/index.py
marlenezw Mar 25, 2021
3a5fd3f
Update python/cudf/cudf/core/index.py
marlenezw Mar 25, 2021
5e38756
Update python/cudf/cudf/core/index.py
marlenezw Mar 25, 2021
4140525
Update python/cudf/cudf/core/index.py
marlenezw Mar 25, 2021
c1cdb93
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 29, 2021
25a9112
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 29, 2021
f38e34b
adding initial updated changes.
marlenezw Mar 29, 2021
dbfcd13
addressing review comments.
marlenezw Mar 30, 2021
121a9be
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 30, 2021
1790abd
updated changes.
marlenezw Mar 31, 2021
a5fa304
Update python/cudf/cudf/core/index.py
marlenezw Mar 31, 2021
191743e
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Mar 31, 2021
8175120
changes for determining bin_edges dtype
marlenezw Mar 31, 2021
edb68ab
removing my example notebook :)
marlenezw Mar 31, 2021
caab20e
style changes
marlenezw Mar 31, 2021
d48e8f0
Merge branch 'intervalIndex' of https://github.com/marlenezw/cudf int…
marlenezw Mar 31, 2021
b66d16c
determining final dtype.
marlenezw Apr 1, 2021
375a00f
slight changes to docs.
marlenezw Apr 1, 2021
a7eddb7
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Apr 1, 2021
a9f6eb0
adding changes to start.
marlenezw Apr 1, 2021
c00b609
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Apr 1, 2021
ecb6245
Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into i…
marlenezw Apr 1, 2021
4b37080
changing periods to int
marlenezw Apr 1, 2021
00ae319
finding common type to get overall type.
marlenezw Apr 1, 2021
9a306a0
figuring out mypy issues.
marlenezw Apr 1, 2021
33e6699
updates to code and style changes.
marlenezw Apr 1, 2021
31dfa59
Adding support for heterogenous data type spec for start, end, freq
isVoid Apr 2, 2021
f5f1453
Update python/cudf/cudf/core/index.py
marlenezw Apr 2, 2021
479b6df
Update python/cudf/cudf/core/index.py
marlenezw Apr 2, 2021
5c192ff
column empty
marlenezw Apr 2, 2021
d3e2204
Merge branch 'intervalIndex' of https://github.com/marlenezw/cudf int…
marlenezw Apr 2, 2021
dd290ae
change use of as_host_type to as_type
isVoid Apr 2, 2021
ba75420
argument data type check logic fix
isVoid Apr 2, 2021
62fe73d
bug fix, coerce periods to int before
isVoid Apr 2, 2021
f4ff52c
adding empty column.
marlenezw Apr 2, 2021
9044d57
Remove list instantiation for any
isVoid Apr 2, 2021
e715c6c
Merge branch 'intervalIndex' of https://github.com/marlenezw/cudf int…
isVoid Apr 2, 2021
603ba75
Update python/cudf/cudf/core/index.py
marlenezw Apr 2, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions python/cudf/cudf/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
from cudf.core import (
NA,
CategoricalIndex,
interval_range,
IntervalIndex,
DataFrame,
DatetimeIndex,
Float32Index,
Expand Down
2 changes: 2 additions & 0 deletions python/cudf/cudf/core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
from cudf.core.dataframe import DataFrame, from_pandas, merge
from cudf.core.index import (
CategoricalIndex,
interval_range,
IntervalIndex,
DatetimeIndex,
Float32Index,
Float64Index,
Expand Down
69 changes: 63 additions & 6 deletions python/cudf/cudf/core/column/column.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
from cudf.core.abc import Serializable
from cudf.core.buffer import Buffer
from cudf.core.dtypes import CategoricalDtype
from cudf.core.dtypes import IntervalDtype
from cudf.utils import ioutils, utils
from cudf.utils.dtypes import (
NUMERIC_TYPES,
Expand Down Expand Up @@ -1044,11 +1045,7 @@ def astype(self, dtype: Dtype, **kwargs) -> ColumnBase:
)
return self
elif is_interval_dtype(self.dtype):
if not self.dtype == dtype:
raise NotImplementedError(
"Casting interval columns not currently supported"
)
return self
return self.as_interval_column(dtype, **kwargs)
elif is_decimal_dtype(dtype):
return self.as_decimal_column(dtype, **kwargs)
elif np.issubdtype(dtype, np.datetime64):
Expand Down Expand Up @@ -1111,6 +1108,11 @@ def as_datetime_column(
) -> "cudf.core.column.DatetimeColumn":
raise NotImplementedError

def as_interval_column(
self, dtype: Dtype, **kwargs
) -> "cudf.core.column.IntervalColumn":
raise NotImplementedError

def as_timedelta_column(
self, dtype: Dtype, **kwargs
) -> "cudf.core.column.TimeDeltaColumn":
Expand Down Expand Up @@ -1624,6 +1626,15 @@ def build_column(
null_count=null_count,
children=children,
)
elif is_interval_dtype(dtype):
return cudf.core.column.IntervalColumn(
dtype=dtype,
mask=mask,
size=size,
offset=offset,
children=children,
null_count=null_count,
)
elif is_struct_dtype(dtype):
if size is None:
raise TypeError("Must specify size")
Expand Down Expand Up @@ -1704,6 +1715,52 @@ def build_categorical_column(
return cast("cudf.core.column.CategoricalColumn", result)


def build_interval_column(
left_col,
right_col,
mask=None,
size=None,
offset=0,
null_count=None,
closed="right",
):
"""
Build an IntervalColumn

Parameters
----------
left_col : Column
Column of values representing the left of the interval
right_col : Column
Column of representing the right of the interval
mask : Buffer
Null mask
size : int, optional
offset : int, optional
closed : {"left", "right", "both", "neither"}, default "right"
Whether the intervals are closed on the left-side, right-side,
both or neither.
"""
left = as_column(left_col)
right = as_column(right_col)
if closed not in {"left", "right", "both", "neither"}:
closed = "right"
if type(left_col) is not list:
dtype = IntervalDtype(left_col.dtype, closed)
else:
dtype = IntervalDtype("int64", closed)
size = len(left)
return build_column(
data=None,
dtype=dtype,
mask=mask,
size=size,
offset=offset,
null_count=null_count,
children=(left, right),
)


def as_column(
arbitrary: Any,
nan_as_null: bool = None,
Expand Down Expand Up @@ -2088,7 +2145,7 @@ def as_column(
data = as_column(sr, nan_as_null=nan_as_null)
elif is_interval_dtype(dtype):
sr = pd.Series(arbitrary, dtype="interval")
data = as_column(sr, nan_as_null=nan_as_null)
data = as_column(sr, nan_as_null=nan_as_null, dtype=dtype)
else:
data = as_column(
_construct_array(arbitrary, dtype),
Expand Down
37 changes: 29 additions & 8 deletions python/cudf/cudf/core/column/interval.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
import pyarrow as pa
import cudf
from cudf.core.column import StructColumn
from cudf.core.dtypes import IntervalDtype
from cudf.utils.dtypes import is_interval_dtype


class IntervalColumn(StructColumn):
Expand Down Expand Up @@ -38,7 +40,7 @@ def closed(self):
def from_arrow(self, data):
new_col = super().from_arrow(data.storage)
size = len(data)
dtype = cudf.core.dtypes.IntervalDtype.from_arrow(data.type)
dtype = IntervalDtype.from_arrow(data.type)
mask = data.buffers()[0]
if mask is not None:
mask = cudf.utils.utils.pa_mask_buffer_to_mask(mask, len(data))
Expand All @@ -60,14 +62,17 @@ def from_arrow(self, data):

def to_arrow(self):
typ = self.dtype.to_arrow()
return pa.ExtensionArray.from_storage(typ, super().to_arrow())
struct_arrow = super().to_arrow()
if len(struct_arrow) == 0:
# struct arrow is pa.struct array with null children types
# we need to make sure its children have non-null type
struct_arrow = pa.array([], typ.storage_type)
return pa.ExtensionArray.from_storage(typ, struct_arrow)

def from_struct_column(self, closed="right"):
return IntervalColumn(
size=self.size,
dtype=cudf.core.dtypes.IntervalDtype(
self.dtype.fields["left"], closed
),
dtype=IntervalDtype(self.dtype.fields["left"], closed),
mask=self.base_mask,
offset=self.offset,
null_count=self.null_count,
Expand All @@ -80,12 +85,28 @@ def copy(self, deep=True):
struct_copy = super().copy(deep=deep)
return IntervalColumn(
size=struct_copy.size,
dtype=cudf.core.dtypes.IntervalDtype(
struct_copy.dtype.fields["left"], closed
),
dtype=IntervalDtype(struct_copy.dtype.fields["left"], closed),
mask=struct_copy.base_mask,
offset=struct_copy.offset,
null_count=struct_copy.null_count,
children=struct_copy.base_children,
closed=closed,
)

def as_interval_column(self, dtype, **kwargs):
if is_interval_dtype(dtype):
# a user can directly input the string `interval` as the dtype
# when creating an interval series or interval dataframe
if dtype == "interval":
dtype = IntervalDtype(self.dtype.fields["left"], self.closed)
return IntervalColumn(
size=self.size,
dtype=dtype,
mask=self.mask,
offset=self.offset,
null_count=self.null_count,
children=self.children,
closed=dtype.closed,
)
else:
raise ValueError("dtype must be IntervalDtype")
Loading