[BUG] iloc is not performing exclusively integer based indexing #1459

beckernick · 2019-04-18T20:06:24Z

Describe the bug
As a user, I expect iloc to exclusively perform integer based indexing. Currently, passing a column name to the second argument lets me return that column. The equivalent pandas code fails with a ValueError. This is due to the enforcement of a key-based column access with the second argument if two are passed and row indexer is a slice because this takes us to the DataFrame getitem (noted in #1444 ).

Steps/Code to reproduce bug

import numpy as np
import cudf

M = 1e3

df = cudf.DataFrame()
df['col1'] = np.random.rand(int(M))
df['col2'] = np.random.rand(int(M))

print(df.head())
# Key error on grabbing 0th col
X = df.iloc[:, 'col2']
0       0.878490475957518
1     0.35049769650203455
2    0.048280145683080145
3      0.3844068665321222
4    0.021345988158677165
5      0.5875369468577619
6      0.5527104075530989
7     0.09191724409680513
8     0.12458085283893261
9     0.37619273446670787
[990 more rows]
Name: col2, dtype: float64

Expected behavior
I expect iloc to fail when being based a column name as the second argument.

Environment details (please complete the following information):
Built from source cudf 0.7

Environment

**git*** commit f6ad6de (HEAD -> branch-0.7, origin/branch-0.7, origin/HEAD) Merge: ea880d0 84c3d5b Author: Mark Harris Date: Sun Apr 14 08:43:14 2019 +1000

Merge pull request #1389 from eyalroz/fix-issue-1368

[REVIEW] refactored set_null_count()

OS Information
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Linux cd319eb30a15 4.4.0-134-generic #160-Ubuntu SMP Wed Aug 15 14:58:00 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+

CPU
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
Stepping: 1
CPU MHz: 2737.539
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000
BogoMIPS: 4392.00
Virtualization: VT-x
Hypervisor vendor: vertical
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 51200K
NUMA node0 CPU(s): 0-19,40-59
NUMA node1 CPU(s): 20-39,60-79
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbesyscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts flush_l1d

CMake
/conda/envs/cudf/bin/cmake
cmake version 3.14.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

g++
/usr/bin/g++
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nvcc
/usr/local/cuda/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148

Python
/conda/envs/cudf/bin/python
Python 3.7.3

Environment Variables
PATH : /conda/envs/cudf/bin:/conda/condabin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/conda/bin
LD_LIBRARY_PATH : /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/lib
NUMBAPRO_NVVM : /usr/local/cuda/nvvm/lib64/libnvvm.so
NUMBAPRO_LIBDEVICE : /usr/local/cuda/nvvm/libdevice/
CONDA_PREFIX : /conda/envs/cudf
PYTHON_PATH :

conda packages
/conda/condabin/conda
WARNING: The conda.compat module is deprecated and will be removed in a future release.

packages in environment at /conda/envs/cudf:

The text was updated successfully, but these errors were encountered:

awthomp · 2019-04-29T17:18:18Z

This is impeding my work with Whitewater where I need to handle arbitrary data sizes.

The current method of data handling is to expect a csv file of data with the first N-1 columns consisting of our feature X matrix and the last column consisting of our 'y' observation vector, e.g.

import numpy as np
import cudf

M = 1e3

df = cudf.DataFrame()
df['col1'] = np.random.rand(int(M))
df['col2'] = np.random.rand(int(M))
df['y'] = np.random.rand(int(M))

Where:

X = df.iloc[: , :-1]
y = df.iloc[:, -1]

iloc is currently unable to handle:

grabbing columns by index (it returns the whole dataframe)
grabbing groups of columns with splicing, e.g. df.iloc[:, 0:2]

My workaround in my project is (in this example):

X, y = df.loc[:, ['col1', 'col2']], df.loc[:, 'y']

But this obviously requires a-priori knowledge of the column names.

beckernick · 2019-05-21T23:24:16Z

Resolved by #1622.

beckernick added bug Something isn't working Needs Triage Need team to review and classify labels Apr 18, 2019

kkraus14 added Python Affects Python cuDF API. and removed Needs Triage Need team to review and classify labels Apr 18, 2019

beckernick mentioned this issue May 2, 2019

[BUG] Loc and iloc do not consistently match pandas #1592

Closed

shwina self-assigned this May 8, 2019

shwina mentioned this issue May 14, 2019

[REVIEW] Add Series.loc and DataFrame.loc #1622

Merged

beckernick closed this as completed May 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] iloc is not performing exclusively integer based indexing #1459

[BUG] iloc is not performing exclusively integer based indexing #1459

beckernick commented Apr 18, 2019 •

edited

Loading

packages in environment at /conda/envs/cudf:

Name Version Build Channel

awthomp commented Apr 29, 2019

beckernick commented May 21, 2019

[BUG] iloc is not performing exclusively integer based indexing #1459

[BUG] iloc is not performing exclusively integer based indexing #1459

Comments

beckernick commented Apr 18, 2019 • edited Loading

packages in environment at /conda/envs/cudf:

Name Version Build Channel

awthomp commented Apr 29, 2019

beckernick commented May 21, 2019

beckernick commented Apr 18, 2019 •

edited

Loading