Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

df.drop(cs.ends_with()) returning ColumnNotFoundError when columns exist in 1.4.0 (worked in 1.3.0) #18016

Closed
2 tasks done
ATL2001 opened this issue Aug 2, 2024 · 4 comments · Fixed by #18034
Closed
2 tasks done
Assignees
Labels
accepted Ready for implementation bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars regression Issue introduced by a new release

Comments

@ATL2001
Copy link

ATL2001 commented Aug 2, 2024

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import os
os.environ["POLARS_VERBOSE"] = "1"

import polars as pl
from polars import selectors as cs 

print(pl.__version__)
df = pl.DataFrame([["x"], [1]], schema={"foo":pl.String, "foo_right":pl.Int8})
print(df)
print(df.drop(
    cs.ends_with("_right")
))

Log output

ColumnNotFoundError: "^.*(_right)$" not found

Issue description

version 1.4.0 is telling me it can't find a column that exists when I use cs.ends_with() the same code produces the expected result in 1.3.0

Expected behavior

in version 1.3.0 the same code resulted in the expected result of just the "foo" column

1.3.0
shape: (1, 2)
+-----+-----------+
| foo | foo_right |
| --- | ---       |
| str | i8        |
+=================+
| x   | 1         |
+-----+-----------+
shape: (1, 1)
+-----+
| foo |
| --- |
| str |
+=====+
| x   |
+-----+

Installed versions

--------Version info---------
Polars:               1.4.0
Index type:           UInt32
Platform:             Windows-10-10.0.19045-SP0
Python:               3.9.16 [MSC v.1931 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          2.0.0
connectorx:           0.3.2
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               2022.7.1
gevent:               <not installed>
great_tables:         <not installed>
hvplot:               <not installed>
matplotlib:           3.6.0
nest_asyncio:         1.5.5
numpy:                1.20.1
openpyxl:             3.0.10
pandas:               1.4.4
pyarrow:              1.0.1
pydantic:             2.7.1
pyiceberg:            <not installed>
sqlalchemy:           1.4.39
torch:                <not installed>
xlsx2csv:             0.8.2
xlsxwriter:           3.2.0
@ATL2001 ATL2001 added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Aug 2, 2024
@cmdlineluser
Copy link
Contributor

cmdlineluser commented Aug 2, 2024

It seems to be specific to .drop

>>> df.select(~cs.ends_with("_right"))
shape: (1, 1)
┌─────┐
│ foo │
│ --- │
│ str │
╞═════╡
│ x   │
└─────┘

There was a .drop fix pushed for #18008 earlier which may be related?


Update: Just confirmed that it worked up to that commit.

Perhaps @ritchie46 can take a look.

@ATL2001 ATL2001 changed the title cs.ends_with returning ColumnNotFoundError when columns exist in 1.4.0 (worked in 1.3.0) df.drop(cs.ends_with()) returning ColumnNotFoundError when columns exist in 1.4.0 (worked in 1.3.0) Aug 2, 2024
@ATL2001
Copy link
Author

ATL2001 commented Aug 2, 2024

good call, I updated the title with the df.drop()

@kszlim
Copy link
Contributor

kszlim commented Aug 2, 2024

I'm experiencing this too, affects cs.starts_with and cs.contains as well

@alexander-beedie alexander-beedie added the regression Issue introduced by a new release label Aug 3, 2024
@ATL2001
Copy link
Author

ATL2001 commented Aug 4, 2024

As always, Thanks Ritchie!

@c-peters c-peters added the accepted Ready for implementation label Aug 5, 2024
@c-peters c-peters added this to Backlog Aug 5, 2024
@c-peters c-peters moved this to Done in Backlog Aug 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted Ready for implementation bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars regression Issue introduced by a new release
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

6 participants