-
Notifications
You must be signed in to change notification settings - Fork 603
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add workarounds for duckdb MAP bugs #8632
Comments
Thanks! Looks like you've gone down a map 🐰 🕳️ I think it's reasonable to add these workarounds for now. |
yeah seriously talk about a 🐰 🕳️ . I was convinced for hours it was bugs in my code wheeeeee here is what I currently use. I have unit tests for functions that use these, but I don't have unit tests for these as individuals. def map_keys(m: ir.MapValue) -> ir.ArrayValue:
"""workaround for https://github.com/duckdb/duckdb/issues/11116"""
normal = m.keys()
null = ibis.literal(None, type=dt.Array(value_type=m.type().key_type))
return m.isnull().ifelse(null, normal)
def map_values(m: ir.MapValue) -> ir.ArrayValue:
"""workaround for https://github.com/duckdb/duckdb/issues/11116"""
normal = m.values()
null = ibis.literal(None, type=dt.Array(value_type=m.type().value_type))
return m.isnull().ifelse(null, normal)
def map_(keys: ir.ArrayValue, values: ir.ArrayValue) -> ir.MapValue:
"""workaround for https://github.com/duckdb/duckdb/issues/11115"""
either_null = keys.isnull() | values.isnull()
regular = ibis.map(keys, values)
null = ibis.literal(
None,
type=dt.Map(
key_type=keys.type().value_type, value_type=values.type().value_type
),
)
return either_null.ifelse(null, regular) |
Yeah, this would be good to get in for 9.0. @NickCrews do you have the bandwidth to PR this in? We probably do want unit tests for them, and probably also some xfailed tests that will start to XPASS when the upstream issues are fixed. |
partially addresses ibis-project#8632
partially addresses ibis-project#8632
partially addresses ibis-project#8632
In that PR I addressed duckdb/duckdb#11115, which
Once that is in then I can tackle duckdb/duckdb#11116, which I think
|
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
partially addresses ibis-project#8632 postgres started failing these new test too, so had to adjust that visit_Map()
I'll work on getting the rest of the map family of functions behaving consistently here. |
Closes #8632. Note that of the backends that support nullable maps, DuckDB is the only one whose `NULL` handling semantics differ. Other backends that support nullable maps and are implemented in Ibis and tested in CI: - Postgres (only maps of string -> string via the postgres `hstore` extension) - PySpark - Snowflake - Trino BREAKING CHANGE: Calling the `get` or `contains` method on `NULL` map values now returns `NULL`. Use `coalesce(map.get(...), default)` or `coalesce(map.contains(), False)` to get the previous behavior.
Is your feature request related to a problem?
see
duckdb/duckdb#11115
and
duckdb/duckdb#11116
This might be out of scope, and we should just wait for duckdb to fix these upstream, but we could add some workarounds for these bugs in ibis. I am running into these and am having to implement my own little wrappers, eg
feel free to close this out if you don't want to take this on (I don't blame you if you don't 😉 )
Describe the solution you'd like
I'm not sure where in the stack these fixes would need to go, at compile time, or just when creating expressions.
What version of ibis are you running?
main
What backend(s) are you using, if any?
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: