fix: replace NaNs with None in some backends when loading from pandas dataframe #9094
Conversation
chloeh13q
changed the title
There was a problem hiding this comment.
Thanks for putting this in @chloeh13q ! I'm on board with trying this out -- I think for mysql, where nan isn't allowed at all, this is a good solution.
For postgres, we should be a little more specific in the conversion.
I also think this could work for mssql and a few other backends that currently can't load a large chunk of the examples.
Sounds good.
I think mssql examples are broken for a different reason, at least on my local; I filed a ticket #9095 yesterday and I'm getting |
…ng from pandas dataframe
…ad of the entire df
|
I'm having trouble spinning up some of the backends so I'm not able to test whether the examples work in these backends. But I can confirm that the examples are now fixed in MySQL and postgres with this PR! |
|
xref #9110 |
cpcloud
added
ux
|
Do we have some tests for this? How do we know this won't regress? |
|
We should enable some of the examples tests for postgres and mysql (I don't think we need to run all of them) |
The palmer penguins data has null valued integers, which when used to create a `memtable`, leads to pandas casting a column to `float` because it reads in the nulls as NaN. We added a fix for this for some backends in ibis-project#9094.
The palmer penguins data has null valued integers, which when used to create a `memtable`, leads to pandas casting a column to `float` because it reads in the nulls as NaN. We added a fix for this for some backends in #9094.
Description of changes
Examples were broken on MySQL backend and PySpark backend when there are null values in a numeric column.
Druid, PySpark, RW don't support examples.
Issues closed
#8792