fix: querying on date partitions (fixes #1445) #1481
Conversation
ChewingGlass
requested review from
houqp,
xianwill,
wjones127,
fvaleye,
roeap,
rtyler and
mosyp
as code owners
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
ChewingGlass
changed the title
| @@ -844,7 +844,7 @@ async fn test_issue_1292_datafusion_sql_projection() -> Result<()> { | |||
| #[tokio::test] | |||
| async fn test_issue_1291_datafusion_sql_partitioned_data() -> Result<()> { | |||
| let ctx = SessionContext::new(); | |||
| let table = deltalake::open_table("./tests/data/http_requests") | |||
| let table = deltalake::open_table("./tests/data/issue_1291") | |||
There was a problem hiding this comment.
Instead of committing a table to the repo, could we write one in a temp directory? I think we want to reserve committing tables to ones that have to be generated by outside tools like Spark.
There was a problem hiding this comment.
@ChewingGlass you willing to update this part soon?
There was a problem hiding this comment.
There seem to be some tests that need to be updated so they no longer expect dictionary encoded columns.
Also need to add the setup where you create a temporary directory and write the partitioned table :)
This results in duplicating the same table for delta-io#1291 and delta-io#1292 but I think the clarity is worth it.
Co-authored-by: Will Jones <[email protected]>
ChewingGlass
force-pushed
the
bugfix/issue-1445
branch
from
2e53a7c
to
2773930
Compare
|
What can I do to help move this along? We're currently pinning on a copy of this PR plus some merges and would very much like to move to a more official version. |
|
@watfordkcf - seems like there are some tests that need fixing. Haven't had a chance to properly review yet, but it seems they are partially related to a table that was previously commited to the (within this PR) and later removed not being created within the tests. Other failures are related to column encodings. On first glance related to how we encode the partitioned columns. |
|
Closing since we merged #1594 |
Description
Fix querying on date partitions
Related Issue(s)