DeltaTable DataFusion TableProvider does not support filter pushdown

[alexwilcoxson-rel]

Environment

Delta-rs version: 0.6

Binding: Rust

Environment:

• Cloud provider: Azure
• OS: macOS
• Other: M1

---

Bug

What happened:
When executing a DataFusion SQL/DataFrame with DeltaTable TableProvider, filters are not pushed down into DeltaTable scan.

What you expected to happen:
Filter pushdown to be supported on DeltaTable so files can be pruned (see related issue #1063)

How to reproduce it:
Run the following snippet with attached table data.zip

    let table = Arc::new(open_table("./data/table").await?);
    let ctx = SessionContext::new();
    let df = ctx.read_table(table)?;
    let df = df.filter(col("n").eq(lit(1 as i64)))?;
    let logical_plan = df.to_logical_plan()?;
    println!("{:?}", logical_plan);

Notice the TableScan has no filters and thus no pruning will occur. If you explain or show the physical plan you'll see the full list of parquet files.

Filter: ?table?.n = Int64(1)
  TableScan: ?table? projection=[n]

More details:
When adding this to the TableProvider impl for DeltaTable

    fn supports_filter_pushdown(
        &self,
        _filter: &Expr,
    ) -> DataFusionResult<TableProviderFilterPushDown> {
        Ok(TableProviderFilterPushDown::Inexact)
    }

the logical plan is

Filter: ?table?.n = Int64(1)
  TableScan: ?table? projection=[n], partial_filters=[?table?.n = Int64(1)]

In conjunction with a fix for #1063 I get proper pruning of files based on statistics in the delta log.

[houqp]

Good catch, do you mind sending a PR combining fixes from both issues?

[alexwilcoxson-rel]

Thanks @roeap for including these fixes. Looks good on my end!