Support (or ignore) scalar function type signatures

[gforsyth]

The substrait integration is experimental. Support is currently only available on request.

• I have permission to ask for support.

What happens?

The Substrait spec was updated to mandate the inclusion of type signatures in the name of a scalar function definition.

For the Ibis equivalent of this SQL statement:

SELECT
  t0.a,
  t0.b,
  t0.a + t0.a AS c,
  t0.b + t0.b AS d
FROM unbound_table_0 AS t0

where t0.a is int64 and t0.b is int32,

previously, the top of the corresponding substrait plan would've read something like:

version {
  minor_number: 30
  producer: "ibis-substrait"
}
extension_uris {
  extension_uri_anchor: 1
  uri: "https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic.yaml"
}
extensions {
  extension_function {
    extension_uri_reference: 1
    function_anchor: 1
    name: "add"
  }
}
<snip>

but now it should read (and will, shortly, after I fix a bug in ibis-substrait):

version {
  minor_number: 30
  producer: "ibis-substrait"
}
extension_uris {
  extension_uri_anchor: 1
  uri: "https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic.yaml"
}
extensions {
  extension_function {
    extension_uri_reference: 1
    function_anchor: 1
    name: "add:i64_i64"
  }
}
extensions {
  extension_function {
    extension_uri_reference: 1
    function_anchor: 2
    name: "add:i32_i32"
  }
}
<snip>

effectively, there's a function anchor and definition for each pair of types that gets compiled in, so long as the function is defined for those types in the corresponding extension URI file.

Currently, Acero just discards the : and everything after it, and I think that's a reasonable way to restore functionality for the moment. Ibis-substrait won't submit plans with invalid type combinations, but I can't speak for other plan producers.

To Reproduce

In [38]: import ibis

In [39]: from ibis_substrait.compiler.core import SubstraitCompiler

In [40]: compiler = SubstraitCompiler()

In [41]: t = ibis.table({'a': 'int64', 'b': 'int32'}, name='t')

In [42]: expr = t.mutate(c=t.a + t.a)

In [43]: plan = compiler.compile(expr)

In [44]: import duckdb

In [45]: con = duckdb.connect()

In [46]: con.execute("CREATE TABLE t(a INT8, b INT4)")
Out[46]: <duckdb.DuckDBPyConnection at 0x7f5f1a0faa30>

In [47]: con.install_extension("substrait")

In [48]: con.load_extension("substrait")

In [49]: con.from_substrait(plan.SerializeToString())
---------------------------------------------------------------------------
CatalogException                          Traceback (most recent call last)
Cell In[49], line 1
----> 1 con.from_substrait(plan.SerializeToString())

CatalogException: Catalog Error: Scalar Function with name add:i64_i64 does not exist!
Did you mean "add"?

OS:

Ubuntu

Substrait-Extension Version:

whatever ships with 0.8.1

DuckDB Version:

0.8.1

DuckDB Client:

Python

Have you tried this on the latest master branch?

• I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• I agree