ARRAY_AGG of column of type list ORDER BY column of type non-list

[giacomorebecchi]

Describe the bug

In version 33.0.0, I encountered the following bug (not present in version 32.0.0):
Executing an aggregation with operator ARRAY_AGG() of a column of type list, ORDER BY a column of type non-list, returns the following error:
Execution error: Expects values arguments and/or ordering_values arguments to have same size

To Reproduce

I have an MRE in python:
pip install "pyarrow==14.0.0" "datafusion==33.0.0"

import datetime
import random

import datafusion
import pyarrow as pa
import pyarrow.dataset as pda

N_ROWS = 10_000
N_CARDS = 1_000
N_PRODUCTS = 50

ta = pa.Table.from_pydict(
    {
        "Card.Id": random.choices([str(i) for i in range(N_CARDS)], k=N_ROWS),
        "Date": (datetime.date(2023, (i % 12) + 1, (i % 28) + 1) for i in range(N_ROWS)),
        "Product.Ids": [random.choices([i for i in range(N_PRODUCTS)], k=2) for i in range(N_ROWS)]
    }
)

query = """
SELECT
    "Card.Id"
    , FIRST_VALUE("Product.Ids" ORDER BY "Date")
    , LAST_VALUE("Product.Ids" ORDER BY "Date")
    , ARRAY_AGG("Product.Ids" ORDER BY "Date")
FROM "table"
GROUP BY "Card.Id"
"""

ctx = datafusion.SessionContext()
ctx.register_dataset(name="table",
                     dataset=pda.dataset(ta))
df = ctx.sql(query)
compute_ta = pa.Table.from_batches(df.collect())

Expected behavior

No response

Additional context

No response

[alamb]

Thank you for the report @giacomorebecchi 🙏

@jayzhan211 and @Veeupup and @Weijun-H have done some significant work on various array functions in 34.0.0 (about to be released). This issue may be fixed

[giacomorebecchi]

Great, thanks! Maybe it could be worth adding a test?

[jayzhan211]

@giacomorebecchi I think it is resolved. Feel free to reopen the issue if it fails in release version 36

[jayzhan211]

@giacomorebecchi I think it is resolved. Feel free to reopen the issue if it fails in release version 36

It seems to be 37