Support writing hive style partitioned files in COPY command

[alamb]

Is your feature request related to a problem or challenge?

A user asked on ASF Slack: https://the-asf.slack.com/archives/C04RJ0C85UZ/p1702248979379239

Does the COPY command support creating parquet files that are partitioned using hive style partitioning?

The usecase is creating Hive-sty;e partitioned datasets (e.g as described here)

DataFusion does not support this today, but you can use an external table like this https://github.com/apache/arrow-datafusion/blob/93b21bdcd3d465ed78b610b54edf1418a47fc497/datafusion/sqllogictest/test_files/insert.slt#L45-L57

Describe the solution you'd like

@devinjdangelo notes that

The COPY statement does not have a built in PARTITION BY clause in its syntax currently, but we could support syntax like:

COPY table to 'folder/location' (format parquet, partition_by year)

which is the same syntax that duckdb supports for this.

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

I think this is a relative good project for intermediate contributors. It could be done in a few PRs as all the underlying code exists and we already have an example of writing to partitioned datasets, implementing this PR would be a matter of hooking up the APIs correctly

There are also already good examples of copy tests in https://github.com/apache/arrow-datafusion/blob/93b21bdcd3d465ed78b610b54edf1418a47fc497/datafusion/sqllogictest/test_files/copy.slt that can be extended

[Veeupup]

hi @alamb Can I try this issue? It seems very interesting!

[alamb]

hi @alamb Can I try this issue? It seems very interesting!

Can't wait to see what you come up wtih @Veeupup 🚀

[JacobOgle]

@Veeupup are you still working on this?

[Veeupup]

@JacobOgle Hi, sorry then! I have been a little busy these days and I'll start it lately ~