[Feature Request][Spark] Support for categorical/enum data types in delta

[tunayokumus]

Feature request

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Overview

Support for categorical / enum / dictionary data type in delta tables and spark dataframes. This seems to be generally possible with parquet. Example data types: Arrow (Dictionary), pandas (categorical), duckdb (enum).

Motivation

This would be useful to save space for large tables with low-cardinality columns, while also achieving simplicity by not having to create a separate dimension table and a PK - FK relationship just for a couple of value mapping.

Further details

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently.
• Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
• [ X ] No. I cannot contribute this feature at this time.