Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoC: Add Variant type support in Iceberg #11201

Closed
wants to merge 8 commits into from
Closed

Conversation

aihuaxu
Copy link
Contributor

@aihuaxu aihuaxu commented Sep 24, 2024

This is to prototype the Variant type support in Iceberg. Spark is able to read and write Variant data to Iceberg table. Some of the changes e.g., Spark 3.5 should be new Spark 4.0, but this is to show the changes.

This PR includes:

  • Add Variant data type in Iceberg API.
  • Iceberg Parquet module to map Variant type to a struct type with Value and Metadata fields.
  • Iceberg Spark module to support read and write Iceberg Variant.

I will try to break the changes into multiple PRs - API, Parquet and Spark changes.
Fix: #11178

@aihuaxu aihuaxu changed the title api: Add Variant type support in Iceberg PoC: Add Variant type support in Iceberg Oct 3, 2024
Copy link

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Nov 14, 2024
Copy link

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Nov 21, 2024
@aihuaxu aihuaxu mentioned this pull request Dec 2, 2024
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Proposal: add Variant type to iceberg
2 participants