GH-459: Add Variant logical type annotation #460
Conversation
LogicalTypes.md
Outdated
|
|
||
| * The top level must be a group annotated with `VARIANT` that contains a | ||
| `binary` field named `metadata`, and a `binary` field named `value`. | ||
| * Additional fields which start with `_` (underscore) can be ignored. |
There was a problem hiding this comment.
Why is this needed? None of the other types allow writing columns that should be ignored.
There was a problem hiding this comment.
This was desired in case there were some additional (but redundant) metadata or values we might store, and still allow it to be a valid Variant value (group).
There was a problem hiding this comment.
I don't think that we want to add ignored columns. If we need to update the spec because something is missing, we should just do that directly instead of working around it with unspecified columns that only work in certain proprietary cases.
There was a problem hiding this comment.
I see. I was worried that future evolution could break existing stored Variants, but simply adding a new field with optional or redundant semantics achieves the same compatibility story. This is removed.
| /** | ||
| * Embedded Variant logical type annotation | ||
| */ | ||
| struct VariantType { |
LogicalTypes.md
Outdated
|
|
||
| * The top level must be a group annotated with `VARIANT` that contains a | ||
| `binary` field named `metadata`, and a `binary` field named `value`. | ||
| * Additional fields which start with `_` (underscore) can be ignored. |
There was a problem hiding this comment.
This was desired in case there were some additional (but redundant) metadata or values we might store, and still allow it to be a valid Variant value (group).
|
This looks close to me. I think we just need to fix two things:
|
LogicalTypes.md
Outdated
|
|
||
| * The top level must be a group annotated with `VARIANT` that contains a | ||
| `binary` field named `metadata`, and a `binary` field named `value`. | ||
| * Additional fields which start with `_` (underscore) can be ignored. |
There was a problem hiding this comment.
I see. I was worried that future evolution could break existing stored Variants, but simply adding a new field with optional or redundant semantics achieves the same compatibility story. This is removed.
Rationale for this change
Add a variant logical type.
What changes are included in this PR?
Additions to the types thrift definition, and the description of the logical type. The actual Variant spec documents are unchanged, and will be addressed later in a separate PR.
Closes #459