API: Add Variant data type #11324

aihuaxu · 2024-10-15T18:33:18Z

Add Variant data type to API module. We only add limited required type to API for now and we will promote stable methods to API after full implementation to the interface VariantLike later.

Add Variant as a new data type

Fix #11178.

api/src/main/java/org/apache/iceberg/VariantLike.java

api/src/test/java/org/apache/iceberg/util/RandomUtil.java

api/src/test/java/org/apache/iceberg/types/TestSerializableTypes.java

api/src/main/java/org/apache/iceberg/VariantLike.java

api/src/test/java/org/apache/iceberg/TestHelpers.java

api/src/main/java/org/apache/iceberg/types/Type.java

rdblue · 2024-10-24T22:54:09Z

api/src/main/java/org/apache/iceberg/types/TypeUtil.java

@@ -534,6 +534,7 @@ private static int estimateSize(Type type) {
      case FIXED:
        return ((Types.FixedType) type).length();
      case BINARY:
+      case VARIANT:


What is the rationale for this size?

We can't have the accurate size for Variant similar to Binary. So I use the same value as Binary. I'm wondering how we come up with 80 for Binary.

rdblue · 2024-10-24T22:57:04Z

api/src/main/java/org/apache/iceberg/expressions/ExpressionUtil.java

@@ -562,7 +563,7 @@ private static String sanitize(Literal<?> literal, long now, int today) {
    } else if (literal instanceof Literals.DoubleLiteral) {
      return sanitizeNumber(((Literals.DoubleLiteral) literal).value(), "float");
    } else {
-      // for uuid, decimal, fixed, and binary, match the string result
+      // for uuid, decimal, fixed, variant, and binary, match the string result


This is okay but we may be missing information by not sanitizing based on the variant's type (i.e. date) and it would be nice to have some idea of the structure in the future.

Are you suggesting something like

{ foo: bar: 3 baz: bozz: "flew" }

to

{ (hash-foo): (hash-bar) : (1 digit number) (hash-baz) : (hash-bozz) : (hash-xaxa) }

I think that would be a nice feature but probably ok for it's own issue

Thanks for helping me understand the concept. I have filed a followup issue #11479 and will work on separately.

Actually to support this, we need to have defined the interface VariantLike on how to access the subfields and then recursively sanitize the values based on the subfield types.

When we promote the VariantLike to API, then we can add such implementation.

Yeah, I don't think this needs to be done right away, but being able to have some representation that goes into the variant types would be great.

api/src/test/java/org/apache/iceberg/TestHelpers.java

api/src/test/java/org/apache/iceberg/TestAccessors.java

api/src/main/java/org/apache/iceberg/VariantLike.java

api/src/main/java/org/apache/iceberg/transforms/Identity.java

api/src/main/java/org/apache/iceberg/types/Type.java

RussellSpitzer · 2024-11-01T19:48:00Z

core/src/test/java/org/apache/iceberg/TestTableMetadata.java

@@ -1687,6 +1687,44 @@ public void testV3TimestampNanoTypeSupport() {
        3);
  }

+  @Test
+  public void testV3VariantTypeSupport() {


I know this is copying a lot of tests in this class but we should start future proofing a bit more imho. We also have tests around this sort of thing in TestSchema.java. I think it is probably ok to just keep all of our schema validation tests there, but it wouldn't hurt to have some redundancy here as well.

Refactoring the whole suite can come in another pr but I think we should build a templated test that's something like

@ParameterizedTest @ValueSource(types = {Types.TimetstampNanos, Types.Variant, ....}) testTypeSupport(Type type) { Schema schemaWithType = new Schema( Types.NestedField.required(1, "id", Types.LongType.get()), Types.NestedField.optional(2, type.name, type), Types.NestedField.optional(3, "arr", Types.ListType.ofRequired(4, type)), Types.NestedField.required(5, "struct", Types.StructType.of( Types.NestedField.optional(6, "inner_" + type.name, type), Types.NestedField.required(7, "data", Types.StringType.get()))), Types.NestedField.optional(8, "struct_arr", Types.StructType.of( Types.NestedField.optional(9, "ts", type)))); //Psuedo code here from 0 -> MIN_FORMAT_VERSION.get(type) fail to make metadata from MIN_FORMAT_VERSION.get(type) -> SUPPORTED_TABLE_VERSION succeed }

The most important part about this is that we wouldn't have to continually update tests every time a new valid metadata version is added. It also would be much easier to test type compatibility. (I'm thinking that Geo is going to need the exact same thing soon)

For this PR I think it is enough to write a parameterized version for just Variant, then we could raise another PR to add in nanos and remove the redundant tests

Not this is part of my goal to remove all tests that have V3 or V2 in their title.

Also todo add Variant to Schema.MIN_FORMAT_VERSION

Thanks @RussellSpitzer for already creating a PR to address that. I will include that when it's merged.

+1 for moving this to TestSchema. The test here should really just be validating that schemas are validated against the table version, but the actual validation tests should be in TestSchema.

Lets just drop this test then (or add a single variant field into the above test) and just change the above test to be called testV3SchemaValidation or something

I removed the actual validation for both variant and timestamp_ns since it's already covered in TestSchema. I added version check/validation coverage for TableMetadata.newTableMetadata().

rdblue · 2024-11-07T16:32:43Z

api/src/main/java/org/apache/iceberg/transforms/Identity.java

@@ -38,6 +38,9 @@ class Identity<T> implements Transform<T, T> {
   */
  @Deprecated
  public static <I> Identity<I> get(Type type) {
+    Preconditions.checkArgument(
+        type.typeId() != Type.TypeID.VARIANT, "Unsupported type for identity: %s", type);


I think we also need to make sure that canTransform doesn't accept variants. We could special case it, but I'm actually wondering if it makes sense for variant to be a primitive type because it is only primitive sometimes.

We do only need to special case it if we make Variant a primitive. I'm not sure what primitive really means in this context though. I think anything that may or may not be scalar is probably not a primitive?

I have the test coverage in TestBucketing.java TestIdentity.java to make canTransform doesn't accept variants.

rdblue · 2024-11-07T16:39:59Z

api/src/main/java/org/apache/iceberg/types/Types.java

@@ -412,6 +413,29 @@ public String toString() {
    }
  }

+  public static class VariantType extends PrimitiveType {


I don't think that variant should be either a primitive type or a nested type. There are at least a few places where making variant a primitive creates problems. For example, Schema checks that a field is a primitive to allow it as an identifier field and the Identity transform will bind to a type if it is a primitive. There are also places that check whether a type is nested and make decisions based on whether there are sub-fields.

Neither one of those seems appropriate to me so I'd suggest making this extend just Type. Then we can adjust the logic in those places without assuming either one and potentially introducing bugs when the assumption is bad.

I have isPrimitiveType() to return false. I was debating on that - if VariantType extends PrimitiveType, some internal change would be easier. But concept-wise t should be from Type directly and let me change it.

rdblue · 2024-11-07T16:41:20Z

api/src/test/java/org/apache/iceberg/TestPartitionSpecValidation.java

@@ -312,4 +314,15 @@ public void testAddPartitionFieldsWithAndWithoutFieldIds() {
    assertThat(spec.fields().get(2).fieldId()).isEqualTo(1006);
    assertThat(spec.lastAssignedFieldId()).isEqualTo(1006);
  }
+
+  @Test
+  public void testVariantUnsupported() {


What about other transforms? None of them are allowed, right?

That's right. I have the coverage in TestIdentity.java and TestBucketing.java and I think I have checked all the transforms.

rdblue · 2024-11-07T16:42:50Z

api/src/main/java/org/apache/iceberg/types/Types.java

+
+    @Override
+    public boolean isPrimitiveType() {
+      return false;


Ah, I see that although this extends PrimitiveType, it reports that it is not one. I think it would be safer not to extend PrimitiveType. Good that this is already used to get the tests passing.

… full implementation. Block Transforms, SortOrder

RussellSpitzer

Looks good to me, I think we could clean up TestTableMetadata more (i left a comment there) but we can always push that out as well. I mean to take a pass on cleaning some of that up later as well.

RussellSpitzer · 2024-11-14T17:43:51Z

@rdblue I think you are the last reviewer on this, do you have any further comments for this one?

RussellSpitzer · 2024-11-15T18:52:21Z

Sync'd up with Ryan this morning, we are good to go.

Thanks @aihuaxu for working on this and @rdblue, @gene-db and @Fokko for reviews!

rdblue · 2024-11-16T00:40:55Z

api/src/main/java/org/apache/iceberg/types/Types.java

@@ -412,6 +412,41 @@ public String toString() {
    }
  }

+  public static class VariantType implements Type {


@aihuaxu, could you also implement the same Java serialization that we use for PrimitiveType? The existing PrimitiveHolder should work:

Object writeReplace() throws ObjectStreamException { return new PrimitiveHolder(toString()); }

rdblue · 2024-11-16T00:46:51Z

Late +1 from me! Thanks @aihuaxu!

github-actions bot added the API label Oct 15, 2024

aihuaxu marked this pull request as ready for review October 15, 2024 18:34

RussellSpitzer reviewed Oct 15, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Oct 15, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Oct 15, 2024

View reviewed changes

api/src/test/java/org/apache/iceberg/util/RandomUtil.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Oct 15, 2024

View reviewed changes

api/src/test/java/org/apache/iceberg/types/TestSerializableTypes.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Oct 15, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

aihuaxu force-pushed the variant-type-api branch from 3a3282d to 15d8ed6 Compare October 18, 2024 03:47

aihuaxu requested review from RussellSpitzer and gene-db October 18, 2024 04:05

aihuaxu force-pushed the variant-type-api branch from 15d8ed6 to 15b2b71 Compare October 18, 2024 04:08

RussellSpitzer reviewed Oct 21, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Oct 21, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

aihuaxu requested a review from RussellSpitzer October 22, 2024 15:42

RussellSpitzer reviewed Oct 22, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Oct 22, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Oct 22, 2024

View reviewed changes

api/src/test/java/org/apache/iceberg/TestHelpers.java Outdated Show resolved Hide resolved

aihuaxu force-pushed the variant-type-api branch from ca226ee to 1d3056b Compare October 22, 2024 21:49

aihuaxu requested a review from RussellSpitzer October 22, 2024 21:50

aihuaxu force-pushed the variant-type-api branch from 1d3056b to d4af2b3 Compare October 23, 2024 16:26

rdblue reviewed Oct 24, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/types/Type.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 24, 2024

View reviewed changes

api/src/test/java/org/apache/iceberg/TestHelpers.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 24, 2024

View reviewed changes

api/src/test/java/org/apache/iceberg/TestAccessors.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 24, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 24, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 24, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 24, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/VariantLike.java Outdated Show resolved Hide resolved

aihuaxu force-pushed the variant-type-api branch from d4af2b3 to 5c07698 Compare October 29, 2024 04:33

aihuaxu force-pushed the variant-type-api branch 2 times, most recently from 327736b to 82f2e7b Compare November 1, 2024 17:20

RussellSpitzer reviewed Nov 1, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/transforms/Identity.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Nov 1, 2024

View reviewed changes

api/src/main/java/org/apache/iceberg/types/Type.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Nov 1, 2024

View reviewed changes

aihuaxu mentioned this pull request Nov 5, 2024

API: Follow up on adding Variant data type to implement sanitizing for Variant #11479

Open

3 tasks

aihuaxu requested a review from RussellSpitzer November 5, 2024 19:58

aihuaxu force-pushed the variant-type-api branch 2 times, most recently from 14665cb to ca2b5de Compare November 7, 2024 04:00

rdblue reviewed Nov 7, 2024

View reviewed changes

aihuaxu added 4 commits November 7, 2024 09:48

API: Add Variant data type

ed67f0a

Update the interface.

7ff5e2b

Add JavaDoc

5e76dd2

Only add the type to API. Will add VariantLike stable interface after…

95596cf

… full implementation. Block Transforms, SortOrder

aihuaxu force-pushed the variant-type-api branch from ca2b5de to 5eebb6e Compare November 7, 2024 18:27

RussellSpitzer approved these changes Nov 7, 2024

View reviewed changes

aihuaxu force-pushed the variant-type-api branch 2 times, most recently from 9b77a5f to 4050ac0 Compare November 7, 2024 21:44

aihuaxu requested a review from rdblue November 7, 2024 21:50

VariantType inherits from Type directly

1e2fc66

aihuaxu force-pushed the variant-type-api branch from 4050ac0 to 1e2fc66 Compare November 7, 2024 22:43

Fokko approved these changes Nov 13, 2024

View reviewed changes

RussellSpitzer merged commit 821aec3 into apache:main Nov 15, 2024
50 checks passed

rdblue reviewed Nov 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API: Add Variant data type #11324

API: Add Variant data type #11324

aihuaxu commented Oct 15, 2024 •

edited

Loading

rdblue Oct 24, 2024

aihuaxu Oct 29, 2024

rdblue Oct 24, 2024

RussellSpitzer Nov 1, 2024

RussellSpitzer Nov 1, 2024

aihuaxu Nov 5, 2024

aihuaxu Nov 6, 2024

rdblue Nov 7, 2024

RussellSpitzer Nov 1, 2024 •

edited

Loading

RussellSpitzer Nov 1, 2024

RussellSpitzer Nov 1, 2024

aihuaxu Nov 2, 2024

rdblue Nov 7, 2024

RussellSpitzer Nov 7, 2024 •

edited

Loading

aihuaxu Nov 7, 2024

rdblue Nov 7, 2024

RussellSpitzer Nov 7, 2024 •

edited

Loading

aihuaxu Nov 7, 2024

rdblue Nov 7, 2024

aihuaxu Nov 7, 2024

rdblue Nov 7, 2024

aihuaxu Nov 7, 2024

rdblue Nov 7, 2024

RussellSpitzer left a comment

RussellSpitzer commented Nov 14, 2024

RussellSpitzer commented Nov 15, 2024

rdblue Nov 16, 2024

rdblue commented Nov 16, 2024

API: Add Variant data type #11324

API: Add Variant data type #11324

Conversation

aihuaxu commented Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer Nov 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer left a comment

Choose a reason for hiding this comment

RussellSpitzer commented Nov 14, 2024

RussellSpitzer commented Nov 15, 2024

Choose a reason for hiding this comment

rdblue commented Nov 16, 2024

aihuaxu commented Oct 15, 2024 •

edited

Loading

RussellSpitzer Nov 1, 2024 •

edited

Loading

RussellSpitzer Nov 7, 2024 •

edited

Loading

RussellSpitzer Nov 7, 2024 •

edited

Loading