Simplify Variant shredding and refactor for clarity

[rdblue]

Rationale for this change

Updating the Variant and shredding specs from a thorough review.

What changes are included in this PR?

Spec updates, mostly to the shredding spec to minimize it and make it clear. This also attempts to make the variant spec more consistent (for example, by using value in both).

• Removes object and array in favor of always using typed_value
• Makes list element and object field groups required to avoid unnecessary null cases
• Separates cases for primitives, arrays, and objects
• Adds individual examples for primitives, arrays, and objects
• Adds Variant to Parquet type mapping for shredded columns
• Clarifies that metadata must be valid for all variant values without modification
• Updates reconstruction algorithm to be more pythonic

Do these changes have PoC implementations?

No.

[rdblue]

I've updated these to not use more than one H1, which can cause issues with TOC. Pages should have just one H1.

[RussellSpitzer]

TIL as well

[rdblue]

This may be controversial. I'm trying to say that you should not need to modify the metadata when reading. The reconstructed object should be able to use the stored metadata without adding fields.

[sfc-gh-aixu]

I'm a little confused. When the field is not shredded, we will not have metadata for it, right? When it's getting shredded, then it will be like a column and we will generate metadata so it can be used for filtering/pruning?

[rdblue]

@sfc-gh-aixu, this is saying that when writing, the metadata for a shredded value and the metadata for a non-shredded value should be identical. Writers should not alter the metadata by removing shredded field names so that readers do not need to rewrite the metadata (and values) to add it back.

For example, consider an event that looks like this:

{
  "id": 102,
  "event_type": "signup",
  "event_timestamp": "2024-10-21T20:06:34.198724",
  "payload": {
    "a": 1,
    "b": 2
  }
}

And a shredding schema:

optional group event (VARIANT) {
  required binary metadata;
  optional binary value;
  optional group typed_value {
    required group event_type {
      optional binary value;
      optional binary typed_value (STRING);
    }
    required group event_timestamp {
      optional binary value;
      optional int64 typed_value (TIMESTAMP(true, MICROS));
    }
  }
}

The top-level event_type and event_timestamp fields are shredded. But this is saying that the Variant metadata must include those field names. That ensure that the existing binary metadata can be returned to the engine without adding event_type and event_timestamp fields when merging those fields into the top-level Variant value when the entire Variant is projected.

[sfc-gh-saya]

Why each shredded field should be a required group is not clear to me. If fields were allowed to be optional, that would be another way of indicating non-existence of fields.

[rdblue]

The primary purpose is to reduce the number of cases that implementers have to deal with. If all of the cases can be expressed with 2 optional fields rather than 2 optional fields inside an optional group, then the group should be required to simplify as much as possible.

In addition, every level in Parquet that is optional introduces another repetition/definition level. That adds up quickly with nested structures and ends up taking unnecessary space.

[RussellSpitzer]

Another place I'd like to just remove some of the text here. My main goal here is just to reduce the amount of text in the spec

[rdblue]

I reduced this, but I don't think it's a problem to have a bit of context that answers the question "What is shredding and why do I care?"

[rdblue]

@RussellSpitzer and @gene-db, this could use some attention.

Here, if both value and typed_value are non-null I initially thought it made more sense to prefer value because it doesn't need to be re-encoded and may have been coerced by an engine to the shredded type.

However, this conflicts with object fields, where the value of typed_value is preferred so that data skipping is correct. If the object's value could contains a field that conflicts with a sub-field's typed_value there is no way of knowing from field stats. If we preferred the field value stored in the object's value then data skipping could be out of sync with the value returned in the case of a conflict.

[gene-db]

For exact numerics, we should allow truncating trailing zeros. For example, int8 value 1 and decimal(5,2) value 100 can both be represented as a JSON value 1.

Also, should the example be quoted to stay consistent?

Suggested change

	| Exact Numeric | number | Digits in fraction must match scale, no exponent | `34`, 34.00 |
	| Exact Numeric | number | Digits in fraction must match scale, no exponent | `34`, `34.00` |

[cashmand]

At this point, isn't non-shredded just a special case of shredded with no typed_value in the top level struct? I think it's automatically backwards compatible.

[cashmand]

Just to be clear, this is only allowed for object fields, right? You mention in the array section that array elements must have one of them non-null, but I think that's also true for the top-level value/typed_value, right?

[cashmand]

As mentioned in my previous comment, I think it would be invalid for measurement to have both value and typed_value be null, and should be an error. I don't understand why we're recommend returning variant null as an option.

[cashmand]

I think that here and in the array case, it would be good to clarify whether typed_value can be omitted from the schema entirely. E.g. if there's no consistent type for a field, I think we'd still want to shred the field, but put all values in value, and not require that a typed_value type be specified.

[cashmand]

Conversely, is value always required? Would it be valid for a writer to only create a typed_value column if it knows that all values have a predictable type that can be shredded?