Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove lower precision time/date types. #22

Merged
merged 3 commits into from
Sep 10, 2021
Merged

Remove lower precision time/date types. #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
453540c
Remove lower precision time/date types.
jacques-n Sep 9, 2021
0a20684
Fix duplicate name.
jacques-n Sep 9, 2021
3dd9fb8
Apply suggestions from code review
jacques-n Sep 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions binary/expression.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,9 @@ message Expression {
double fp64 = 11;
string string = 12;
bytes binary = 13;
fixed64 timestamp_micro = 14;
fixed64 timestamp_milli = 15;
fixed64 timestamp = 14;
fixed32 date = 16;
fixed32 time_micro = 17;
uint32 time_milli = 18;
uint64 time = 17;
IntervalYearToMonth interval_year_to_month = 19;
IntervalDayToSecond interval_day_to_second = 20;
string fixed_char = 21;
Expand All @@ -42,8 +40,7 @@ message Expression {
bytes decimal = 24;
Struct struct = 25;
Map map = 26;
fixed64 timestamp_micro_tz = 27;
fixed64 timestamp_milli_tz = 28;
fixed64 timestamp_tz = 27;
}

message Map {
Expand Down
33 changes: 7 additions & 26 deletions binary/type.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,9 @@ message Type {
FP64 fp64 = 11;
String string = 12;
Binary binary = 13;
TimestampMicro timestamp_micro = 14;
TimestampMilli timestamp_milli = 15;
Date time = 16;
TimeMicro time_micro = 17;
TimeMilli time_milli = 18;
Timestamp timestamp = 14;
Date date = 16;
Time time = 17;
IntervalYear interval_year = 19;
IntervalDay interval_day = 20;
FixedChar fixed_char = 21;
Expand All @@ -27,8 +25,7 @@ message Type {
NamedStruct named_struct = 26;
List list = 27;
Map map = 28;
TimestampMicroTZ timestamp_micro_tz = 29;
TimestampMilliTZ timestamp_milli_tz = 30;
TimestampTZ timestamp_tz = 29;
UserDefined user_defined = 31;
}

Expand Down Expand Up @@ -140,15 +137,7 @@ message Type {
}
}

message TimestampMicro {
PhysicalType physical_type = 1;

enum PhysicalType {
SYSTEM_DEFAULT = 0;
}
}

message TimestampMilli {
message Timestamp {
PhysicalType physical_type = 1;

enum PhysicalType {
Expand All @@ -164,15 +153,7 @@ message Type {
}
}

message TimeMicro {
PhysicalType physical_type = 1;

enum PhysicalType {
SYSTEM_DEFAULT = 0;
}
}

message TimeMilli {
message Time {
PhysicalType physical_type = 1;

enum PhysicalType {
Expand Down Expand Up @@ -302,7 +283,7 @@ message Type {
}
}

message TimestampMilliTZ {
message TimestampTZ {
string timezone = 1;

PhysicalType physical_type = 2;
Expand Down
6 changes: 2 additions & 4 deletions site/docs/types/compound_logical_types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,11 @@ Compound types include any type that is configurable including complex types as
| FIXEDCHAR(L) | A fixed length field of length L. L can be between [1..2,147,483,647]. Values less that are less in length than the length of the field are padded with spaces. | None | None | CharType(L) | CHAR(L) |
| VARCHAR(L) | A field that can holds UTF8 encoded strings between 0 and L length. The length of each value can be between [0..2,147,483,647]. The value of L can be between [1..2,147,483,647]. Values shorter than L are not padded. | None | None | VarcharType(L) | VARCHAR(L) |
| FIXEDBINARY(L) | A binary field that is fixed in width to L. Values that are shorter than L are 0-byte padded. | FixedSizeBinary<L> | FIXED<L> | - | - |
| DECIMAL(P,S) | Fixed point decimal with precision P and scale S. Precision must be 38 or less. | DECIMAL(P,S) | DECIMAL(P,S) | DECIMAL(P,S) | DECIMAL(P,S) |
| DECIMAL(P,S) | A fixed precision decimal value having precision (P, number of digits) <= 38 and Scale (S, number of fractional digits) 0 <= S <= P) | Decimal&lt;P, S, bitwidth=128&gt; | DECIMAL(P,S) | DECIMAL(P,S) | DECIMAL(P,S) |
| STRUCT&lt;N:T1,...,N:T2&gt; | A struct that maps unique names to value types. Each name is a UTF8 string. Each value can have a distinct type. | struct_&lt;*&gt; | struct&lt;*&gt; | struct&lt;*&gt; | row&lt;*&gt; |
| LIST&lt;T&gt; | A list of values of type T. The list can be between [0..2,147,483,647] values in length. Maps to the | list | list | list | array |
| MAP&lt;K, V&gt; | An unordered list of type K keys with type V values. | map&lt;k,v&gt; | map&lt;k,v&gt; | - | map&lt;k,v&gt; |
| TIMESTAMP_MICRO_TZ(TZ) | A timestamp in microseconds with a timezone TZ. | timestamp&lt;micro;tz&gt; | timestamptz | - | timestamp(6) with time zone |
| TIMESTAMP_MILLI_TZ(TZ) | A timestamp in microseconds with a timezone TZ. | timestamp&lt;milli;tz&gt; | - | - | timestamp(3) with time zone |
| Decimal(P, S) | A fixed precision decimal value having precision (P, number of digits) <= 38 and Scale (S, number of fractional digist) 0 <= S <= P) | Decimal<P, S, bitwidth=128> | decimal(P,S) | decimal(P, S) | DECIMAL(P, S)
| TIMESTAMP_TZ(TZ) | A timestamp with microseconds precision and a type declared timezone TZ. | timestamp&lt;micro;tz&gt; | timestamptz | - | timestamp(6) with time zone |

#### Discussion Points

Expand Down
52 changes: 24 additions & 28 deletions site/docs/types/physical_types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,34 +6,30 @@ For each logical type, we declare one or more physical representations of that l

In many cases, a system will only have a single physical representation of each type. In those cases, it is expected that the binding of an operation is associated with the system default representation of the data. While a physical types are defined as discrete from logical types within the specification, the serialization formats will typically collapse these into a singular concept.

| Logical Type | Physical Representations | Support Dictionary Encoding |
| ------------------ | ----------------------------------------------------------- | --------------------------- |
| boolean | 0=System default | no |
| i8 | 0=System default | no |
| i16 | 0=System default | no |
| i32 | 0=System default | no |
| i64 | 0=System default | no |
| fp32 | 0=System default | no |
| fp64 | 0=System default | no |
| string | 0=System default, 1=Arrow Large String | yes |
| binary | 0=System default, Arrow Large Binary | yes |
| timestamp_micro | 0=System default | no |
| timestamp_milli | 0=System default | no |
| date | 0=System default | no |
| date_micro | 0=System default | no |
| time_micro | 0=System default | no |
| time_milli | 0=System default | no |
| interval_year | 0=System default | no |
| interval_day | 0=System default, 1=Arrow MONTH_DAY_NANO | no |
| fixedchar | 0=System default | yes |
| varchar | 0=System default | yes |
| fixedbinary | 0=System default | yes |
| decimal | 0=System default, 1=Arrow 128 Bit Width | no |
| struct | 0=System default | yes |
| list | 0=System default, 1=Arrow Large List | yes |
| map | 0=System default, 1=Map where keys are utf8 ordered strings | yes |
| timestamp_micro_tz | 0=System default | no |
| timestamp_milli_tz | 0=System default | no |
| Logical Type | Physical Representations | Support Dictionary Encoding |
| ------------- | ----------------------------------------------------------- | --------------------------- |
| boolean | 0=System default | no |
| i8 | 0=System default | no |
| i16 | 0=System default | no |
| i32 | 0=System default | no |
| i64 | 0=System default | no |
| fp32 | 0=System default | no |
| fp64 | 0=System default | no |
| string | 0=System default, 1=Arrow Large String | yes |
| binary | 0=System default, Arrow Large Binary | yes |
| timestamp | 0=System default | no |
| date | 0=System default | no |
| time | 0=System default | no |
| interval_year | 0=System default | no |
| interval_day | 0=System default, 1=Arrow MONTH_DAY_NANO | no |
| fixedchar | 0=System default | yes |
| varchar | 0=System default | yes |
| fixedbinary | 0=System default | yes |
| decimal | 0=System default, 1=Arrow 128 Bit Width | no |
| struct | 0=System default | yes |
| list | 0=System default, 1=Arrow Large List | yes |
| map | 0=System default, 1=Map where keys are utf8 ordered strings | yes |
| timestamp_tz | 0=System default | no |



Expand Down
8 changes: 3 additions & 5 deletions site/docs/types/simple_logical_types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,9 @@ To minimize type explosion, the project currently follows the guideline that a l
| fp64 | An 8 byte double precision floating point number with range as defined [here](https://en.wikipedia.org/wiki/Double-precision_floating-point_format). | Float&lt;DOUBLE&gt; | double | DecimalType | double |
| string | A string of text that can be up to 2,147,483,647 bytes in length. String is encoded in UTF8 | Utf8 | string | StringType | varchar (no len) |
| binary | A binary value that can be up to 2,147,483,647 bytes in length. | Binary | binary | BinaryType | Varbinary |
| timestamp_micro | A timestamp with microsecond precision | Timestamp&lt;MICROSECOND&gt; | timestamp | TimestampType | timestamp(6) |
| timestamp_milli | A timestamp with millisecond precision | Timestamp&lt;MILLISECOND&gt; | - | - | timestamp(3) |
| date | Date, expressed as number of seconds since epoch | Date&lt;MILLISECOND&gt; | date | DateType | Date |
| time_micro | A time expressed in microseconds since start of day | Time&lt;MICROSECOND;64&gt; | time | time(6) | time(6) |
| time_milli | A time expressed in milliseconds since start of day | Time&lt;MILLISECOND;32&gt; | - | time(3) | time(3) |
| timestamp | A timestamp with microsecond precision | Timestamp&lt;MICROSECOND&gt; | timestamp | TimestampType | timestamp(6) |
| date | A date | Date&lt;MILLISECOND&gt; | date | DateType | Date |
| time | A time with microsecond precision since the beginning of any day. Maximum value is one day. | Time&lt;MICROSECOND;64&gt; | time | time(6) | time(6) |
| interval_year | Interval day to month | INTERVAL&lt;YEAR_MONTH&gt; | - | - | Interval year to month |
| interval_day | Interval day to second | INTERVAL&lt;DAY_TIME&gt; | - | - | Interval day to second |

Expand Down