From 5e091a766cbcbdaec54a534b2a282a8b494bf305 Mon Sep 17 00:00:00 2001 From: Ewan Keith Date: Mon, 26 Jun 2023 23:08:28 +0100 Subject: [PATCH 1/3] add support for spark TimestampNtz type to docs --- docs/spark-writes.md | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/docs/spark-writes.md b/docs/spark-writes.md index 5a312da7f1ee..84736301b956 100644 --- a/docs/spark-writes.md +++ b/docs/spark-writes.md @@ -397,25 +397,26 @@ so you may want to understand the type conversion in Iceberg in prior to design This type conversion table describes how Spark types are converted to the Iceberg types. The conversion applies on both creating Iceberg table and writing to Iceberg table via Spark. -| Spark | Iceberg | Notes | -|-----------------|-------------------------|-------| -| boolean | boolean | | -| short | integer | | -| byte | integer | | -| integer | integer | | -| long | long | | -| float | float | | -| double | double | | -| date | date | | -| timestamp | timestamp with timezone | | -| char | string | | -| varchar | string | | -| string | string | | -| binary | binary | | -| decimal | decimal | | -| struct | struct | | -| array | list | | -| map | map | | +| Spark | Iceberg | Notes | +|-----------------|----------------------------|-------| +| boolean | boolean | | +| short | integer | | +| byte | integer | | +| integer | integer | | +| long | long | | +| float | float | | +| double | double | | +| date | date | | +| timestamp | timestamp with timezone | | +| timestampNtz | timestamp without timezone | | +| char | string | | +| varchar | string | | +| string | string | | +| binary | binary | | +| decimal | decimal | | +| struct | struct | | +| array | list | | +| map | map | | {{< hint info >}} The table is based on representing conversion during creating table. In fact, broader supports are applied on write. Here're some points on write: @@ -438,7 +439,7 @@ This type conversion table describes how Iceberg types are converted to the Spar | date | date | | | time | | Not supported | | timestamp with timezone | timestamp | | -| timestamp without timezone | | Not supported | +| timestamp without timezone | timestampNtz | | | string | string | | | uuid | string | | | fixed | binary | | From d17d10dfdcce21d449bc6750578a4c95ac6bd847 Mon Sep 17 00:00:00 2001 From: Ewan Keith Date: Tue, 27 Jun 2023 10:29:42 +0100 Subject: [PATCH 2/3] Update type casing in spark docs Co-authored-by: Fokko Driesprong --- docs/spark-writes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/spark-writes.md b/docs/spark-writes.md index 84736301b956..7870ec30cfa8 100644 --- a/docs/spark-writes.md +++ b/docs/spark-writes.md @@ -408,7 +408,7 @@ This type conversion table describes how Spark types are converted to the Iceber | double | double | | | date | date | | | timestamp | timestamp with timezone | | -| timestampNtz | timestamp without timezone | | +| timestamp_ntz | timestamp without timezone | | | char | string | | | varchar | string | | | string | string | | From 4e7b4c6c5b6b66d04b2d54e50a0d9fbc1167bb58 Mon Sep 17 00:00:00 2001 From: Ewan Keith Date: Tue, 27 Jun 2023 10:30:01 +0100 Subject: [PATCH 3/3] Update type casing in spark docs Co-authored-by: Fokko Driesprong --- docs/spark-writes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/spark-writes.md b/docs/spark-writes.md index 7870ec30cfa8..bffbbf9be0b9 100644 --- a/docs/spark-writes.md +++ b/docs/spark-writes.md @@ -439,7 +439,7 @@ This type conversion table describes how Iceberg types are converted to the Spar | date | date | | | time | | Not supported | | timestamp with timezone | timestamp | | -| timestamp without timezone | timestampNtz | | +| timestamp without timezone | timestamp_ntz | | | string | string | | | uuid | string | | | fixed | binary | |