Skip to content

Commit

Permalink
[apache#2323] docs: document UnparsedType data type which handles a…
Browse files Browse the repository at this point in the history
…n unresolvable type from the catalog (apache#2337)

### What changes were proposed in this pull request?

Document `UnparsedType` data type which handles an unresolvable type
from the catalog.

### Why are the changes needed?

`UnparsedType` data type that handles an unresolvable type from the
catalog is supported in apache#2140.

Fix: apache#2323

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

No.
  • Loading branch information
SteNicholas authored Feb 28, 2024
1 parent 0bc2237 commit 2ea1eef
Show file tree
Hide file tree
Showing 6 changed files with 61 additions and 6 deletions.
4 changes: 4 additions & 0 deletions docs/apache-hive-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,10 @@ The following table lists the data types mapped from the Hive catalog to Graviti
| `struct` | `struct` | 0.2.0 |
| `uniontype` | `uniontype` | 0.2.0 |

:::info
Since 0.5.0, the data types other than listed above are mapped to Gravitino **[Unparsed Type](./manage-metadata-using-gravitino.md#unparsed-type)** that represents an unresolvable data type from the Hive catalog.
:::

### Table properties

Table properties supply or set metadata for the underlying Hive tables.
Expand Down
1 change: 1 addition & 0 deletions docs/jdbc-mysql-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ Please refer to [Manage Metadata Using Gravitino](./manage-metadata-using-gravit

:::info
MySQL doesn't support Gravitino `Boolean` `Fixed` `Struct` `List` `Map` `Timestamp_tz` `IntervalDay` `IntervalYear` `Union` `UUID` type.
Meanwhile, the data types other than listed above are mapped to Gravitino **[Unparsed Type](./manage-metadata-using-gravitino.md#unparsed-type)** that represents an unresolvable data type since 0.5.0.
:::

#### Table column auto-increment
Expand Down
1 change: 1 addition & 0 deletions docs/jdbc-postgresql-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ Please refer to [Manage Metadata Using Gravitino](./manage-metadata-using-gravit

:::info
PostgreSQL doesn't support Gravitino `Fixed` `Struct` `List` `Map` `IntervalDay` `IntervalYear` `Union` `UUID` type.
Meanwhile, the data types other than listed above are mapped to Gravitino **[Unparsed Type](./manage-metadata-using-gravitino.md#unparsed-type)** that represents an unresolvable data type since 0.5.0.
:::

#### Table column auto-increment
Expand Down
1 change: 1 addition & 0 deletions docs/lakehouse-iceberg-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,7 @@ Apache Iceberg doesn't support Gravitino `EvenDistribution` type.

:::info
Apache Iceberg doesn't support Gravitino `Varchar` `Fixedchar` `Byte` `Short` `Union` type.
Meanwhile, the data types other than listed above are mapped to Gravitino **[Unparsed Type](./manage-metadata-using-gravitino.md#unparsed-type)** that represents an unresolvable data type since 0.5.0.
:::

### Table properties
Expand Down
37 changes: 32 additions & 5 deletions docs/manage-metadata-using-gravitino.md
Original file line number Diff line number Diff line change
Expand Up @@ -609,8 +609,8 @@ You can create a table by sending a `POST` request to the `/api/metalakes/{metal
```shell
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"name": "exmaple_table",
"comment": "This is an exmaple table",
"name": "example_table",
"comment": "This is an example table",
"columns": [
{
"name": "id",
Expand Down Expand Up @@ -756,7 +756,7 @@ Map<String, String> tablePropertiesMap = ImmutableMap.<String, String>builder()
.build();

tableCatalog.createTable(
NameIdentifier.of("metalake", "catalog", "schema", "exmaple_table"),
NameIdentifier.of("metalake", "catalog", "schema", "example_table"),
new Column[] {
Column.of("id", Types.IntegerType.get(), "id column comment", false, true, Literals.integerLiteral(-1)),
Column.of("name", Types.VarCharType.of(500), "name column comment", true, false, Literals.NULL),
Expand All @@ -768,7 +768,7 @@ tableCatalog.createTable(
), "info column comment", true, false, null),
Column.of("dt", Types.DateType.get(), "dt column comment", true, false, null)
},
"This is an exmaple table",
"This is an example table",
tablePropertiesMap,
new Transform[] {Transforms.identity("id")},
Distributions.of(Strategy.HASH, 32, NamedReference.field("id")),
Expand Down Expand Up @@ -823,6 +823,32 @@ The following types that Gravitino supports:

The related java doc is [here](pathname:///docs/0.4.0/api/java/com/datastrato/gravitino/rel/types/Type.html).

##### Unparsed type

Unparsed type is a special type of column type, currently serves exclusively for presenting the data type of a column when it's unsolvable.
The following shows the data structure of an unparsed type in JSON and Java, enabling easy retrieval of its value.

<Tabs>
<TabItem value="Json" label="Json">

```json
{
"type": "unparsed",
"unparsedType": "user-defined"
}
```

</TabItem>
<TabItem value="java" label="Java">

```java
// The result of the following type is a string "user-defined"
String unparsedValue = ((UnparsedType) type).unparsedType();
```

</TabItem>
</Tabs>

#### Table column default value

When defining a table column, you can specify a [literal](./expression.md#literal) or an [expression](./expression.md) as the default value. The default value typically applies to new rows that are inserted into the table by the underlying catalog.
Expand Down Expand Up @@ -906,7 +932,8 @@ tableCatalog.loadTable(NameIdentifier.of("metalake", "hive_catalog", "schema", "
</Tabs>

:::note
When Gravitino loads a table from a catalog that supports default value, if Gravitino is unable to parse the default value, it will use an **[Unparsed Expression](./expression.md#unparsed-expression)** to preserve the original default value, ensuring that the table can be loaded successfully.
- When Gravitino loads a table from a catalog with various data types, if Gravitino is unable to parse the data type, it will use an **[Unparsed Type](#unparsed-type)** to preserve the original data type, ensuring that the table can be loaded successfully.
- When Gravitino loads a table from a catalog that supports default value, if Gravitino is unable to parse the default value, it will use an **[Unparsed Expression](./expression.md#unparsed-expression)** to preserve the original default value, ensuring that the table can be loaded successfully.
:::

### Alter a table
Expand Down
23 changes: 22 additions & 1 deletion docs/open-api/datatype.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ components:
- $ref: "#/components/schemas/ListType"
- $ref: "#/components/schemas/MapType"
- $ref: "#/components/schemas/UnionType"
- $ref: "#/components/schemas/UnparsedType"

PrimitiveType:
type: string
Expand Down Expand Up @@ -162,4 +163,24 @@ components:
comment:
type: string
description: The comment of the struct field
nullable: true
nullable: true

UnparsedType:
type: object
required:
- type
- unparsedType
properties:
type:
type: string
enum:
- "unparsed"
unparsedType:
type: string
description: The unparsed type
example: {
"type": "unparsed",
"unparsedType": [
"user-defined"
]
}

0 comments on commit 2ea1eef

Please sign in to comment.