Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Snowflake External Table with Virtual Column support a kind of Computed Column as T-SQL #218

Closed
dmaresma opened this issue Jan 3, 2024 · 0 comments

Comments

@dmaresma
Copy link
Contributor

dmaresma commented Jan 3, 2024

I would like to support CREATE EXTERNAL TABLE statement support as the following test
here the reference in Snowflake documentation
https://docs.snowflake.com/en/sql-reference/sql/create-external-table

def test_virtual_column_table():
    ddl = """
    create or replace external table if not exists TABLE_DATA_SRC.EXT_PAYLOAD_MANIFEST_WEB(
        "type" VARCHAR(255) AS (SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 1), '=', 2)),
        "year" VARCHAR(255) AS (SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 2), '=', 2)),
        "month" VARCHAR(255) AS (SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 3), '=', 2)),
        "day" VARCHAR(255) AS (SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 4), '=', 2)),
        "path" VARCHAR(255) AS (METADATA$FILENAME))
    partition by ("type", "year", "month", "day", "path")
    location=@ADL_Azure_Storage_Account_Container_Name/
    auto_refresh=false
    file_format=(TYPE=JSON NULL_IF=())
    ;
    """
    result_ext_table = DDLParser(ddl, normalize_names=True, debug=True).run(
        output_mode="snowflake"
    )

       expected_ext_table = [
        {
            "alter": {},
            "checks": [],
            "clone": None,
            "columns": [
                {
                    "name": "type",
                    "type": "VARCHAR",
                    "size": 255,
                    "comment": "",
                    "references": None,
                    "unique": False,
                    "nullable": True,
                    "default": None,
                    "check": None,
                   "generated" : {"as"  : "SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 1), '=', 2)" , "stored": False, "always" : True}
                },
                {
                    "name": "year",
                    "type": "VARCHAR",
                    "size": 255,
                    "comment": "",
                    "references": None,
                    "unique": False,
                    "nullable": True,
                    "default": None,
                    "check": None,
                   "generated" : {"as"  : "SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 2), '=', 2)" , "stored": False, "always" : True}
                },
                {
                    "name": "month",
                    "type": "VARCHAR",
                    "size": 255,
                    "comment": "",
                    "references": None,
                    "unique": False,
                    "nullable": True,
                    "default": None,
                    "check": None,
                   "generated" : {"as" : "SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 3), '=', 2)", "stored": False, "always" : True}
                },
                {
                    "name": "day",
                    "type": "VARCHAR",
                    "size": 255,
                    "comment": "",
                    "references": None,
                    "unique": False,
                    "nullable": True,
                    "default": None,
                    "check": None,
                   "generated" : {"as" : "SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 4), '=', 2)", "stored": False, "always" : True}
                },
                {
                    "name": "path",
                    "type": "VARCHAR",
                    "size": 255,
                    "comment": "",
                    "references": None,
                    "unique": False,
                    "nullable": True,
                    "default": None,
                    "check": None,
                   "generated" : {"as" : "METADATA$FILENAME)", "stored": False, "always" : True}
                }
            ],
            "constraints": {
                "primary_keys": []
            },
            "index": [],
            "partitioned_by": []
            "partition_by" : ["type", "year", "month", "day", "path"],
            "primary_key": [],
            "primary_key_enforced": None,
            "location" : "@ADL_Azure_Storage_Account_Container_Name/",
            "auto_refresh" : False,
            "schema": "TABLE_DATA_SRC",
            "table_name": "EXT_PAYLOAD_MANIFEST_WEB",
        }
    ]

    assert result_ext_table == expected_ext_table

Implementing Virutal Column
"type" VARCHAR(255) AS (SPLIT_PART(SPLIT_PART(METADATA$FILENAME, '/', 1), '=', 2))

improve generated as columns attribute, location , file_format and auto_refresh as table attributes

check partitioned_by still supported

Additional context
ICEBERG Table is comming soon 👍

dmaresma added a commit to dmaresma/simple-ddl-parser that referenced this issue Jan 4, 2024
xnuinside added a commit that referenced this issue Jan 5, 2024
@dmaresma dmaresma closed this as completed Jan 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant