Skip to content

Commit

Permalink
Merge pull request #219 from cfhowes/fix_key_parsing
Browse files Browse the repository at this point in the history
Parse KEY statements within CREATE TABLE statements
  • Loading branch information
xnuinside authored Jan 5, 2024
2 parents bb52593 + 9788bc4 commit e60c7ee
Show file tree
Hide file tree
Showing 4 changed files with 46,254 additions and 462 deletions.
51 changes: 27 additions & 24 deletions CHANGELOG.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
**v0.31.3**
### Improvements
1. Support for KEY statement in CREATE TABLE statements. KEY statements will now create INDEX entries in the DDL parser.

**v0.31.3**
### Improvements
Expand Down Expand Up @@ -26,7 +29,7 @@ Default encoding is utf-8.
Fix for: https://github.com/xnuinside/simple-ddl-parser/issues/199

### Improvements:
1. Added `Snowflake Table DDL support of WITH MASKING POLICY column definition` - https://github.com/xnuinside/simple-ddl-parser/issues/201
1. Added `Snowflake Table DDL support of WITH MASKING POLICY column definition` - https://github.com/xnuinside/simple-ddl-parser/issues/201


### Updates:
Expand Down Expand Up @@ -61,11 +64,11 @@ Added support for ORGANIZE BY statement
### Fixes
1. AUTOINCREMENT statement now parsed validly same way as AUTO_INCREMENT and showed up in output as 'autoincrement' property of the column
Fix for: https://github.com/xnuinside/simple-ddl-parser/issues/170
2. Fix issue ' TypeError argument of type 'NoneType' is not iterable' on some foreigen keys https://github.com/xnuinside/simple-ddl-parser/issues/148
2. Fix issue ' TypeError argument of type 'NoneType' is not iterable' on some foreigen keys https://github.com/xnuinside/simple-ddl-parser/issues/148

### New Features

1. Support for non-numeric column type parameters https://github.com/xnuinside/simple-ddl-parser/issues/171
1. Support for non-numeric column type parameters https://github.com/xnuinside/simple-ddl-parser/issues/171
It shows in column attribute 'type_parameters'.


Expand All @@ -80,8 +83,8 @@ Fixes:
**v0.28.0**

Important Changes (Pay attention):
1. Because of parsing now AUTO_INCREMENT as a separate property of column previous output changed.
Previously it was parsed as a part of type like: 'INT AUTO_INCREMENT'.
1. Because of parsing now AUTO_INCREMENT as a separate property of column previous output changed.
Previously it was parsed as a part of type like: 'INT AUTO_INCREMENT'.
Now type will be only 'INT', but in column property you will see 'autoincrement': True.

Amazing innovation:
Expand Down Expand Up @@ -110,10 +113,10 @@ Fixes:
Improvements:

1. Added support for '*' in size column (ORACLE dialect) - https://github.com/xnuinside/simple-ddl-parser/issues/151
2. Added arg 'debug' to parser, works same way as 'silent' - to get more clear error output.
2. Added arg 'debug' to parser, works same way as 'silent' - to get more clear error output.

New features:
1. Added support for ORACLE 'ORGANIZATION INDEX'
1. Added support for ORACLE 'ORGANIZATION INDEX'
2. Added support for SparkSQL Partition by with procedure call - https://github.com/xnuinside/simple-ddl-parser/issues/154
3. Added support for DEFAULT CHARSET statement MySQL - https://github.com/xnuinside/simple-ddl-parser/issues/153

Expand Down Expand Up @@ -165,7 +168,7 @@ Improvements:
New features:
1. Support SparkSQL USING - https://github.com/xnuinside/simple-ddl-parser/issues/117
Updates initiated by ticket https://github.com/xnuinside/simple-ddl-parser/issues/120:
2. In Parser you can use argument json_dump=True in method .run() if you want get result in JSON format.
2. In Parser you can use argument json_dump=True in method .run() if you want get result in JSON format.
- README updated

Fixes:
Expand All @@ -184,7 +187,7 @@ Fixes:
1. Added flag to raise errors if parser cannot parse statement DDLParser(.., silent=False) - https://github.com/xnuinside/simple-ddl-parser/issues/109
2. Added flag to DDLParser(.., normalize_names=True) that change output of parser:
if flag is True (default 'False') then all identifiers will be returned without '[', '"' and other delimiters that used in different SQL dialects to separate custom names from reserved words & statements.
For example, if flag set 'True' and you pass this input:
For example, if flag set 'True' and you pass this input:

CREATE TABLE [dbo].[TO_Requests](
[Request_ID] [int] IDENTITY(1,1) NOT NULL,
Expand All @@ -201,30 +204,30 @@ In output you will have names like 'dbo' and 'TO_Requests', not '[dbo]' and '[TO
**v0.24.1**

## Fixes:
### HQL:
### HQL:
1. fields_terminated_by now parses , as "','", not as '' previously

### Common:
1. To output added 'if_not_exists' field in result to get availability 1-to-1 re-create ddl by metadata.
1. To output added 'if_not_exists' field in result to get availability 1-to-1 re-create ddl by metadata.

**v0.24.0**

## Fixes:

### HQL:
### HQL:

1. More then 2 tblproperties now are parsed correctly https://github.com/xnuinside/simple-ddl-parser/pull/104
1. More then 2 tblproperties now are parsed correctly https://github.com/xnuinside/simple-ddl-parser/pull/104


### Common:

2. 'set' in lower case now also parsed validly.
3. Now names like 'schema', 'database', 'table' can be used as names in CREATE DATABASE | SCHEMA | TABLESPACE | DOMAIN | TYPE statements and after INDEX and CONSTRAINT.
3. Now names like 'schema', 'database', 'table' can be used as names in CREATE DATABASE | SCHEMA | TABLESPACE | DOMAIN | TYPE statements and after INDEX and CONSTRAINT.
4. Creation of empty tables also parsed correctly (like CREATE Table table;).

## New Statements Support:
## New Statements Support:

### HQL:
### HQL:
1. Added support for CLUSTERED BY - https://github.com/xnuinside/simple-ddl-parser/issues/103
2. Added support for INTO ... BUCKETS
3. CREATE REMOTE DATABASE | SCHEMA
Expand All @@ -235,7 +238,7 @@ Big refactoring: less code complexity & increase code coverage. Radon added to p

### Fixes:

1. Fix for issue with ALTER UNIQUE - https://github.com/xnuinside/simple-ddl-parser/issues/101
1. Fix for issue with ALTER UNIQUE - https://github.com/xnuinside/simple-ddl-parser/issues/101

### New Features

Expand All @@ -262,8 +265,8 @@ PostgreSQL:
### Fixes:

### BigQuery:
1. CREATE TABLE statement with 'project_id' in format like project.dataset.table_name now is parsed validly.
'project' added to output.
1. CREATE TABLE statement with 'project_id' in format like project.dataset.table_name now is parsed validly.
'project' added to output.
Also added support project.dataset.name format in CREATE SCHEMA and ALTER statement


Expand Down Expand Up @@ -293,7 +296,7 @@ I started to add partial support for BigQuery

## MSSQL:

1. Added support for PRIMARY KEY CLUSTERED - full details about clusterisation are parsed now in separate key 'clustered_primary_key'.
1. Added support for PRIMARY KEY CLUSTERED - full details about clusterisation are parsed now in separate key 'clustered_primary_key'.
I don't like that but when I started I did not thought about all those details, so in version 1.0.* I will work on more beautiful and logically output structure.
https://github.com/xnuinside/simple-ddl-parser/issues/91

Expand All @@ -302,11 +305,11 @@ Pay attention: previously they parsed somehow, but in incorrect structure.
### Improvements:

1. Strings in double quotes moved as separate token from ID to fix a lot of issues with strings with spaces inside
2. Now parser can parse statements separated by new line also (without GO or ; at the end of statement) - https://github.com/xnuinside/simple-ddl-parser/issues/90
2. Now parser can parse statements separated by new line also (without GO or ; at the end of statement) - https://github.com/xnuinside/simple-ddl-parser/issues/90

### Fixes:

1. Now open strings is not valid in checks (previously the was parsed.) Open string sample 'some string (exist open quote, but there is no close quote)
1. Now open strings is not valid in checks (previously the was parsed.) Open string sample 'some string (exist open quote, but there is no close quote)
2. Order like ASC, DESK in primary keys now parsed valid (not as previously as column name)

**v0.21.2**
Expand All @@ -323,13 +326,13 @@ Fixies:

## MSSQL:

1. Added support for statements:
1. Added support for statements:
1. PERIOD FOR SYSTEM_TIME in CREATE TABLE statement
2. ON [PRIMARY] after CREATE TABLE statement (sample in test files test_mssql_specific.py)
3. WITH statement for TABLE properties
4. TEXTIMAGE_ON statement
5. DEFAULT NEXT VALUE FOR in COLUMN DEFAULT

2. Added support for separating tables DDL by 'GO' statement as in output of MSSQL
3. Added support for CREATE TYPE as TABLE

Expand Down
28 changes: 28 additions & 0 deletions simple_ddl_parser/dialects/sql.py
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,15 @@ def p_column(self, p: List) -> None:
| column LP id COMMA id RP
| column LP id COMMA id RP c_type
"""
if p[1] == "KEY":
# This is an index
p[0] = {"index_stmt": True, "name": p[2]["type"], "columns": ""}
return
if p[1] and isinstance(p[1], dict) and p[1].get("index_stmt") is True:
p[1]["columns"] = remove_par(list(p))[2]
p[0] = p[1]
return

p[0] = self.set_base_column_propery(p)
identity = False
p_list = list(p)
Expand Down Expand Up @@ -853,6 +862,25 @@ def p_expression_table(self, p: List) -> None: # noqa R701
if not p[0].get("columns"):
p[0]["columns"] = []
p[0]["columns"].append(p_list[-1])
elif "index_stmt" in p_list[-1]:
if not p[0].get("index"):
p[0]["index"] = []
index_data = p_list[-1]
p[0]["index"].append(
{
"clustered": False,
"columns": [index_data["columns"]],
"detailed_columns": [
{
"name": index_data["columns"],
"nulls": "LAST",
"order": "ASC",
}
],
"index_name": index_data["name"],
"unique": False,
}
)
elif "check" in p_list[-1]:
p[0] = self.extract_check_data(p, p_list)
elif "enforced" in p_list[-1]:
Expand Down
Loading

0 comments on commit e60c7ee

Please sign in to comment.