diff --git a/AddingCapabilities.md b/AddingCapabilities.md new file mode 100644 index 0000000..827a2e2 --- /dev/null +++ b/AddingCapabilities.md @@ -0,0 +1,146 @@ +# Adding capabilities + +## Sync parser code with cloud spanner emulator + +1) Clone the Cloud Spanner Emulator repo from + +`https://github.com/GoogleCloudPlatform/cloud-spanner-emulator` + +1) Generate the `ddl_keywords.jjt` file in backend/schema/parser module in order +to build the `ddl_keywords.jjt` + +`bazel build backend/schema/parser:ddl_keywords_jjt` + +1) Compare and synchronize the generated `ddl_keywords.jjt` +in `bazel-bin/bazel-bin/backend/schema/parser/` with +the [ddl_keywords.jjt](src%2Fmain%2Fjjtree-sources%2Fddl_keywords.jjt) in +this repo + +3) Compare and synchronise following files +in [src/main/jjtree-sources](src%2Fmain%2Fjjtree-sources) with the emulator's +versions in +the [backend/schema/parser](https://github.com/GoogleCloudPlatform/cloud-spanner-emulator/tree/master/backend/schema/parser) +directory) +* `ddl_expression.jjt` +* `ddl_parser.jjt` +* `ddl_string_bytes_tokens.jjt` +* `ddl_whitespace.jjt` - Note that this file does not have +the `ValidateStringLiteral()` +and `ValidateBytesLiteral()` functions - this is intentional. + +## Invalidate AST wrappers that have been added + +1) Look at the differences in `ddk_parser.jjt` + +2) Make notes of all the statements and parameters that have been added + +4) Run `mvn clean compile` + +This will compile the JJT files, and generate any missing AST classes +in `target/generated-sources/jjtree` + +6) For each parser capability that has been added, do the following +(A new parser capability may be a new top-level statement, or a table or +column option): + +* Copy the AST java file + to `src/main/java/com/google/cloud/solutions/spannerddl/parser` +* Clean the parser prefix/suffix comments, and add an Apache II licence + header +* Add the following code to the constructors: \ + `throw new UnsupportedOperationException("Not Implemented");` + +Doing this for all the newly added parser classes ensures that the parser +will fail when parsing DDL with these statements, rather than generating +invalid output. + +You do not necessarily have to do this for all classes, but for ones which +add new high level parser capabilities such as top level statements, table +options, etc. + +5) Optionally, but recommended add test cases to the `DDLParserTest` class to +verify that these unsupported +operations fail during parsing. + +## Run a test and fix bugs + +The changes to the parser may have introduced new AST classes that the existing +parser code may not be expecting. Run a `mvn clean verify` in order to re-run +the tests. + +## Implement the toString and equals methods for the new AST capability classes + +The `toString()` and probably the `equals()` methods need to be implemented in +the new AST classes for implementing the capability. + +For end-branches of the AST, the toString() method can be as simple as +regenerating the original tokens using the helper method: + +```java +ASTTreeUtils.tokensToString(node.firstToken,node.lastToken); +``` + +and the equals method can do a string compare... + +However for more complex classes, like Tables, the implementation is necessarily +more complex, extracting the optional and repeated nodes (eg for Table, columns, +constraints, primary key, interleave etc), and then rebuilding the original +statement in the toString() + +If you only implement some of the functionality, the `toString()` method is a +good place to put some validation - checking only for supported child nodes. + +Once this is done, you can run some tests in the `DDLParserTest` class to verify +that the parser works, and that the toString() method regenerates the original +statement. + +## Implement difference generation. + +Once you have a valid `equals()` method, the bulk of the work is handled +in `DDLDiff.build()` The DDL is split into its components, and Maps.difference() +is used to compare. + +If you have new top-level types, then they should be added here. + +If you have new table capabilities, then they can usually be added inline when +creating the table, or by ALTER statements. These are handled by extracting the +inline statements from the create table, and then assuming everything is an +ALTER, so the comparison is made on the ALTER statements (see the handling of +constraints, foreign keys and row deletion policies). + +Then you can add code to generate the DDL statements for the differences - +adding new items, dropping old ones, and replacing existing ones. This is done +in `DdlDiff.generateDifferenceStatements()`. The order of things is very +important in this function as some statements are dependent on each other. in +general the order is: + +* Drop DDL objects that have been removed in order of dependency + * constraints + * foreign keys + * tables - in reverse order in which they were created +* Create new DDL objects in order of dependency + * tables in order of which they were created + * foreign keys + * constraints +* Update existing objects, by dropping and recreating them. + +Care must be taken when updating that a long running operation is not +triggered - eg for indexes, foreign keys. If this is possible then add a command +line option to prevent this step. + +## Add diff generation tests + +The easiest way to add tests is using the files in the `test/resources` +subdirectory. These have 3 files containing blocks with original, new and +expected diff statements. Adding new test cases should be straightforward, and +add as many test cases that you can think of. + +Normally you will want tests for: + +* Adding a DDL feature/object in the new DDL +* Removing a DDL feature/object in the new DDL +* Changing a DDL feature/object. + +For a DDL object like a constraint that can be added inline in a Create Table or +by an Alter statement, you will need to add multiple versions of the add/remove +tests to handle each case. diff --git a/README.md b/README.md index 795f000..7444d16 100644 --- a/README.md +++ b/README.md @@ -42,8 +42,8 @@ statements in the DDL file. This has the following implications: * Tables and indexes must be created with a single `CREATE` statement (not by using `CREATE` then `ALTER` statements). The exception to this is when - foreign key constraints are created - the tool supports creating them in the - table creation DDL statement, and also using `ALTER` statements. + constraints and row deletion policies are created - the tool supports creating them in the + table creation DDL statement, and also by using `ALTER` statements after the table has been created. ### Note on dropped database objects. @@ -57,8 +57,8 @@ differences found: * Drop Indexes. * Drop Columns in a Table. -Foreign key constraints will always be dropped if they are not present in the -new DDL. +Constraints and row deletion policies will always be dropped if they are not +present in the new DDL. ### Note on modified indexes @@ -102,12 +102,26 @@ foreign key relationship is which would also be dropped and recreated. Therefore this option is disabled by default, and FOREIGN KEY differences will cause the tool to fail. +## Unsupported Spanner DDL features + +This tool by neccessity will lag behind the implementation of new DDL features +in Spanner. If you need this tool to support a specific feature, please log an +issue, or [implement it yourself](AddingCapabilities.md) and submit a Pull +Request. + +As of the last edit of this file, the features known not to be supported are: + +* Change Streams (create, drop, alter) +* `ALTER DATABASE` statements +* Default column values (`DEFAULT` clause) +* Views (create, drop, alter) + ## Usage: ### Prerequisites: Install a [JAVA development kit](https://jdk.java.net/) (supporting Java 8 -or above) and [Apache Maven])(https://maven.apache.org/) +or above) and [Apache Maven](https://maven.apache.org/) There are 2 options for running the tool: either compiling and running from source, or by building a runnable JAR with all dependencies included and then @@ -319,14 +333,14 @@ java -jar target/spanner-ddl-diff-*-jar-with-dependencies.jar \ # Apply alter statements to the database. gcloud spanner databases ddl update "${SPANNER_DATABASE}" --instance="${SPANNER_INSTANCE}" \ - --ddl="$(cat /tmp/alter.ddl)" + --ddl-file=/tmp/alter.ddl ``` ## License ``` -Copyright 2020 Google LLC +Copyright 2023 Google LLC Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/diff/DdlDiff.java b/src/main/java/com/google/cloud/solutions/spannerddl/diff/DdlDiff.java index af8a4e3..4486995 100644 --- a/src/main/java/com/google/cloud/solutions/spannerddl/diff/DdlDiff.java +++ b/src/main/java/com/google/cloud/solutions/spannerddl/diff/DdlDiff.java @@ -18,6 +18,7 @@ import static java.nio.charset.StandardCharsets.UTF_8; +import com.google.cloud.solutions.spannerddl.parser.ASTadd_row_deletion_policy; import com.google.cloud.solutions.spannerddl.parser.ASTalter_table_statement; import com.google.cloud.solutions.spannerddl.parser.ASTcheck_constraint; import com.google.cloud.solutions.spannerddl.parser.ASTcolumn_def; @@ -27,12 +28,14 @@ import com.google.cloud.solutions.spannerddl.parser.ASTddl_statement; import com.google.cloud.solutions.spannerddl.parser.ASTforeign_key; import com.google.cloud.solutions.spannerddl.parser.ASToptions_clause; +import com.google.cloud.solutions.spannerddl.parser.ASTrow_deletion_policy_clause; import com.google.cloud.solutions.spannerddl.parser.DdlParser; import com.google.cloud.solutions.spannerddl.parser.DdlParserTreeConstants; import com.google.cloud.solutions.spannerddl.parser.ParseException; import com.google.cloud.solutions.spannerddl.parser.SimpleNode; import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Joiner; +import com.google.common.base.Strings; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.MapDifference; @@ -48,7 +51,9 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Map.Entry; import java.util.Objects; +import java.util.Optional; import java.util.TreeMap; import org.apache.commons.cli.CommandLine; import org.apache.commons.cli.DefaultParser; @@ -93,6 +98,7 @@ public class DdlDiff { private final MapDifference constraintDifferences; private final Map newTablesCreationOrder; private final Map originalTablesCreationOrder; + private final MapDifference ttlDifferences; private static class ConstraintWrapper { @@ -132,12 +138,14 @@ private DdlDiff( Map originalTablesCreationOrder, Map newTablesCreationOrder, MapDifference indexDifferences, - MapDifference constraintDifferences) { + MapDifference constraintDifferences, + MapDifference ttlDifferences) { this.tableDifferences = tableDifferences; this.originalTablesCreationOrder = originalTablesCreationOrder; this.newTablesCreationOrder = newTablesCreationOrder; this.indexDifferences = indexDifferences; this.constraintDifferences = constraintDifferences; + this.ttlDifferences = ttlDifferences; } public List generateDifferenceStatements(Map options) @@ -191,6 +199,11 @@ public List generateDifferenceStatements(Map options) + fkDiff.leftValue().getName()); } + // Drop deleted TTLs + for (String tableName : ttlDifferences.entriesOnlyOnLeft().keySet()) { + output.add("ALTER TABLE " + tableName + " DROP ROW DELETION POLICY"); + } + if (allowDropStatements) { // Drop tables that have been deleted -- need to do it in reverse creation order. List reverseOrderedTableNames = new ArrayList<>(originalTablesCreationOrder.keySet()); @@ -220,6 +233,22 @@ public List generateDifferenceStatements(Map options) } } + // Create new TTLs + for (Map.Entry newTtl : + ttlDifferences.entriesOnlyOnRight().entrySet()) { + output.add("ALTER TABLE " + newTtl.getKey() + " ADD " + newTtl.getValue()); + } + + // updateExisting TTLs + for (Entry> differentTtl : + ttlDifferences.entriesDiffering().entrySet()) { + output.add( + "ALTER TABLE " + + differentTtl.getKey() + + " REPLACE " + + differentTtl.getValue().rightValue()); + } + // Create new indexes for (ASTcreate_index_statement index : indexDifferences.entriesOnlyOnRight().values()) { LOG.info("Creating new index: {}", index.getIndexName()); @@ -233,9 +262,9 @@ public List generateDifferenceStatements(Map options) output.add(difference.rightValue().toString()); } - // Create new constrants. + // Create new constraints. for (ConstraintWrapper fk : constraintDifferences.entriesOnlyOnRight().values()) { - output.add("ALTER TABLE " + fk.tableName + " ADD " + fk.constraint.toString()); + output.add("ALTER TABLE " + fk.tableName + " ADD " + fk.constraint); } // Re-create modified Foreign Keys. @@ -439,24 +468,30 @@ private static void addColumnDiffs( } static DdlDiff build(String originalDDL, String newDDL) throws DdlDiffException { - List originalStatements = parseDDL(originalDDL); - List newStatements = parseDDL(newDDL); + List originalStatements = parseDDL(Strings.nullToEmpty(originalDDL)); + List newStatements = parseDDL(Strings.nullToEmpty(newDDL)); Map originalTablesCreationOrder = new LinkedHashMap<>(); Map originalIndexes = new TreeMap<>(); Map originalConstraints = new TreeMap<>(); + Map originalTtls = new TreeMap<>(); - separateTablesIndexesConstraints( - originalStatements, originalTablesCreationOrder, originalIndexes, originalConstraints); + separateTablesIndexesConstraintsTtls( + originalStatements, + originalTablesCreationOrder, + originalIndexes, + originalConstraints, + originalTtls); Map originalTablesNameOrder = new TreeMap<>(originalTablesCreationOrder); Map newTablesCreationOrder = new LinkedHashMap<>(); Map newIndexes = new TreeMap<>(); Map newConstraints = new TreeMap<>(); + Map newTtls = new TreeMap<>(); - separateTablesIndexesConstraints( - newStatements, newTablesCreationOrder, newIndexes, newConstraints); + separateTablesIndexesConstraintsTtls( + newStatements, newTablesCreationOrder, newIndexes, newConstraints, newTtls); Map newTablesNameOrder = new TreeMap<>(newTablesCreationOrder); @@ -465,14 +500,16 @@ static DdlDiff build(String originalDDL, String newDDL) throws DdlDiffException originalTablesCreationOrder, newTablesCreationOrder, Maps.difference(originalIndexes, newIndexes), - Maps.difference(originalConstraints, newConstraints)); + Maps.difference(originalConstraints, newConstraints), + Maps.difference(originalTtls, newTtls)); } - private static void separateTablesIndexesConstraints( + private static void separateTablesIndexesConstraintsTtls( List statements, Map tables, Map indexes, - Map constraints) { + Map constraints, + Map ttls) { for (ASTddl_statement statement : statements) { if (statement.jjtGetChild(0) instanceof ASTcreate_table_statement) { ASTcreate_table_statement createTable = @@ -482,29 +519,41 @@ private static void separateTablesIndexesConstraints( tables.put(createTable.getTableName(), createTable.clearConstraints()); // convert embedded constraint statements into wrapper object with table name - // use a single map for all foreign keys, whether created in table or externally + // use a single map for all foreign keys, constraints and row deletion polcies whether + // created in table or externally createTable.getConstraints().values().stream() .map(c -> new ConstraintWrapper(createTable.getTableName(), c)) .forEach(c -> constraints.put(c.getName(), c)); + + final Optional rowDeletionPolicyClause = + createTable.getRowDeletionPolicyClause(); + rowDeletionPolicyClause.ifPresent(rdp -> ttls.put(createTable.getTableName(), rdp)); + } else if (statement.jjtGetChild(0) instanceof ASTcreate_index_statement) { ASTcreate_index_statement createIndex = (ASTcreate_index_statement) statement.jjtGetChild(0); indexes.put(createIndex.getIndexName(), createIndex); - } else if (statement.jjtGetChild(0) instanceof ASTalter_table_statement - && - // use a single map for all foreign keys, whether created in table or externally - (statement.jjtGetChild(0).jjtGetChild(1) instanceof ASTforeign_key - || statement.jjtGetChild(0).jjtGetChild(1) instanceof ASTcheck_constraint)) { + } else if (statement.jjtGetChild(0) instanceof ASTalter_table_statement) { + // use a single map for all foreign keys, and constraints and row deletion policies + // whether created in table or externally ASTalter_table_statement alterTable = (ASTalter_table_statement) statement.jjtGetChild(0); - ConstraintWrapper constraint = - new ConstraintWrapper( - alterTable.jjtGetChild(0).toString(), (SimpleNode) alterTable.jjtGetChild(1)); - constraints.put(constraint.getName(), constraint); - } else { - throw new IllegalArgumentException( - "Unsupported statement type: " - + DdlParserTreeConstants.jjtNodeName[statement.jjtGetChild(0).getId()]); + + final String tableName = alterTable.jjtGetChild(0).toString(); + if (alterTable.jjtGetChild(1) instanceof ASTforeign_key + || alterTable.jjtGetChild(1) instanceof ASTcheck_constraint) { + ConstraintWrapper constraint = + new ConstraintWrapper(tableName, (SimpleNode) alterTable.jjtGetChild(1)); + constraints.put(constraint.getName(), constraint); + + } else if (statement.jjtGetChild(0).jjtGetChild(1) instanceof ASTadd_row_deletion_policy) { + ttls.put( + tableName, (ASTrow_deletion_policy_clause) alterTable.jjtGetChild(1).jjtGetChild(0)); + } else { + throw new IllegalArgumentException( + "Unsupported statement type: " + + DdlParserTreeConstants.jjtNodeName[statement.jjtGetChild(0).getId()]); + } } } } @@ -537,12 +586,13 @@ static List parseDDL(String original) throws DdlDiffException // child 0 = table name // child 1 = alter statement. Only ASTforeign_key is supported if (!(alterTableStatement.jjtGetChild(1) instanceof ASTforeign_key) - && !(alterTableStatement.jjtGetChild(1) instanceof ASTcheck_constraint)) { + && !(alterTableStatement.jjtGetChild(1) instanceof ASTcheck_constraint) + && !(alterTableStatement.jjtGetChild(1) instanceof ASTadd_row_deletion_policy)) { throw new IllegalArgumentException( "Unsupported statement:\n" + statement - + "\nCan only create diffs from 'CREATE TABLE, CREATE INDEX, and " - + "'ALTER TABLE table_name ADD CONSTRAINT' DDL statements"); + + "\nCan only create diffs from 'CREATE TABLE, CREATE INDEX and " + + "'ALTER TABLE table_name ADD ' DDL statements"); } // only foreign key statements here: if (alterTableStatement.jjtGetChild(1) instanceof ASTforeign_key @@ -636,7 +686,7 @@ public static void main(String[] args) { } catch (InvalidPathException e) { System.err.println("Invalid file path: " + e.getInput() + "\n" + e.getReason()); } catch (IOException e) { - System.err.println("Cannot read DDL file: " + e.toString()); + System.err.println("Cannot read DDL file: " + e); } catch (DdlDiffException e) { e.printStackTrace(); } diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTalter_change_stream_statement.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTalter_change_stream_statement.java new file mode 100644 index 0000000..2cc4d52 --- /dev/null +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTalter_change_stream_statement.java @@ -0,0 +1,28 @@ +/* + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.solutions.spannerddl.parser; + +public class ASTalter_change_stream_statement extends SimpleNode { + public ASTalter_change_stream_statement(int id) { + super(id); + throw new UnsupportedOperationException("Not Implemented"); + } + + public ASTalter_change_stream_statement(DdlParser p, int id) { + super(p, id); + throw new UnsupportedOperationException("Not Implemented"); + } +} diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTalter_table_statement.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTalter_table_statement.java index b9b43c1..b5384be 100644 --- a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTalter_table_statement.java +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTalter_table_statement.java @@ -16,6 +16,8 @@ package com.google.cloud.solutions.spannerddl.parser; +import com.google.cloud.solutions.spannerddl.diff.ASTTreeUtils; + /** Abstract Syntax Tree parser object for "alter_table_statement" token */ public class ASTalter_table_statement extends SimpleNode { @@ -26,4 +28,36 @@ public ASTalter_table_statement(int id) { public ASTalter_table_statement(DdlParser p, int id) { super(p, id); } + + @Override + public boolean equals(Object other) { + if (other instanceof ASTalter_table_statement) { + // lazy: compare text rendering. + return this.toString().equals(other.toString()); + } + return false; + } + + @Override + public String toString() { + // perform validation. Supported Alter Table statements are: + // ADD (FOREIGN KEY|CHECK CONSTRAINT|ROW DELETION POLICY) + StringBuilder ret = new StringBuilder(); + ret.append("ALTER TABLE "); + ret.append(jjtGetChild(0)); // tablename + ret.append(" ADD "); + final Node alterTableAction = jjtGetChild(1); + if (alterTableAction instanceof ASTforeign_key) { + ret.append(alterTableAction); + } else if (alterTableAction instanceof ASTcheck_constraint) { + ret.append(alterTableAction); + } else if (alterTableAction instanceof ASTadd_row_deletion_policy) { + ret.append(alterTableAction.jjtGetChild(0)); + } else { + throw new IllegalArgumentException( + "Unrecognised Alter Table action in: " + + ASTTreeUtils.tokensToString(jjtGetFirstToken(), jjtGetLastToken())); + } + return ret.toString(); + } } diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTanalyze_statement.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTanalyze_statement.java new file mode 100644 index 0000000..e8599f9 --- /dev/null +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTanalyze_statement.java @@ -0,0 +1,28 @@ +/* + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.solutions.spannerddl.parser; + +public class ASTanalyze_statement extends SimpleNode { + public ASTanalyze_statement(int id) { + super(id); + throw new UnsupportedOperationException("Not Implemented"); + } + + public ASTanalyze_statement(DdlParser p, int id) { + super(p, id); + throw new UnsupportedOperationException("Not Implemented"); + } +} diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcolumn_default_clause.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcolumn_default_clause.java new file mode 100644 index 0000000..a426067 --- /dev/null +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcolumn_default_clause.java @@ -0,0 +1,28 @@ +/* + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.solutions.spannerddl.parser; + +public class ASTcolumn_default_clause extends SimpleNode { + public ASTcolumn_default_clause(int id) { + super(id); + throw new UnsupportedOperationException("not implemented"); + } + + public ASTcolumn_default_clause(DdlParser p, int id) { + super(p, id); + throw new UnsupportedOperationException("not implemented"); + } +} diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_change_stream_statement.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_change_stream_statement.java new file mode 100644 index 0000000..d8b927a --- /dev/null +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_change_stream_statement.java @@ -0,0 +1,29 @@ +/* + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.solutions.spannerddl.parser; + +public class ASTcreate_change_stream_statement extends SimpleNode { + public ASTcreate_change_stream_statement(int id) { + + super(id); + throw new UnsupportedOperationException("Not Implemented"); + } + + public ASTcreate_change_stream_statement(DdlParser p, int id) { + super(p, id); + throw new UnsupportedOperationException("Not Implemented"); + } +} diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_database_statement.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_database_statement.java new file mode 100644 index 0000000..f44dab7 --- /dev/null +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_database_statement.java @@ -0,0 +1,28 @@ +/* + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.solutions.spannerddl.parser; + +public class ASTcreate_database_statement extends SimpleNode { + public ASTcreate_database_statement(int id) { + super(id); + throw new UnsupportedOperationException("Not Implemented"); + } + + public ASTcreate_database_statement(DdlParser p, int id) { + super(p, id); + throw new UnsupportedOperationException("Not Implemented"); + } +} diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_table_statement.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_table_statement.java index 9c829ec..6d8d340 100644 --- a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_table_statement.java +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_table_statement.java @@ -19,13 +19,13 @@ import com.google.cloud.solutions.spannerddl.diff.ASTTreeUtils; import com.google.common.base.Joiner; import java.util.ArrayList; -import java.util.Comparator; import java.util.LinkedHashMap; import java.util.List; import java.util.Optional; /** Abstract Syntax Tree parser object for "create_table_statement" token */ public class ASTcreate_table_statement extends SimpleNode { + private boolean withConstraints = true; public ASTcreate_table_statement(int id) { @@ -78,6 +78,15 @@ public synchronized Optional getInterleaveClause() { } } + public synchronized Optional getRowDeletionPolicyClause() { + try { + return Optional.of( + ASTTreeUtils.getChildByType(children, ASTrow_deletion_policy_clause.class)); + } catch (IllegalArgumentException e) { + return Optional.empty(); + } + } + public ASTcreate_table_statement clearConstraints() { this.withConstraints = false; return this; @@ -92,8 +101,7 @@ public String toString() { ret.append(getTableName()); ret.append(" ("); // append column and constraint definitions. - List tableElements = new ArrayList<>(); - tableElements.addAll(getColumns().values()); + List tableElements = new ArrayList<>(getColumns().values()); if (this.withConstraints) { tableElements.addAll(getConstraints().values()); @@ -107,6 +115,12 @@ public String toString() { ret.append(", "); ret.append(interleaveClause.get()); // interleave optional } + Optional rowDeletionPolicyClause = getRowDeletionPolicyClause(); + if (this.withConstraints && rowDeletionPolicyClause.isPresent()) { + ret.append(", "); + ret.append(rowDeletionPolicyClause.get()); // TTL optional + } + return ret.toString(); } @@ -117,16 +131,14 @@ private void verifyTableElements() { && !(child instanceof ASTname) && !(child instanceof ASTcheck_constraint) && !(child instanceof ASTprimary_key) - && !(child instanceof ASTtable_interleave_clause)) { + && !(child instanceof ASTtable_interleave_clause) + && !(child instanceof ASTrow_deletion_policy_clause)) { throw new IllegalArgumentException( "Unknown child type " + child.getClass().getSimpleName() + " - " + child); } } } - public static Comparator COMPARE_BY_NAME = - Comparator.comparing(ASTcreate_table_statement::getTableName); - @Override public boolean equals(Object other) { if (other instanceof ASTcreate_table_statement) { diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_view_statement.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_view_statement.java new file mode 100644 index 0000000..3310d8d --- /dev/null +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTcreate_view_statement.java @@ -0,0 +1,28 @@ +/* + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.solutions.spannerddl.parser; + +public class ASTcreate_view_statement extends SimpleNode { + public ASTcreate_view_statement(int id) { + super(id); + throw new UnsupportedOperationException("Not Implemented"); + } + + public ASTcreate_view_statement(DdlParser p, int id) { + super(p, id); + throw new UnsupportedOperationException("Not Implemented"); + } +} diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTdrop_row_deletion_policy.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTdrop_row_deletion_policy.java new file mode 100644 index 0000000..8c84f85 --- /dev/null +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTdrop_row_deletion_policy.java @@ -0,0 +1,28 @@ +/* + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.solutions.spannerddl.parser; + +public class ASTdrop_row_deletion_policy extends SimpleNode { + public ASTdrop_row_deletion_policy(int id) { + super(id); + throw new UnsupportedOperationException("Not Implemented"); + } + + public ASTdrop_row_deletion_policy(DdlParser p, int id) { + super(p, id); + throw new UnsupportedOperationException("Not Implemented"); + } +} diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTdrop_statement.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTdrop_statement.java index d9d80bf..c88c372 100644 --- a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTdrop_statement.java +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTdrop_statement.java @@ -21,9 +21,11 @@ public class ASTdrop_statement extends SimpleNode { public ASTdrop_statement(int id) { super(id); + throw new UnsupportedOperationException("Not Implemented"); } public ASTdrop_statement(DdlParser p, int id) { super(p, id); + throw new UnsupportedOperationException("Not Implemented"); } } diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTreplace_row_deletion_policy.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTreplace_row_deletion_policy.java new file mode 100644 index 0000000..a1082cb --- /dev/null +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTreplace_row_deletion_policy.java @@ -0,0 +1,28 @@ +/* + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.solutions.spannerddl.parser; + +public class ASTreplace_row_deletion_policy extends SimpleNode { + public ASTreplace_row_deletion_policy(int id) { + super(id); + throw new UnsupportedOperationException("Not Implemented"); + } + + public ASTreplace_row_deletion_policy(DdlParser p, int id) { + super(p, id); + throw new UnsupportedOperationException("Not Implemented"); + } +} diff --git a/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTrow_deletion_policy_clause.java b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTrow_deletion_policy_clause.java new file mode 100644 index 0000000..53cafac --- /dev/null +++ b/src/main/java/com/google/cloud/solutions/spannerddl/parser/ASTrow_deletion_policy_clause.java @@ -0,0 +1,48 @@ +/* + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.google.cloud.solutions.spannerddl.parser; + +import com.google.cloud.solutions.spannerddl.diff.ASTTreeUtils; + +/** + * This defines the row deletion policy used in: Create Table, Alter Table add ROW DELETION POLICY + * etc. + */ +public class ASTrow_deletion_policy_clause extends SimpleNode { + public ASTrow_deletion_policy_clause(int id) { + super(id); + } + + public ASTrow_deletion_policy_clause(DdlParser p, int id) { + super(p, id); + } + + @Override + public String toString() { + ASTrow_deletion_policy_expression expression = + ASTTreeUtils.getChildByType(children, ASTrow_deletion_policy_expression.class); + return "ROW DELETION POLICY (" + + ASTTreeUtils.tokensToString(expression.firstToken, expression.lastToken) + + ")"; + } + + @Override + public boolean equals(Object other) { + // use text comparison + return (other instanceof ASTrow_deletion_policy_clause + && this.toString().equals(other.toString())); + } +} diff --git a/src/main/jjtree-sources/ddl_expression.jjt b/src/main/jjtree-sources/ddl_expression.jjt index c6d7e20..1b4cbe1 100644 --- a/src/main/jjtree-sources/ddl_expression.jjt +++ b/src/main/jjtree-sources/ddl_expression.jjt @@ -39,12 +39,10 @@ void googlesql_operator() : | "^" | "&" } - -void googlesql_punctuation_no_paren() : +void googlesql_punctuation_no_paren_without_dot() : {} { "," - | "." | "[" | "]" | ":" @@ -54,7 +52,14 @@ void googlesql_punctuation_no_paren() : | "?" } -void googlesql_statement_token_no_paren() : +void googlesql_punctuation_no_paren() : +{} +{ + googlesql_punctuation_no_paren_without_dot() + | "." +} + +void googlesql_statement_token_no_paren_without_dot() : {} { ( @@ -63,5 +68,13 @@ void googlesql_statement_token_no_paren() : | identifier() | any_reserved_word() | googlesql_operator() - | googlesql_punctuation_no_paren() ) + | googlesql_punctuation_no_paren_without_dot() ) +} + +void googlesql_statement_token_no_paren() : +{} +{ + ( googlesql_statement_token_no_paren_without_dot() + | "." ) } + diff --git a/src/main/jjtree-sources/ddl_keywords.jjt b/src/main/jjtree-sources/ddl_keywords.jjt index c3c5c3a..01a1334 100755 --- a/src/main/jjtree-sources/ddl_keywords.jjt +++ b/src/main/jjtree-sources/ddl_keywords.jjt @@ -46,7 +46,7 @@ TOKEN: | | | - | + | | | | @@ -101,7 +101,7 @@ TOKEN: | | | - | + | | | | @@ -118,36 +118,68 @@ TOKEN: | | + | | + | | | + | | | | | | + | | + | | + | + | | | + | + | | + | + | | | + | | | + | | + | | | | + | | + | + | | | + | + | + | + | + | + | + | + | + | + | + | | | + | | | | + | | + | + | } void pseudoReservedWord() #void : @@ -156,36 +188,68 @@ void pseudoReservedWord() #void : | | + | | + | | | + | | | | | | + | | + | | + | + | | | + | + | | + | + | | | + | | | + | | + | | | | + | | + | + | | | + | + | + | + | + | + | + | + | + | + | + | | | + | | | | + | | + | + | } void any_reserved_word() : @@ -221,7 +285,7 @@ void any_reserved_word() : | | | - | + | | | | @@ -276,7 +340,7 @@ void any_reserved_word() : | | | - | + | | | | diff --git a/src/main/jjtree-sources/ddl_parser.jjt b/src/main/jjtree-sources/ddl_parser.jjt index c2046e7..836ae8a 100644 --- a/src/main/jjtree-sources/ddl_parser.jjt +++ b/src/main/jjtree-sources/ddl_parser.jjt @@ -38,12 +38,24 @@ void identifier() #void : | pseudoReservedWord() } +void qualified_identifier() #void : +{} +{ + ((identifier() #part) (("." identifier() #part)*)) +} + void identifier_list() : {} { (identifier() #identifier) ("," identifier() #identifier)* } +void qualified_identifier_list() : +{} +{ + (qualified_identifier() #identifier) ("," qualified_identifier() #identifier)* +} + void path() : {} { @@ -65,6 +77,7 @@ void ddl_statement() : ( alter_statement() | create_statement() | drop_statement() + | analyze_statement() ) } @@ -74,8 +87,22 @@ void create_statement() #void : { ( create_database_statement() + | create_table_statement() - | create_index_statement() + | LOOKAHEAD(2) create_index_statement() + + | create_or_replace_statement() + | create_change_stream_statement() + ) +} + + +void create_or_replace_statement() : +{} +{ + [ or_replace() ] + ( + create_view_statement() ) } @@ -96,10 +123,12 @@ void create_database_statement() : void create_table_statement() : {} { -
identifier() #name +
+ qualified_identifier() #name "(" [ table_element() ( LOOKAHEAD(2) "," table_element() )* [ "," ] ] ")" primary_key() [ LOOKAHEAD(2) "," table_interleave_clause() ] + [ LOOKAHEAD(2) "," row_deletion_policy_clause() ] } void table_element() #void : @@ -116,7 +145,7 @@ void column_def() : identifier() #name column_type() [ #not_null ] - [ generation_clause() ] + [ generation_clause() | column_default_clause() ] [ options_clause() ] } @@ -125,13 +154,16 @@ void column_def_alter_attrs() : { column_type() [ #not_null ] - [ generation_clause() ] + [ generation_clause() | column_default_clause() ] } void column_def_alter() : {} { - #set_options_clause options_clause() + LOOKAHEAD(2) ( options_clause() + | column_default_clause() + ) + | LOOKAHEAD(2) #drop_column_default | column_def_alter_attrs() } @@ -146,10 +178,14 @@ void column_type() : | | | + | "." qualified_identifier() #pgtype | | "<" column_type() ">" + + } + void column_length() : {} { @@ -163,6 +199,12 @@ void generation_clause() : [ #stored] } +void column_default_clause() : +{} +{ + "(" expression() #column_default_expression ")" +} + void primary_key() #void : {} { @@ -175,10 +217,11 @@ void foreign_key() : [ identifier() #constraint_name ] "(" identifier_list() #referencing_columns ")" - identifier() #referenced_table + qualified_identifier() #referenced_table "(" identifier_list() #referenced_columns ")" } + void statement_token_no_paren() : {} { @@ -195,10 +238,29 @@ void check_constraint() : void table_interleave_clause() : {} { - identifier() #interleave_in + qualified_identifier() #interleave_in [ on_delete_clause() ] } +void row_deletion_policy_clause(): +{} +{ + "(" row_deletion_policy_expression() ")" +} + +void row_deletion_policy_expression(): +{} +{ + identifier() #row_deletion_policy_function + "(" identifier() #row_deletion_policy_column "," interval_expression() ")" +} + +void interval_expression(): +{} +{ + int_value() +} + void on_delete_clause() : {} { @@ -210,19 +272,56 @@ void create_index_statement() : { [ #unique_index ] [ #null_filtered ] - identifier() #name - identifier() #table + + qualified_identifier() #name + qualified_identifier() #table key() #columns [ stored_column_list() ] [ LOOKAHEAD(2) "," index_interleave_clause() ] } + +void create_change_stream_statement() : +{} +{ + qualified_identifier() #name + [ change_stream_for_clause() ] + [ options_clause() ] +} + +void change_stream_for_clause() : +{} +{ + + ( #all + | change_stream_tracked_tables() + ) +} + +void change_stream_tracked_tables() : +{} +{ + change_stream_tracked_tables_entry() + ( "," change_stream_tracked_tables_entry() )* +} + +void change_stream_tracked_tables_entry() : +{} +{ + qualified_identifier() #table + [ ( + "(" [ (identifier() #column) ( "," identifier() #column )* ] ")" + ) #explicit_columns // Needed to distinguish between `Table` and `Table()` + ] +} + void index_interleave_clause() : {} { - identifier() #interleave_in + qualified_identifier() #interleave_in } + void key() #void : {} { @@ -271,50 +370,105 @@ void option_key_val() : identifier() #key "=" ( #nulll - | #bool_true_val - | #bool_false_val + | #bool_true_val + | #bool_false_val | #integer_val - | string_value() #str_val + | string_value() #str_val ) } -void drop_statement() : +void or_replace() : {} { - (
#table | #index ) identifier() #name + } -void alter_statement() #void : +void sql_security_invoker() : {} { - - ( alter_database_statement() - | alter_table_statement() - ) + +} + +void create_view_statement() : +{} +{ + ( ) + ( qualified_identifier() #name ) + [ sql_security_invoker() ] + statement_tokens() #view_definition +} + +void statement_tokens() : +{} +{ + ( statement_token_no_paren() + | "(" + | ")" )+ } -void alter_database_statement() : +void drop_statement() : {} { - (identifier() #database_name) options_clause() + + + (
#table + | #index + | #view + | #change_stream + ) qualified_identifier() #name + + + +} + +void alter_statement() #void : +{} +{ + + ( + alter_table_statement() + | alter_change_stream_statement() + + ) } void alter_table_statement() : {} { -
(identifier() #table_name) - ( LOOKAHEAD(3) #add_column [LOOKAHEAD(2) ] column_def() - | LOOKAHEAD(3) #drop_constraint - identifier() #constraint_name - | #drop_column [LOOKAHEAD(2) ] (identifier() #column_name) - | LOOKAHEAD(3) #alter_column [LOOKAHEAD(2) ] +
(qualified_identifier() #table_name) + ( LOOKAHEAD(3) #drop_constraint identifier() #constraint_name + | LOOKAHEAD(3) #drop_row_deletion_policy + | LOOKAHEAD(3) #drop_column [LOOKAHEAD(2) ] + (identifier() #column_name) + | LOOKAHEAD(1) #alter_column [LOOKAHEAD(2) ] (identifier() #name) column_def_alter() | #set_on_delete on_delete_clause() | LOOKAHEAD(4) foreign_key() - | check_constraint() + | LOOKAHEAD(4) check_constraint() + | LOOKAHEAD(4) row_deletion_policy_clause() #add_row_deletion_policy + | LOOKAHEAD(1) #add_column [LOOKAHEAD(6) + ] column_def() + | row_deletion_policy_clause() #replace_row_deletion_policy + ) +} + +void alter_change_stream_statement() : +{} +{ + (qualified_identifier() #name) + ( LOOKAHEAD(2) change_stream_for_clause() + | options_clause() + | #drop_for_all ) } +void analyze_statement() : +{} +{ + +} + + TOKEN: { diff --git a/src/main/jjtree-sources/readme.md b/src/main/jjtree-sources/readme.md index a14af82..fceaece 100644 --- a/src/main/jjtree-sources/readme.md +++ b/src/main/jjtree-sources/readme.md @@ -19,15 +19,20 @@ that call `ValidateBytesLiteral()` and `ValidateStringBytesLiteral()` ```shell # Build ddl_keywords.jjt -sudo apt install bazel-2.2.0 +sudo apt install bazel-5.4.0 git clone https://github.com/GoogleCloudPlatform/cloud-spanner-emulator.git cd cloud-spanner-emulator -bazel-2.2.0 build backend/schema/parser:ddl_keywords_jjt +bazel build backend/schema/parser:ddl_keywords_jjt + +# generated file in bazel-bin/backend/schema/parser/ddl_keywords.jjt + # Compare JJT files diff bazel-bin/backend/schema/parser/ddl_keywords.jjt ../spanner-schema-diff-tool/src/main/jjtree-sources/ddl_keywords.jjt +diff backend/schema/parser/ddl_expression.jjt ../spanner-schema-diff-tool/src/main/jjtree-sources/ddl_expression.jjt diff backend/schema/parser/ddl_parser.jjt ../spanner-schema-diff-tool/src/main/jjtree-sources/ddl_parser.jjt diff backend/schema/parser/ddl_string_bytes_tokens.jjt ../spanner-schema-diff-tool/src/main/jjtree-sources/ddl_string_bytes_tokens.jjt +diff backend/schema/parser/ddl_whitespace.jjt ../spanner-schema-diff-tool/src/main/jjtree-sources/ddl_whitespace.jjt ``` diff --git a/src/test/java/com/google/cloud/solutions/spannerddl/diff/DdlDiffTest.java b/src/test/java/com/google/cloud/solutions/spannerddl/diff/DdlDiffTest.java index 33cd4c6..5db3c5d 100644 --- a/src/test/java/com/google/cloud/solutions/spannerddl/diff/DdlDiffTest.java +++ b/src/test/java/com/google/cloud/solutions/spannerddl/diff/DdlDiffTest.java @@ -31,6 +31,7 @@ import java.io.FileReader; import java.io.IOException; import java.util.Arrays; +import java.util.Collections; import java.util.Iterator; import java.util.LinkedHashMap; import java.util.LinkedList; @@ -43,7 +44,7 @@ public class DdlDiffTest { @Test - public void parseDDL() throws DdlDiffException { + public void parseMultiDdlStatements() throws DdlDiffException { String DDL = "create table test1 (col1 int64) primary key (col1);\n\n" + "-- comment ; with semicolon.\n" @@ -70,26 +71,6 @@ public void parseDDL() throws DdlDiffException { assertThat(result.get(3).toString()).isEqualTo("CREATE INDEX index1 ON table1 (col1 ASC)"); } - @Test - public void parseDDLCreateTableSyntaxError() { - parseDdlCheckDdlDiffException( - "Create table test1 ( col1 int64 )", "Was expecting:\n\n\"primary\" ..."); - } - - @Test - public void parseDDLCreateIndexSyntaxError() { - parseDdlCheckDdlDiffException("Create index index1 on test1", "Was expecting:\n\n\"(\" ..."); - } - - private void parseDdlCheckDdlDiffException(String DDL, String exceptionContains) { - try { - DdlDiff.parseDDL(DDL); - fail("Expected DdlDiffException not thrown."); - } catch (DdlDiffException e) { - assertThat(e.getMessage()).contains(exceptionContains); - } - } - @Test public void parseCreateTable_anonForeignKey() throws DdlDiffException { try { @@ -122,55 +103,37 @@ public void parseCreateTable_anonCheckConstraint() throws DdlDiffException { @Test(expected = IllegalArgumentException.class) public void parseDDLNoAlterTableAlterColumn() throws DdlDiffException { - String DDL = "alter table test1 alter column col1 int64 not null"; - DdlDiff.parseDDL(DDL); + DdlDiff.parseDDL("alter table test1 alter column col1 int64 not null"); } @Test(expected = IllegalArgumentException.class) public void parseDDLNoAlterTableAddColumn() throws DdlDiffException { - String DDL = "alter table test1 add column col1 int64 not null"; - DdlDiff.parseDDL(DDL); + DdlDiff.parseDDL("alter table test1 add column col1 int64 not null"); } @Test(expected = IllegalArgumentException.class) public void parseDDLNoAlterTableDropColumn() throws DdlDiffException { - String DDL = "alter table test1 drop column col1"; - DdlDiff.parseDDL(DDL); + DdlDiff.parseDDL("alter table test1 drop column col1"); } @Test(expected = IllegalArgumentException.class) public void parseDDLNoAlterTableDropConstraint() throws DdlDiffException { - String DDL = "alter table test1 drop constraint xxx"; - DdlDiff.parseDDL(DDL); + DdlDiff.parseDDL("alter table test1 drop constraint xxx"); } @Test(expected = IllegalArgumentException.class) public void parseDDLNoAlterTableSetOnDelete() throws DdlDiffException { - String DDL = "alter table test1 set on delete cascade"; - DdlDiff.parseDDL(DDL); + DdlDiff.parseDDL("alter table test1 set on delete cascade"); } + @Test public void parseDDLAlterTableAddConstraint() throws DdlDiffException { - String DDL = "alter table test1 add constraint XXX FOREIGN KEY (yyy) references zzz(xxx)"; - DdlDiff.parseDDL(DDL); + DdlDiff.parseDDL("alter table test1 add constraint XXX FOREIGN KEY (yyy) references zzz(xxx)"); } @Test(expected = IllegalArgumentException.class) public void parseDDLNoAlterTableAddAnonConstraint() throws DdlDiffException { - String DDL = "alter table test1 add FOREIGN KEY (yyy) references zzz(xxx)"; - DdlDiff.parseDDL(DDL); - } - - @Test(expected = IllegalArgumentException.class) - public void parseDDLNoDropTable() throws DdlDiffException { - String DDL = "drop table test1"; - DdlDiff.parseDDL(DDL); - } - - @Test(expected = IllegalArgumentException.class) - public void parseDDLNoDropIndex() throws DdlDiffException { - String DDL = "drop index test1"; - DdlDiff.parseDDL(DDL); + DdlDiff.parseDDL("alter table test1 add FOREIGN KEY (yyy) references zzz(xxx)"); } @Test @@ -376,7 +339,7 @@ public void generateAlterTable_changeInterleaving() throws DdlDiffException { } @Test - public void generateAlterTable_changeGenerationClause() throws DdlDiffException { + public void generateAlterTable_changeGenerationClause() { // remove interleave getTableDiffCheckDdlDiffException( "create table test1 (col1 int64, col2 int64, col3 int64 as ( col1*col2 ) stored) primary" @@ -588,7 +551,7 @@ public void compareDddTextFiles() throws IOException { List expectedDiff = expectedOutput.getValue() != null ? Arrays.asList(expectedOutput.getValue().split("\n")) - : Arrays.asList(); + : Collections.emptyList(); DdlDiff ddlDiff = DdlDiff.build(originalSegment.getValue(), newSegment.getValue()); // Run diff with allowRecreateIndexes and allowDropStatements diff --git a/src/test/java/com/google/cloud/solutions/spannerddl/parser/DDLParserTest.java b/src/test/java/com/google/cloud/solutions/spannerddl/parser/DDLParserTest.java index f83d182..b239653 100644 --- a/src/test/java/com/google/cloud/solutions/spannerddl/parser/DDLParserTest.java +++ b/src/test/java/com/google/cloud/solutions/spannerddl/parser/DDLParserTest.java @@ -17,6 +17,7 @@ package com.google.cloud.solutions.spannerddl.parser; import static com.google.common.truth.Truth.assertThat; +import static org.junit.Assert.fail; import java.io.StringReader; import org.junit.Test; @@ -47,7 +48,8 @@ public void parseCreateTable() throws ParseException { + "constraint check_some_value CHECK ((length(sizedstring)>100 or sizedstring= \"xxx\") AND boolcol= true and intcol > -123.4 and numericcol < 1.5)" + ") " + "primary key (intcol ASC, floatcol desc, boolcol), " - + "interleave in parent other_table on delete cascade ") + + "interleave in parent other_table on delete cascade," + + "row deletion policy (OLDER_THAN(timestampcol, INTERVAL 10 DAY))") .jjtGetChild(0); assertThat(statement.toString()) @@ -69,11 +71,12 @@ public void parseCreateTable() throws ParseException { + "CONSTRAINT fk_col_remote FOREIGN KEY (col1, col2) REFERENCES other_table (other_col1, other_col2), " + "CONSTRAINT check_some_value CHECK (( length ( sizedstring ) > 100 OR sizedstring = \"xxx\" ) AND boolcol = TRUE AND intcol > -123.4 AND numericcol < 1.5)" + ") PRIMARY KEY (intcol ASC, floatcol DESC, boolcol ASC), " - + "INTERLEAVE IN PARENT other_table ON DELETE CASCADE"); + + "INTERLEAVE IN PARENT other_table ON DELETE CASCADE, " + + "ROW DELETION POLICY (OLDER_THAN ( timestampcol, INTERVAL 10 DAY ))"); // Test re-parse of toString output. ASTcreate_table_statement statement2 = - (ASTcreate_table_statement) parse(statement.toString()).jjtGetChild(0); + (ASTcreate_table_statement) parseAndVerifyToString(statement.toString()).jjtGetChild(0); assertThat(statement).isEqualTo(statement2); } @@ -103,10 +106,86 @@ public void parseCreateIndex() throws ParseException { // Test re-parse of toString output. ASTcreate_index_statement statement2 = - (ASTcreate_index_statement) parse(statement.toString()).jjtGetChild(0); + (ASTcreate_index_statement) parseAndVerifyToString(statement.toString()).jjtGetChild(0); assertThat(statement).isEqualTo(statement2); } + @Test + public void parseDDLCreateTableSyntaxError() { + parseCheckingException( + "Create table test1 ( col1 int64 )", "Was expecting:\n\n\"primary\" ..."); + } + + @Test + public void parseDDLCreateIndexSyntaxError() { + parseCheckingException("Create index index1 on test1", "Was expecting one of:\n\n\"(\" ..."); + } + + @Test(expected = UnsupportedOperationException.class) + public void parseDDLNoDropTable() throws ParseException { + parseAndVerifyToString("drop table test1"); + } + + @Test(expected = UnsupportedOperationException.class) + public void parseDDLNoDropIndex() throws ParseException { + parseAndVerifyToString("drop index test1"); + } + + @Test(expected = UnsupportedOperationException.class) + public void parseDDLNoDropChangeStream() throws ParseException { + parseAndVerifyToString("drop change stream test1"); + } + + @Test(expected = UnsupportedOperationException.class) + public void parseDDLNoCreateChangeStream() throws ParseException { + parseAndVerifyToString("Create change stream test1 for test2"); + } + + @Test(expected = UnsupportedOperationException.class) + public void parseDDLNoCreateView() throws ParseException { + parseAndVerifyToString("CREATE VIEW test1 SQL SECURITY INVOKER AS SELECT * from test2"); + } + + @Test(expected = UnsupportedOperationException.class) + public void parseDDLNoCreateorReplaceView() throws ParseException { + parseAndVerifyToString( + "CREATE OR REPLACE VIEW test1 SQL SECURITY INVOKER AS SELECT * from test2"); + } + + @Test + public void parseDDLNoAlterTableRowDeletionPolicy() throws ParseException { + parseAndVerifyToString( + "ALTER TABLE Albums " + + "ADD ROW DELETION POLICY (OLDER_THAN ( timestamp_column, INTERVAL 1 DAY ))"); + } + + @Test(expected = UnsupportedOperationException.class) + public void parseDDLNoAlterTableReplaceRowDeletionPolicy() throws ParseException { + String DDL = + "ALTER TABLE Albums " + + "REPLACE ROW DELETION POLICY (OLDER_THAN(timestamp_column, INTERVAL 1 DAY))"; + parseAndVerifyToString(DDL); + } + + @Test(expected = UnsupportedOperationException.class) + public void parseDDLNoDropRowDeletionPolicy() throws ParseException { + parseAndVerifyToString("ALTER TABLE Albums DROP ROW DELETION POLICY;"); + } + + @Test(expected = UnsupportedOperationException.class) + public void parseDDLNoDefaultValue() throws ParseException { + parseAndVerifyToString("CREATE TABLE test1 ( keycol INT64 DEFAULT (123) ) PRIMARY KEY keycol;"); + } + + private static void parseCheckingException(String ddlStatement, String exceptionContains) { + try { + parseAndVerifyToString(ddlStatement); + fail("Expected ParseException not thrown."); + } catch (ParseException e) { + assertThat(e.getMessage()).contains(exceptionContains); + } + } + private static ASTddl_statement parse(String DDLStatement) throws ParseException { try (StringReader in = new StringReader(DDLStatement)) { DdlParser parser = new DdlParser(in); @@ -114,4 +193,11 @@ private static ASTddl_statement parse(String DDLStatement) throws ParseException return (ASTddl_statement) parser.jjtree.rootNode(); } } + + private static ASTddl_statement parseAndVerifyToString(String DDLStatement) + throws ParseException { + ASTddl_statement node = parse(DDLStatement); + assertThat(node.toString()).isEqualTo(DDLStatement); // validates statement regeneration + return node; + } } diff --git a/src/test/resources/expectedDdlDiff.txt b/src/test/resources/expectedDdlDiff.txt index 81ef516..18e9d00 100644 --- a/src/test/resources/expectedDdlDiff.txt +++ b/src/test/resources/expectedDdlDiff.txt @@ -150,6 +150,44 @@ CREATE TABLE test_fkey (col1 INT64, col2 INT64) PRIMARY KEY (col1 ASC) ALTER TABLE test_fkey ADD CONSTRAINT fk_col1 FOREIGN KEY (col1) REFERENCES test1 (col1) == TEST 25 move foreign key out of create table + # No change -== +== TEST 26 unchanged Row Deletion Policy + +# No change + +== TEST 27 unchanged Row Deletion Policy - moved to ALTER + +# No change + +== TEST 28 drop Row Deletion Policy + +ALTER TABLE test1 DROP ROW DELETION POLICY + +== TEST 29 drop Row Deletion Policy added by ALTER + +ALTER TABLE test1 DROP ROW DELETION POLICY + +== TEST 30 Add Row Deletion Policy + +ALTER TABLE test1 ADD ROW DELETION POLICY (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )) + +== TEST 31 Add Row Deletion Policy by Alter + +ALTER TABLE test1 ADD ROW DELETION POLICY (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )) + +== TEST 32 Change Row Deletion Policy + +ALTER TABLE test1 REPLACE ROW DELETION POLICY (OLDER_THAN ( other_column, INTERVAL 20 DAY )) + +== TEST 32 Change Row Deletion Policy added by Alter + +ALTER TABLE test1 REPLACE ROW DELETION POLICY (OLDER_THAN ( other_column, INTERVAL 20 DAY )) + +== TEST 33 Drop table with Row Deletion Policy + +ALTER TABLE test1 DROP ROW DELETION POLICY +DROP TABLE test1 + +== diff --git a/src/test/resources/newDdl.txt b/src/test/resources/newDdl.txt index b481b02..722e36a 100644 --- a/src/test/resources/newDdl.txt +++ b/src/test/resources/newDdl.txt @@ -238,5 +238,84 @@ create table test1 ( primary key (col1); alter table test1 add constraint fk_in_table foreign key (col1) references othertable(othercol1) +== TEST 26 unchanged Row Deletion Policy + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1), +row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + +== TEST 27 unchanged Row Deletion Policy - moved to ALTER + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1); + +ALTER table test1 ADD row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + +== TEST 28 drop Row Deletion Policy + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1) + +== TEST 29 drop Row Deletion Policy added by ALTER + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1); + +== TEST 30 Add Row Deletion Policy + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1), +row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + +== TEST 31 Add Row Deletion Policy by Alter + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1); +alter table test1 ADD row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )) + +== TEST 32 Change Row Deletion Policy + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1), +row deletion policy (OLDER_THAN ( other_column, INTERVAL 20 DAY )); + +== TEST 32 Change Row Deletion Policy added by Alter + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1); + +Alter table test1 add row deletion policy (OLDER_THAN ( other_column, INTERVAL 20 DAY )); + +== TEST 33 Drop table with Row Deletion Policy + +# nothing here + == + + + diff --git a/src/test/resources/originalDdl.txt b/src/test/resources/originalDdl.txt index d0dc890..ef46525 100644 --- a/src/test/resources/originalDdl.txt +++ b/src/test/resources/originalDdl.txt @@ -241,6 +241,90 @@ create table test1 ( ) primary key (col1); +== TEST 26 unchanged Row Deletion Policy + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1), +row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + +== TEST 27 unchanged Row Deletion Policy - moved to ALTER + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1), +row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + +== TEST 28 drop Row Deletion Policy + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1), +row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + +== TEST 29 drop Row Deletion Policy added by ALTER + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1); + +Alter table test1 add row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + +== TEST 30 Add Row Deletion Policy + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1); + +== TEST 31 Add Row Deletion Policy by Alter + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1); + +== TEST 32 Change Row Deletion Policy + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1), +row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + +== TEST 32 Change Row Deletion Policy added by Alter + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1); + +Alter table test1 add row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + +== TEST 33 Drop table with Row Deletion Policy + +create table test1 ( + col1 int64, + time timestamp +) +primary key (col1), +row deletion policy (OLDER_THAN ( timestamp_column, INTERVAL 10 DAY )); + == + + +