Merge branch 'master' into kennytm/backup-restore-statements

README.md

@@ -1,80 +1,56 @@
-# Parser
+# Parser - A MySQL Compatible SQL Parser
 
-[![Go Report Card](https://goreportcard.com/badge/github.com/pingcap/parser)](https://goreportcard.com/report/github.com/pingcap/parser) [![CircleCI Status](https://circleci.com/gh/pingcap/parser.svg?style=shield)](https://circleci.com/gh/pingcap/parser) [![GoDoc](https://godoc.org/github.com/pingcap/parser?status.svg)](https://godoc.org/github.com/pingcap/parser)
+[![Go Report Card](https://goreportcard.com/badge/github.com/pingcap/parser)](https://goreportcard.com/report/github.com/pingcap/parser)
+[![CircleCI Status](https://circleci.com/gh/pingcap/parser.svg?style=shield)](https://circleci.com/gh/pingcap/parser)
+[![GoDoc](https://godoc.org/github.com/pingcap/parser?status.svg)](https://godoc.org/github.com/pingcap/parser)
 [![codecov](https://codecov.io/gh/pingcap/parser/branch/master/graph/badge.svg)](https://codecov.io/gh/pingcap/parser)
 
-TiDB SQL Parser
+The goal of this project is to build a Golang parser that is fully compatible with MySQL syntax, easy to extend, and high performance. Currently, features supported by parser are as follows:
 
-## How to use it
-
-```go
-import (
-	"fmt"
-	"github.com/pingcap/parser"
-	_ "github.com/pingcap/tidb/types/parser_driver"
-)
-
-// This example show how to parse a text sql into ast.
-func example() {
-
-	// 0. make sure import parser_driver implemented by TiDB(user also can implement own driver by self).
-	// and add `import _ "github.com/pingcap/tidb/types/parser_driver"` in the head of file.
-
-	// 1. Create a parser. The parser is NOT goroutine safe and should
-	// not be shared among multiple goroutines. However, parser is also
-	// heavy, so each goroutine should reuse its own local instance if
-	// possible.
-	p := parser.New()
+- Highly compatible with MySQL: it supports almost all features of MySQL. For the complete details, see [parser.y](https://github.com/pingcap/parser/blob/master/parser.y) and [hintparser.y](https://github.com/pingcap/parser/blob/master/hintparser.y).
+- Extensible: adding a new syntax requires only a few lines of Yacc and Golang code changes. As an example, see [PR-680](https://github.com/pingcap/parser/pull/680/files).
+- Good performance: the parser is generated by goyacc in a bottom-up approach. It is efficient to build an AST tree with a state machine.
 
-	// 2. Parse a text SQL into AST([]ast.StmtNode).
-	stmtNodes, _, err := p.Parse("select * from tbl where id = 1", "", "")
-
-	// 3. Use AST to do cool things.
-	fmt.Println(stmtNodes[0], err)
-}
-```
-
-See [https://godoc.org/github.com/pingcap/parser](https://godoc.org/github.com/pingcap/parser)
-
-## How to update parser for TiDB
-
-Assuming that you want to file a PR (pull request) to TiDB, and your PR includes a change in the parser, follow these steps to update the parser in TiDB.
+## How to use it
 
-### Step 1: Make changes in your parser repository
+Please read the [quickstart](https://github.com/pingcap/parser/blob/master/docs/quickstart.md).
 
-Fork this repository to your own account and commit the changes to your repository.
+## Future
 
-> **Note:**
->
-> - Don't forget to run `make test` before you commit!
-> - Make sure `parser.go` is updated.
+- Support more MySQL syntax
+- Optimize the code structure, make it easier to extend
+- Improve performance and benchmark
+- Improve the quality of code and comments
 
-Suppose the forked repository is `https://github.com/your-repo/parser`.
+## Getting Help
 
-### Step 2: Make your parser changes take effect in TiDB and run CI
+- [GitHub Issue](https://github.com/pingcap/parser/issues)
+- [Stack Overflow](https://stackoverflow.com/questions/tagged/tidb)
+- [User Group (Chinese)](https://asktug.com/)
 
-1. In your TiDB repository, execute the `replace` instruction to make your parser changes take effect:
+If you have any questions, feel free to join and discuss in #sig-ddl channel of [Slack-community](https://pingcap.com/tidbslack/).
 
-    ```
-    GO111MODULE=on go mod edit -replace github.com/pingcap/parser=github.com/your-repo/parser@your-branch
-    ```
+If you want to join as a special interest group member, see [DDL Special Interest Group](https://github.com/pingcap/community/tree/master/special-interest-groups/sig-ddl).
 
-2. `make dev` to run CI in TiDB.
+## Users
 
-3. File a PR to TiDB.
+These projects use this parser. Please feel free to extend this list if you
+found you are one of the users but not listed here:
 
-### Step 3: Merge the PR about the parser to this repository
+- [pingcap/tidb](https://github.com/pingcap/tidb)
+- [XiaoMi/soar](https://github.com/XiaoMi/soar)
+- [XiaoMi/Gaea](https://github.com/XiaoMi/Gaea)
+- [sql-machine-learning/sqlflow](https://github.com/sql-machine-learning/sqlflow)
 
-File a PR to this repository. **Link the related PR in TiDB in your PR description or comment.**
+## Contributing
 
-This PR will be reviewed, and if everything goes well, it will be merged.
+Contributions are welcomed and greatly appreciated. See [CONTRIBUTING.md](https://github.com/pingcap/community/blob/master/CONTRIBUTING.md) for details on submitting patches and the contribution workflow.
 
-### Step 4: Update TiDB to use the latest parser
+Here is how to [update parser for TiDB](https://github.com/pingcap/parser/blob/master/docs/update-parser-for-tidb.md).
 
-In your TiDB pull request, modify the `go.mod` file manually or use this command:
+## Acknowledgments
 
-```
-GO111MODULE=on go get -u github.com/pingcap/parser@master
-```
+Thanks [cznic](https://github.com/cznic) for providing some great open-source tools.
 
-Make sure the `replace` instruction is changed back to the `require` instruction and the version is the latest.
+## License
+Parser is under the Apache 2.0 license. See the LICENSE file for details.

ast/expressions.go

@@ -47,6 +47,7 @@ var (
 	_ ExprNode = &ValuesExpr{}
 	_ ExprNode = &VariableExpr{}
 	_ ExprNode = &MatchAgainst{}
+	_ ExprNode = &SetCollationExpr{}
 
 	_ Node = &ColumnName{}
 	_ Node = &WhenClause{}
@@ -64,7 +65,7 @@ type ValueExpr interface {
 }
 
 // NewValueExpr creates a ValueExpr with value, and sets default field type.
-var NewValueExpr func(interface{}) ValueExpr
+var NewValueExpr func(value interface{}, charset string, collate string) ValueExpr
 
 // NewParamMarkerExpr creates a ParamMarkerExpr.
 var NewParamMarkerExpr func(offset int) ParamMarkerExpr
@@ -1396,3 +1397,43 @@ func (n *MatchAgainst) Accept(v Visitor) (Node, bool) {
 	n.Against = newAgainst.(ExprNode)
 	return v.Leave(n)
 }
+
+// SetCollationExpr is the expression for the `COLLATE collation_name` clause.
+type SetCollationExpr struct {
+	exprNode
+	// Expr is the expression to be set.
+	Expr ExprNode
+	// Collate is the name of collation to set.
+	Collate string
+}
+
+// Restore implements Node interface.
+func (n *SetCollationExpr) Restore(ctx *format.RestoreCtx) error {
+	if err := n.Expr.Restore(ctx); err != nil {
+		return errors.Trace(err)
+	}
+	ctx.WriteKeyWord(" COLLATE ")
+	ctx.WritePlain(n.Collate)
+	return nil
+}
+
+// Format the ExprNode into a Writer.
+func (n *SetCollationExpr) Format(w io.Writer) {
+	n.Expr.Format(w)
+	fmt.Fprintf(w, " COLLATE %s", n.Collate)
+}
+
+// Accept implements Node Accept interface.
+func (n *SetCollationExpr) Accept(v Visitor) (Node, bool) {
+	newNode, skipChildren := v.Enter(n)
+	if skipChildren {
+		return v.Leave(newNode)
+	}
+	n = newNode.(*SetCollationExpr)
+	node, ok := n.Expr.Accept(v)
+	if !ok {
+		return n, false
+	}
+	n.Expr = node.(ExprNode)
+	return v.Leave(n)
+}

ast/expressions_test.go

@@ -17,6 +17,7 @@ import (
 	. "github.com/pingcap/check"
 	. "github.com/pingcap/parser/ast"
 	"github.com/pingcap/parser/format"
+	"github.com/pingcap/parser/mysql"
 )
 
 var _ = Suite(&testExpressionsSuite{})
@@ -88,7 +89,7 @@ func (tc *testExpressionsSuite) TestExpresionsVisitorCover(c *C) {
 			{&PositionExpr{}, 0, 0},
 			{&RowExpr{Values: []ExprNode{ce, ce}}, 2, 2},
 			{&UnaryOperationExpr{V: ce}, 1, 1},
-			{NewValueExpr(0), 0, 0},
+			{NewValueExpr(0, mysql.DefaultCharset, mysql.DefaultCollationName), 0, 0},
 			{&ValuesExpr{Column: &ColumnNameExpr{Name: &ColumnName{}}}, 0, 0},
 			{&VariableExpr{Value: ce}, 1, 1},
 		}

ast/functions.go

@@ -249,6 +249,8 @@ const (
 	TiDBVersion    = "tidb_version"
 	TiDBIsDDLOwner = "tidb_is_ddl_owner"
 	TiDBDecodePlan = "tidb_decode_plan"
+	FormatBytes    = "format_bytes"
+	FormatNanoTime = "format_nano_time"
 
 	// control functions
 	If     = "if"

ast/functions_test.go

@@ -17,6 +17,7 @@ import (
 	. "github.com/pingcap/check"
 	"github.com/pingcap/parser"
 	. "github.com/pingcap/parser/ast"
+	"github.com/pingcap/parser/mysql"
 	"github.com/pingcap/parser/test_driver"
 )
 
@@ -26,7 +27,7 @@ type testFunctionsSuite struct {
 }
 
 func (ts *testFunctionsSuite) TestFunctionsVisitorCover(c *C) {
-	valueExpr := NewValueExpr(42)
+	valueExpr := NewValueExpr(42, mysql.DefaultCharset, mysql.DefaultCollationName)
 	stmts := []Node{
 		&AggregateFuncExpr{Args: []ExprNode{valueExpr}},
 		&FuncCallExpr{Args: []ExprNode{valueExpr}},
@@ -87,8 +88,9 @@ func (ts *testFunctionsSuite) TestFuncCallExprRestore(c *C) {
 		{"weight_string(a)", "WEIGHT_STRING(`a`)"},
 		{"Weight_stRing(test.a)", "WEIGHT_STRING(`test`.`a`)"},
 		{"weight_string('a')", "WEIGHT_STRING('a')"},
-		// TODO(bb7133): collate for literal values cannot be restored.
-		//{"weight_string(_utf8 'a' collate utf8_general_ci)", "WEIGHT_STRING(_UTF8'a' COLLATE utf8_general_ci)"},
+		// Expressions with collations of different charsets will lead to an error in MySQL, but the error check should be done in TiDB, so it's valid here.
+		{"weight_string('a' collate utf8_general_ci collate utf8mb4_general_ci)", "WEIGHT_STRING('a' COLLATE utf8_general_ci COLLATE utf8mb4_general_ci)"},
+		{"weight_string(_utf8 'a' collate utf8_general_ci)", "WEIGHT_STRING(_UTF8'a' COLLATE utf8_general_ci)"},
 		{"weight_string(_utf8 'a')", "WEIGHT_STRING(_UTF8'a')"},
 		{"weight_string(a as char(5))", "WEIGHT_STRING(`a` AS CHAR(5))"},
 		{"weight_string(a as character(5))", "WEIGHT_STRING(`a` AS CHAR(5))"},

ast/misc.go

@@ -459,15 +459,42 @@ func (n *BinlogStmt) Accept(v Visitor) (Node, bool) {
 	return v.Leave(n)
 }
 
+// CompletionType defines completion_type used in COMMIT and ROLLBACK statements
+type CompletionType int8
+
+const (
+	// CompletionTypeDefault refers to NO_CHAIN
+	CompletionTypeDefault CompletionType = iota
+	CompletionTypeChain
+	CompletionTypeRelease
+)
+
+func (n CompletionType) Restore(ctx *format.RestoreCtx) error {
+	switch n {
+	case CompletionTypeDefault:
+		break
+	case CompletionTypeChain:
+		ctx.WriteKeyWord(" AND CHAIN")
+	case CompletionTypeRelease:
+		ctx.WriteKeyWord(" RELEASE")
+	}
+	return nil
+}
+
 // CommitStmt is a statement to commit the current transaction.
 // See https://dev.mysql.com/doc/refman/5.7/en/commit.html
 type CommitStmt struct {
 	stmtNode
+	// CompletionType overwrites system variable `completion_type` within transaction
+	CompletionType CompletionType
 }
 
 // Restore implements Node interface.
 func (n *CommitStmt) Restore(ctx *format.RestoreCtx) error {
 	ctx.WriteKeyWord("COMMIT")
+	if err := n.CompletionType.Restore(ctx); err != nil {
+		return errors.Annotate(err, "An error occurred while restore CommitStmt.CompletionType")
+	}
 	return nil
 }
 
@@ -485,11 +512,16 @@ func (n *CommitStmt) Accept(v Visitor) (Node, bool) {
 // See https://dev.mysql.com/doc/refman/5.7/en/commit.html
 type RollbackStmt struct {
 	stmtNode
+	// CompletionType overwrites system variable `completion_type` within transaction
+	CompletionType CompletionType
 }
 
 // Restore implements Node interface.
 func (n *RollbackStmt) Restore(ctx *format.RestoreCtx) error {
 	ctx.WriteKeyWord("ROLLBACK")
+	if err := n.CompletionType.Restore(ctx); err != nil {
+		return errors.Annotate(err, "An error occurred while restore RollbackStmt.CompletionType")
+	}
 	return nil
 }
 

ast/misc_test.go

@@ -18,6 +18,7 @@ import (
 	"github.com/pingcap/parser"
 	. "github.com/pingcap/parser/ast"
 	"github.com/pingcap/parser/auth"
+	"github.com/pingcap/parser/mysql"
 )
 
 var _ = Suite(&testMiscSuite{})
@@ -44,7 +45,7 @@ func (visitor1) Enter(in Node) (Node, bool) {
 }
 
 func (ts *testMiscSuite) TestMiscVisitorCover(c *C) {
-	valueExpr := NewValueExpr(42)
+	valueExpr := NewValueExpr(42, mysql.DefaultCharset, mysql.DefaultCollationName)
 	stmts := []Node{
 		&AdminStmt{},
 		&AlterUserStmt{},

ast/util.go

@@ -36,7 +36,14 @@ func IsReadOnly(node Node) bool {
 		return checker.readOnly
 	case *ExplainStmt:
 		return !st.Analyze || IsReadOnly(st.Stmt)
-	case *DoStmt:
+	case *DoStmt, *ShowStmt:
+		return true
+	case *UnionStmt:
+		for _, sel := range node.(*UnionStmt).SelectList.Selects {
+			if !IsReadOnly(sel) {
+				return false
+			}
+		}
 		return true
 	default:
 		return false

ast/util_test.go

@@ -71,6 +71,43 @@ func (s *testCacheableSuite) TestCacheable(c *C) {
 	}
 	c.Assert(IsReadOnly(stmt), IsTrue)
 
+	stmt = &ShowStmt{}
+	c.Assert(IsReadOnly(stmt), IsTrue)
+
+	stmt = &ShowStmt{}
+	c.Assert(IsReadOnly(stmt), IsTrue)
+}
+
+func (s *testCacheableSuite) TestUnionReadOnly(c *C) {
+	selectReadOnly := &SelectStmt{}
+	selectForUpdate := &SelectStmt{
+		LockTp: SelectLockForUpdate,
+	}
+	selectForUpdateNoWait := &SelectStmt{
+		LockTp: SelectLockForUpdateNoWait,
+	}
+
+	unionStmt := &UnionStmt{
+		SelectList: &UnionSelectList{
+			Selects: []*SelectStmt{selectReadOnly, selectReadOnly},
+		},
+	}
+	c.Assert(IsReadOnly(unionStmt), IsTrue)
+
+	unionStmt.SelectList.Selects = []*SelectStmt{selectReadOnly, selectReadOnly, selectReadOnly}
+	c.Assert(IsReadOnly(unionStmt), IsTrue)
+
+	unionStmt.SelectList.Selects = []*SelectStmt{selectReadOnly, selectForUpdate}
+	c.Assert(IsReadOnly(unionStmt), IsFalse)
+
+	unionStmt.SelectList.Selects = []*SelectStmt{selectReadOnly, selectForUpdateNoWait}
+	c.Assert(IsReadOnly(unionStmt), IsFalse)
+
+	unionStmt.SelectList.Selects = []*SelectStmt{selectForUpdate, selectForUpdateNoWait}
+	c.Assert(IsReadOnly(unionStmt), IsFalse)
+
+	unionStmt.SelectList.Selects = []*SelectStmt{selectReadOnly, selectForUpdate, selectForUpdateNoWait}
+	c.Assert(IsReadOnly(unionStmt), IsFalse)
 }
 
 // CleanNodeText set the text of node and all child node empty.