Rewriter infrastructure revamp #5611

systay · 2019-12-22T09:50:48Z

Before this change, we have two different AST visitors - one for rewriting expressions, and one for visiting SQLNodes. Both these needed to be manually updated whenever the AST changed.

This change introduces a new visitor pattern, that can do what the older visitors could, but that is generated from the AST instead of being hand coded.

To enable this, I had to restructure the AST a little. Now, the AST structs and interface implementations live in the ast.go file, which is what the visitor generator reads to produce the visitor. The rest of the AST functionality, all methods belonging to structs and not implementing any interfaces, now live in ast_funcs.go.

I'm now using this new visitor to simplify the AST rewriting we do today: auto-parameterization (replacing values with bindvars), last_insert_id() and database() should all happen close to each other and not be done while planning.

deepthi

This is a major refactor/rewrite. Can you write a description for the PR that lays out how things are changing and why?

deepthi · 2020-01-14T23:43:40Z

Makefile

@@ -72,6 +72,11 @@ install: build
 parser:
 	make -C go/vt/sqlparser

+visitor:


shouldn't we be calling this whenever there is a change to sql.y (and sql.go)?

not really. only when there is a change to ast.go

will people need to know to call this manually? is there a test that will catch it if someone forgets to do so after changing ast.go?

added pre-commit hook

deepthi · 2020-01-14T23:44:44Z

go/visitorgen/sast.go

@@ -0,0 +1,161 @@
+package visitorgen


missing license header

deepthi · 2020-01-14T23:45:07Z

go/visitorgen/sast.go

@@ -0,0 +1,161 @@
+package visitorgen
+
+// simplified ast


There should be some explanatory docs here.

deepthi · 2020-01-14T23:45:36Z

go/visitorgen/transformer.go

@@ -0,0 +1,79 @@
+package visitorgen


missing license header.

sougou

I believe this codegen is not as type-safe as I had expected. It can still panic if you pass the wrong type to cursor.Replace. In this particular case, the codegen should have generated two visitor functions: one for *AliasedExpression, and one for Expr. The cursor.Replace for *AliasedExpression would only accept an AliasedExpression instead of SQLNode.

And the pattern I described for handling the substitutions in the rewriter would allow you to compose the two.

However, this is good enough for now. Maybe we can improve this later. Let's move forward with this. I can merge once the minor nits are taken care of.

PS: Note that I am nitpicking. This is otherwise an awesome piece of work!

sougou · 2020-01-19T04:51:31Z

Makefile

@@ -87,6 +88,11 @@ install: build
 parser:
 	make -C go/vt/sqlparser

+visitor:
+	go build -o visitorgen go/visitorgen/main/main.go
+	./visitorgen -input=go/vt/sqlparser/ast.go -output=$(REWRITER)


It's better to just go run go/visitorgen/main/main.go -input go/vt/sqlparser/ast.go -output=go/vt/sqlparser/rewriter.go. Then you don't have to worry about cleanup of the binary. I just tried, and it works as intended. For some reason, I had to remove the = after input.

I can't get this to run on my Mac. All I get is:

> go run -o visitorgen go/visitorgen/main/main.go -input go/vt/sqlparser/ast.go -output=go/vt/sqlparser/rewriter.go flag provided but not defined: -o usage: go run [build flags] [-exec xprog] package [arguments...] Run 'go help run' for details. > go run -o visitorgen go/visitorgen/main/main.go -input=go/vt/sqlparser/ast.go -output=go/vt/sqlparser/rewriter.go flag provided but not defined: -o usage: go run [build flags] [-exec xprog] package [arguments...] Run 'go help run' for details. > go run -o visitorgen go/visitorgen/main/main.go -input=go/vt/sqlparser/ast.go -output go/vt/sqlparser/rewriter.go flag provided but not defined: -o usage: go run [build flags] [-exec xprog] package [arguments...] Run 'go help run' for details. > go run -o visitorgen go/visitorgen/main/main.go -input go/vt/sqlparser/ast.go -output go/vt/sqlparser/rewriter.go flag provided but not defined: -o usage: go run [build flags] [-exec xprog] package [arguments...] Run 'go help run' for details.

I agree with Sugu that go run would be much nicer here. In the error output you saw, it looks like you had go run -o vistorgen and an error about the -o. If you get rid of those arguments it seems like go run should work

sougou · 2020-01-19T05:05:17Z

go/vt/sqlparser/ast_funcs.go

+		return sqltypes.Int24
+	case keywordStrings[INT]:
+		fallthrough
+	case keywordStrings[INTEGER]:


better to use case keywordStrings[INT], keywordStrings[INTEGER]:

sougou · 2020-01-19T05:12:50Z

go/vt/sqlparser/ast_funcs.go

+func ReplaceExpr(root, from, to Expr) Expr {
+	expr, success := Rewrite(root, replaceExpr(from, to), nil).(Expr)
+	if !success {
+		panic("expression rewriting ended up with a non-expression")


Better not to panic. This impossible in the new framework, right? Then it's better to not checkat all.

sougou · 2020-01-19T05:19:55Z

go/vt/sqlparser/ast_funcs.go

+	if right, ok = node.Right.(*SQLVal); !ok {
+		return false
+	}
+	if node.Operator == NotEqualStr && left.Type == right.Type {


use bytes.Equal

Not sure what you mean, and this is not something I changed: https://github.com/vitessio/vitess/blob/master/go/vt/sqlparser/ast.go#L2431

sougou · 2020-01-19T06:10:36Z

go/vt/sqlparser/expression_rewriting.go

+type expressionRewriter struct {
+	lastInsertID, database bool
+	err                    error
+	aliases                []*AliasedExpr


I now see why you were trying to add aliases for every expression :).

If you were already willing to pay the price of generating an alias for every select expression, there is a simpler way: As soon as you encounter an unaliased *AliasedExpression, generate an alias for it locally. Then create a new RewriteASTResult and invole a Rewrite on it. If it returns saying that something has changed, then you substitute the new alias after it returns. And always return false.

This avoids the need to maintain a separate aliases list, and also there is no need for a comingUp function.

Nice idea. Much cleaner.

sougou · 2020-01-19T06:12:51Z

go/vt/sqlparser/rewriter.go

+	cursor    Cursor
+}
+
+func replaceAliasedExprAs(new SQLNode, parent SQLNode) {


Could you use something other than new? It could confuse people that use it as keyword.

Signed-off-by: Andres Taylor <[email protected]>

dweitzman · 2020-01-21T16:44:18Z

I'm wondering if parameterizing the parser could avoid a whole bunch of code and complexity in supporting database() and last_insert_id().

Suppose we could change this part of sql.y:

/*
  Regular function calls without special token or syntax, guaranteed to not
  introduce side effects due to being a simple identifier
*/
function_call_generic:
  sql_id openb select_expression_list_opt closeb
  {
    $$ = &FuncExpr{Name: $1, Exprs: $3}
  }

To something more in the spirit of this:


sql.go:
// vtgate can set up a custom SQL parser that replaces certain
// functions during parse time. Avoids the need to adjust the AST
// later.
func yyNewParser(f FuncCreator) yyParser {
	return &yyParserImpl{funcCreator: f}
}

func (yyrcvr *yyParserImpl) CreateSimpleFunc(name ColIdent, exprs SelectExprs) *FuncExpr {
  if (len(exprs) == 0 && yyrcvr.funcCreator != nil) {
    res := yyrcvr.funcCreator.Create(name)
    if res != nil {
       return res
    }
  }
  return &FuncExpr{Name: name, Exprs: exprs}
}

sql.y:
/*
  Regular function calls without special token or syntax, guaranteed to not
  introduce side effects due to being a simple identifier
*/
function_call_generic:
  sql_id openb select_expression_list_opt closeb
  {
    $$ = yyrcvr.CreateSimpleFunc($1, $3)
  }

If a parameterized parser could be set up, is there any functionality in the rewriter today that a parameterized parser wouldn't be able to handle?

dweitzman

I have the larger question of whether the rewriter can be avoided entirely, but since the code is committed now I'll send some of the pending comments I had for the small fraction of the changes I've read so far

dweitzman · 2020-01-18T02:44:44Z

go/visitorgen/main/main.go

+
+const usage = `Usage of visitorgen:
+
+go run go/visitorgen/main/main.go -input=/path/to/ast.go -output=/path/to/rewriter.go


"go run ./go/visitorgen/main" is a little shorter and avoids a "missing dot in first path element" error

dweitzman · 2020-01-18T02:46:50Z

go/visitorgen/ast_walker.go

@@ -0,0 +1,130 @@
+/*


What do you think about putting this package at go/ast/visitorgen so it's more clear that it's related to ast?

Could do. I was thinking I wanted to use visitorgen for our Primitive plans as well. We don't rewrite them today, but we do visit them using the Find method, and I foresee rewriting of plans in our future as well.

Never mind - it will be easy to move it somewhere else when we do decide to start using it for more than the ast. For now, I'll move it into the sqlparser package

dweitzman · 2020-01-18T02:54:20Z

go/visitorgen/main/main.go

+	if *inputFile == "" || *outputFile == "" {
+		fmt.Println("> " + *inputFile)
+		fmt.Println("> " + *outputFile)
+		panic("need input and output file")


panicing with a printed stack trace feels a bit aggressive. Maybe just log error an exit? Same comment elsewhere

exit.Return from vitess.io/vitess/go/exit seems to be popular for most of the vitess commands, although it's not super clear to me what value it provides over directly calling os.Exit()

dweitzman · 2020-01-18T03:08:25Z

go/visitorgen/main/main.go

+	replacementMethods := visitorgen.EmitReplacementMethods(vd)
+	typeSwitch := visitorgen.EmitTypeSwitches(vd)
+
+	fw := newFileWriter(*outputFile)


Seems like the fileWriter struct could be avoided here. One way:

var b bytes.Buffer fmt.Fprintln(b, fileHeader) fmt.Fprintln(b, ...) err := ioutil.WriteFile(*outputFile, b.Bytes(), 0644)

dweitzman · 2020-01-18T03:11:22Z

Makefile

+visitor:
+	go build -o visitorgen go/visitorgen/main/main.go
+	./visitorgen -input=go/vt/sqlparser/ast.go -output=$(REWRITER)
+	rm ./visitorgen


How do you feel about having the Makefile invoke go generate and the //go:generate annotation could have the go run command?

Being able to use pure go build tools seems nice. Anyone who wants to use make to invoke go would have the option, although I don't think it would do much for them.

dweitzman · 2020-01-21T15:50:23Z

go/vt/sqlparser/expression_rewriting.go

+	switch node := cursor.Node().(type) {
+	case *AliasedExpr:
+		if node.As.IsEmpty() {
+			buf := NewTrackedBuffer(nil)


Seems like this does a serialize / memory allocations / copying even when nothing needs to change

dweitzman · 2020-01-21T21:46:28Z

Makefile

@@ -87,6 +88,11 @@ install: build
 parser:
 	make -C go/vt/sqlparser

+visitor:
+	go build -o visitorgen go/visitorgen/main/main.go
+	./visitorgen -input=go/vt/sqlparser/ast.go -output=$(REWRITER)


I agree with Sugu that go run would be much nicer here. In the error output you saw, it looks like you had go run -o vistorgen and an error about the -o. If you get rid of those arguments it seems like go run should work

systay · 2020-01-27T10:28:27Z

Thanks for the feedback, @dweitzman. Addressed most of it with #5770

systay · 2020-01-27T19:17:12Z

I'm wondering if parameterizing the parser could avoid a whole bunch of code and complexity in supporting database() and last_insert_id().

...

If a parameterized parser could be set up, is there any functionality in the rewriter today that a parameterized parser wouldn't be able to handle?

You are probably right, at least for this case. I'm thinking ahead, being guided by experience and gut feeling and not much hard data. Maybe I'm overestimating my pretty singular experience and not seeing aspects that are new in this situation.

I'll try to be clear about the assumptions that lead to thinking this would be a good idea:

I think we'll soon do more even more rewriting of the AST
We'll want to use the same generator to produce rewriting support for the vtgate engine Primitive trees
I value separation of concerns a lot around these things. It makes it easier to test, to understand and to reason about.
If performance is the issue we're trying to address with the parameterized parser, I think we can use the visitorgen tool to produce pretty fast code.

systay added the Status: Work in progress label Dec 22, 2019

systay requested a review from sougou as a code owner December 22, 2019 09:50

This was referenced Jan 6, 2020

Refactoring of ast.go #5578

Closed

Refactoring of the visitor and rewriter for the AST #5525

Closed

systay force-pushed the rewriter-mk4 branch from 4ec5476 to 230b50c Compare January 9, 2020 19:42

systay removed the Status: Work in progress label Jan 10, 2020

systay force-pushed the rewriter-mk4 branch from 9ca65b4 to 3d50554 Compare January 11, 2020 00:57

deepthi self-requested a review January 13, 2020 01:07

deepthi reviewed Jan 14, 2020

View reviewed changes

systay force-pushed the rewriter-mk4 branch 3 times, most recently from e66b930 to ceaccb7 Compare January 16, 2020 00:14

systay mentioned this pull request Jan 16, 2020

Certain invalid DISTINCT tuple select statements cause vtgate panics #5709

Closed

systay force-pushed the rewriter-mk4 branch 3 times, most recently from a6f81f9 to 4c28604 Compare January 17, 2020 21:42

systay mentioned this pull request Jan 17, 2020

Rewrite more functions #5732

Merged

systay force-pushed the rewriter-mk4 branch from 4c28604 to 6461c79 Compare January 17, 2020 21:52

sougou reviewed Jan 19, 2020

View reviewed changes

systay added 11 commits January 21, 2020 09:09

Restructure the AST

955f4f7

Signed-off-by: Andres Taylor <[email protected]>

Add the visitor generator

65a57ef

Signed-off-by: Andres Taylor <[email protected]>

Add the the rewriter

216d49d

Signed-off-by: Andres Taylor <[email protected]>

Remove old visitor and replace functionality

6933419

Signed-off-by: Andres Taylor <[email protected]>

License headers and more comments

070347d

Signed-off-by: Andres Taylor <[email protected]>

Add commit hook to check the visitor is in sync with the ast

16300d6

Signed-off-by: Andres Taylor <[email protected]>

Make it possible to rewrite the whole AST in one go

a0f6a81

Signed-off-by: Andres Taylor <[email protected]>

Add column aliases when needed

8765802

Signed-off-by: Andres Taylor <[email protected]>

Moved AST rewriting to be done together with Normalization

b92fd33

Signed-off-by: Andres Taylor <[email protected]>

Make sure to rewrite even when using direct target queries

3fef779

Signed-off-by: Andres Taylor <[email protected]>

Update tests

38bf74d

Signed-off-by: Andres Taylor <[email protected]>

systay added 2 commits January 21, 2020 09:09

Add go generate support

bd99abb

Signed-off-by: Andres Taylor <[email protected]>

Removed unused code

b6d18b7

Signed-off-by: Andres Taylor <[email protected]>

systay force-pushed the rewriter-mk4 branch from 9ecef87 to 8ac9130 Compare January 21, 2020 08:09

systay added 3 commits January 21, 2020 09:33

Simplified expression rewriting

8274354

Signed-off-by: Andres Taylor <[email protected]>

Clearer argument names

5c242d3

Signed-off-by: Andres Taylor <[email protected]>

Make sure to not panic while rewriting

a8f2ea1

Signed-off-by: Andres Taylor <[email protected]>

systay force-pushed the rewriter-mk4 branch from 8ac9130 to a8f2ea1 Compare January 21, 2020 08:33

sougou approved these changes Jan 21, 2020

View reviewed changes

sougou merged commit 2cd6a1b into vitessio:master Jan 21, 2020

dweitzman reviewed Jan 21, 2020

View reviewed changes

ajm188 mentioned this pull request Mar 10, 2020

Slack sync upstream 2020 03 10.r0 tinyspeck/vitess#147

Closed

zhyass mentioned this pull request May 27, 2020

[refactoring] refactoring code radondb/radon#480

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewriter infrastructure revamp #5611

Rewriter infrastructure revamp #5611

systay commented Dec 22, 2019 •

edited

Loading

deepthi left a comment

deepthi Jan 14, 2020

systay Jan 15, 2020

deepthi Jan 15, 2020

systay Jan 15, 2020

deepthi Jan 14, 2020

deepthi Jan 14, 2020

deepthi Jan 14, 2020

sougou left a comment

sougou Jan 19, 2020

systay Jan 21, 2020

dweitzman Jan 21, 2020 •

edited

Loading

sougou Jan 19, 2020

sougou Jan 19, 2020

sougou Jan 19, 2020

systay Jan 21, 2020

sougou Jan 19, 2020

systay Jan 21, 2020

sougou Jan 19, 2020

dweitzman commented Jan 21, 2020 •

edited

Loading

dweitzman left a comment

dweitzman Jan 18, 2020

dweitzman Jan 18, 2020

systay Jan 27, 2020

dweitzman Jan 18, 2020

dweitzman Jan 18, 2020

dweitzman Jan 18, 2020

dweitzman Jan 21, 2020

dweitzman Jan 21, 2020 •

edited

Loading

systay commented Jan 27, 2020

systay commented Jan 27, 2020


		const usage = `Usage of visitorgen:

		go run go/visitorgen/main/main.go -input=/path/to/ast.go -output=/path/to/rewriter.go

Rewriter infrastructure revamp #5611

Rewriter infrastructure revamp #5611

Conversation

systay commented Dec 22, 2019 • edited Loading

deepthi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sougou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dweitzman Jan 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dweitzman commented Jan 21, 2020 • edited Loading

dweitzman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dweitzman Jan 21, 2020 • edited Loading

Choose a reason for hiding this comment

systay commented Jan 27, 2020

systay commented Jan 27, 2020

systay commented Dec 22, 2019 •

edited

Loading

dweitzman Jan 21, 2020 •

edited

Loading

dweitzman commented Jan 21, 2020 •

edited

Loading

dweitzman Jan 21, 2020 •

edited

Loading