:copyfrom for MySQL LOAD DATA INFILE #2220
Conversation
Jille
force-pushed
the
loaddata2
branch
5 times, most recently
from
b906c49
to
759860f
Compare
|
@Jille I'm assuming this PR is still in draft because of some upstream mysql driver issues? |
|
Yeah, I need go-sql-driver/mysql to expose the *time.Location to encode correctly. But the maintainers don't seem to want to expose it in a way sqlc can use it. How would you feel about an extra argument to the :loaddata methods that receives the *time.Location with the timezone configured for the connection? It's ugly, but I'm not sure I see another way yet. |
I don't think that will play nicely with the |
This enables the :copyfrom query annotation for people using go-sql-driver/mysql that transforms it into a LOAD DATA LOCAL INFILE. We don't have a way to get the timezone from the connection, so I've simply blocked people from using time.Times in their copyfrom. issue sqlc-dev#2179
|
We haven't found a way to expose the timezone from go-sql-driver/mysql yet. Let's move ahead with this and simply block |
| @@ -186,6 +186,8 @@ import ( | |||
| {{define "copyfromCode"}} | |||
| {{if .SQLDriver.IsPGX }} | |||
| {{- template "copyfromCodePgx" .}} | |||
| {{else}} | |||
There was a problem hiding this comment.
Instead of {{else}}, let's check for the MySQL driver explicitly.
| // {{.MethodName}} uses MySQL's LOAD DATA LOCAL INFILE and is not atomic. Errors and duplicate keys are treated as warnings and insertion will continue, even without an error for some cases. | ||
| // Use this in a transaction and use SHOW WARNINGS to check for any problems and roll back if you want to. | ||
| // This is a MySQL limitation, not sqlc. Check the documentation for more information: https://dev.mysql.com/doc/refman/8.0/en/load-data.html#load-data-error-handling |
There was a problem hiding this comment.
This comment is too wide. Can we edit it to be this instead?
// {{.MethodName}} uses MySQL's LOAD DATA LOCAL INFILE and is not atomic.
//
// Errors and duplicate keys are treated as warnings and insertion will
// continue, even without an error for some cases. Use this in a transaction
// and use SHOW WARNINGS to check for any problems and roll back if you want to.
//
// Check the documentation for more information:
// https://dev.mysql.com/doc/refman/8.0/en/load-data.html#load-data-error-handling
| mysql.RegisterReaderHandler(rh, func() io.Reader { return pr }) | ||
| defer mysql.DeregisterReaderHandler(rh) | ||
| go convertRowsFor{{.MethodName}}(pw, {{.Arg.Name}}) | ||
| result, err := {{if (not $.EmitMethodsWithDBArgument)}}q.{{end}}db.ExecContext(ctx, fmt.Sprintf("LOAD DATA LOCAL INFILE '%s' INTO TABLE {{.TableIdentifierForMySQL}} %s ({{range $index, $name := .Arg.ColumnNames}}{{if gt $index 0}}, {{end}}{{$name}}{{end}})", "Reader::" + rh, mysqltsv.Escaping)) |
There was a problem hiding this comment.
Let's add a comment here explaining that LOAD DATA can't be used with parameters and link to this part of the docs
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed. The file name must be given as a literal string.
|
Going to get this ready to merge in #2545. Thanks for the initial implementation :) |
|
Closed via #2545 |
Similar to PostgreSQL :copyfrom support, this adds support for MySQL's LOAD DATA INFILE.
Usage is the same as PostgreSQL: by writing an INSERT INTO query with :copyfrom, and we'll generate a method that internally calls LOAD DATA INFILE.
The only MySQL driver I found that supports LOAD DATA LOCAL INFILE at all is github.com/go-sql-driver/mysql. In able to support other drivers that might add support in the future, the
sql_packageconfig setting must be set togo-sql-driver/mysql.