lightning: should report access deny error instead of table not found

[july2993]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

• grant the role user to run lighting s3:ListBucket but not GetObject to the objects
• run lightning fails and reporting "table info test.test not found"

[2021/03/30 09:59:14.868 +00:00] [ERROR] [lightning.go:209] ["tidb lightning encountered error"] [error="table info test.test not found"] [errorVerbose="table info test.test not found\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*RestoreController).restoreTables\n\t/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:745\ngithub.com/pingcap/tidb-lightning/lightning/restore.(*RestoreController).Run\n\t/go/src/github.com/pingcap/tidb-lightning/lightning/restore/restore.go:279\ngithub.com/pingcap/tidb-lightning/lightning.(*Lightning).run\n\t/go/src/github.com/pingcap/tidb-lightning/lightning/lightning.go:310\ngithub.com/pingcap/tidb-lightning/lightning.(*Lightning).RunServer\n\t/go/src/github.com/pingcap/tidb-lightning/lightning/lightning.go:206\nmain.main.func2\n\t/go/src/github.com/pingcap/tidb-lightning/cmd/tidb-lightning/main.go:80\nmain.main\n\t/go/src/github.com/pingcap/tidb-lightning/cmd/tidb-lightning/main.go:88\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
--

there's log :

[2021/03/30 09:59:14.227 +00:00] [ERROR] [loader.go:59] ["failed to extract table schema"] [Path=test.test-schema.sql] [error="AccessDenied: Access Denied\n\tstatus code: 403, request id: H71TPE8299JCPWXG, host id: edoDKSHpHMTUkldRW05vzfbBJ9DP9w8YVlLaeUn2B3eXsOJRAtsRJzz0r48VjXw6TNetcb4/LWY="]

the error is just log and ignore there:

br/pkg/lightning/mydump/loader.go

Line 57 in 86407e1

schema, err := ExportStatement(ctx, store, m.SchemaFile, m.charSet)

files like this:

8 -rw-r--r--@ 1 huangjiahao  staff   135B Mar 30 22:35 test.test-schema.sql
8 -rw-r--r--@ 1 huangjiahao  staff    76B Mar 30 22:39 metadata
8 -rw-r--r--@ 1 huangjiahao  staff    95B Mar 30 22:39 test-schema-create.sql
8 -rw-r--r--@ 1 huangjiahao  staff    60B Mar 30 22:39 test.test.000000000.sql

prefix.tar.gz

2. What did you expect to see?
   report error about access deny

3. What did you see instead?
   report error "table info test.test not found"

4. What version of BR and TiDB/TiKV/PD are you using?

5. Operation logs

   • Please upload br.log for BR if possible
   • Please upload tidb-lightning.log for TiDB-Lightning if possible
   • Please upload tikv-importer.log from TiKV-Importer if possible
   • Other interesting logs

6. Configuration of the cluster and the task

   • tidb-lightning.toml for TiDB-Lightning if possible
   • tikv-importer.toml for TiKV-Importer if possible
   • topology.yml if deployed by TiUP

7. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus if possible

[kennytm]

The return type of GetSchema function is unchanged for 3 years and thus is probably out of date when compared with the current product.

For contributors, you can change the function to propagate the error out. Either the caller (there are 2 places) or the callee should distinguish the "file not found" error (which is normal) and other kinds of error (which is unexpected).

[Abingcbc]

I would like to work on this issue :)
When I was reproducing the bug, I found that restoring databases only reads the name of the SQL file ({db}-schema-create.sql), but not cares about the content. Lightning can create a database even when the schema-create file is empty. Is it a feature or a bug?

Furthermore, does lightning support custom database DDL like follows?

CREATE DATABASE test CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;

Thanks

[kennytm]

@Abingcbc when Lightning was first published, TiDB did not support character sets besides subset of UTF-8, so the content of the file is ignored, and then this part is never updated 😅. Feel free to fix it.