Dumpling v5.1.0 can NOT access Google GCS; Works in v5.0.0

[BarDev]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

I ran dumpling v5.1.0 with the output to GCS bucket (see command below). Notice the error Google Cloud Storage can NOT be accessed.

tiup dumpling:v5.1.0 --host 192.168.0.5 -P 3306 -u root --password=<password> --database bikeshare --output 'gs://test_backup_data/test'

Release version: v5.1.0
Git commit hash: 292cbe6c9da0f53a262332d7711dd6ba96567411
Git branch:      heads/refs/tags/v5.1.0
Build timestamp: 2021-06-24 07:09:21Z
...
[2021/06/30 23:00:24.917 +00:00] [WARN] [gcs.go:248] ["failed to list objects on gcs, will use default value for `prefix`"] [error="googleapi: Error 401: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket., required"]
...

2. What did you expect to see?

I expected to see the same outcome as Dumpling v5.0.0 - dump data successfully, dumpling will exit now". The command below is identical to the one above, but instead of referencing dumpling:v5.1.0, it's referencing dumpling:v5.0.0

tiup dumpling:v5.0.0 --host 192.168.0.5 -P 3306 -u root --password=<password> --database bikeshare --output 'gs://test_backup_data/test'

Release version: v5.0.0
Git commit hash: 463e4cc96078fa532baeea05624d9b5ceb044fb1
Git branch:      heads/refs/tags/v5.0.0
Build timestamp: 2021-04-06 04:42:26Z
Go version:      go version go1.13 linux/amd64
...
[2021/06/30 22:59:40.100 +00:00] [INFO] [collector.go:212] ["backup Success summary: total backup ranges: 3, total success: 3, total failed: 0, total take(backup time): 10.414278672s, total take(real time): 10.414327484s, total size(MB): 509.75, avg speed(MB/s): 48.95, total rows: 3757777"]
[2021/06/30 22:59:40.143 +00:00] [INFO] [main.go:81] ["dump data successfully, dumpling will exit now"]

3. What did you see instead?

Release version: v5.1.0
Git commit hash: 292cbe6c9da0f53a262332d7711dd6ba96567411
Git branch:      heads/refs/tags/v5.1.0
Build timestamp: 2021-06-24 07:09:21Z
...
[2021/06/30 23:00:24.917 +00:00] [WARN] [gcs.go:248] ["failed to list objects on gcs, will use default value for `prefix`"] [error="googleapi: Error 401: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket., required"]
...

4. What version of BR and TiDB/TiKV/PD are you using?

Dumpling v5.1.0

Release version: v5.1.0
Git commit hash: 292cbe6c9da0f53a262332d7711dd6ba96567411
Git branch:      heads/refs/tags/v5.1.0
Build timestamp: 2021-06-24 07:09:21Z

5. Operation logs

Starting component `dumpling`: /home/mikebarlow/.tiup/components/dumpling/v5.1.0/dumpling --host 192.168.0.5 -P 3306 -u root --password=********* --database bikeshare --output gs://test_backup_data/test
Release version: v5.1.0
Git commit hash: 292cbe6c9da0f53a262332d7711dd6ba96567411
Git branch:      heads/refs/tags/v5.1.0
Build timestamp: 2021-06-24 07:09:21Z
Go version:      go version go1.16.4 linux/amd64

[2021/06/30 23:00:24.813 +00:00] [INFO] [versions.go:55] ["Welcome to dumpling"] ["Release Version"=v5.1.0] ["Git Commit Hash"=292cbe6c9da0f53a262332d7711dd6ba96567411] ["Git Branch"=heads/refs/tags/v5.1.0] ["Build timestamp"="2021-06-24 07:09:21"] ["Go Version"="go version go1.16.4 linux/amd64"]
[2021/06/30 23:00:24.917 +00:00] [WARN] [gcs.go:248] ["failed to list objects on gcs, will use default value for `prefix`"] [error="googleapi: Error 401: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket., required"]
[2021/06/30 23:00:25.026 +00:00] [WARN] [gcs.go:248] ["failed to list objects on gcs, will use default value for `prefix`"] [error="googleapi: Error 401: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket., required"]
[2021/06/30 23:00:25.030 +00:00] [INFO] [config.go:602] ["detect server type"] [type=MySQL]
[2021/06/30 23:00:25.030 +00:00] [INFO] [config.go:621] ["detect server version"] [version=5.7.33-google-log]
[2021/06/30 23:00:25.030 +00:00] [INFO] [dump.go:83] ["begin to run Dump"] [conf="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"provider\":\"\",\"force-path-style\":true,\"use-accelerate-endpoint\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":false,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"192.168.0.5\",\"Port\":3306,\"Threads\":4,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"info\",\"LogFile\":\"\",\"LogFormat\":\"text\",\"OutputDirPath\":\"gs://test_backup_data/test\",\"StatusAddr\":\":8281\",\"Snapshot\":\"\",\"Consistency\":\"flush\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\",\",\"CsvDelimiter\":\"\\\"\",\"Databases\":[\"bikeshare\"],\"Where\":\"\",\"FileType\":\"sql\",\"ServerInfo\":{\"HasTiKV\":false,\"ServerType\":1,\"ServerVersion\":\"5.7.33-google-log\"},\"Rows\":0,\"ReadTimeout\":900000000000,\"TiDBMemQuotaQuery\":0,\"FileSize\":0,\"StatementSize\":1000000,\"SessionParams\":{},\"Tables\":null}"]
[2021/06/30 23:00:25.056 +00:00] [INFO] [dump.go:178] ["All the dumping transactions have started. Start to unlock tables"]
[2021/06/30 23:00:25.114 +00:00] [ERROR] [writer_util.go:423] ["close file failed"] [path=gcs:/test_backup_data/test/bikeshare.trips-schema.sql] [error="googleapi: Error 401: Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object., required"] [errorVerbose="googleapi: Error 401: Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object., required\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.buildFileWriter.func1\n\tgithub.com/pingcap/dumpling/v4/export/writer_util.go:422\ngithub.com/pingcap/dumpling/v4/export.writeMetaToFile\n\tgithub.com/pingcap/dumpling/v4/export/writer.go:245\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableMeta\n\tgithub.com/pingcap/dumpling/v4/export/writer.go:134\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling/v4/export/writer.go:99\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling/v4/export/writer.go:85\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling/v4/export/dump.go:273\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2021/06/30 23:00:25.114 +00:00] [ERROR] [writer_util.go:423] ["close file failed"] [path=gcs:/test_backup_data/test/bikeshare-schema-create.sql] [error="googleapi: Error 401: Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object., required"] [errorVerbose="googleapi: Error 401: Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object., required\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.buildFileWriter.func1\n\tgithub.com/pingcap/dumpling/v4/export/writer_util.go:422\ngithub.com/pingcap/dumpling/v4/export.writeMetaToFile\n\tgithub.com/pingcap/dumpling/v4/export/writer.go:245\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteDatabaseMeta\n\tgithub.com/pingcap/dumpling/v4/export/writer.go:124\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgithub.com/pingcap/dumpling/v4/export/writer.go:97\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\tgithub.com/pingcap/dumpling/v4/export/writer.go:85\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgithub.com/pingcap/dumpling/v4/export/dump.go:273\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1371"]
[2021/06/30 23:00:25.415 +00:00] [ERROR] [writer_util.go:398] ["writing failed"] [string="(111651,727,'2017-10-09 17:03:50','2017-10-09 17:15:57',31267,'17th St & Massachusetts Ave NW',31638,'1st & H St NW','W22469','Casual'),\n(111652,1489,'2017-10-09 17:03:53','2017-10-09 17:28:43',31406,"] [writer="&export.InterceptFileWriter{ExternalFileWriter:(*storage.flushStorageWriter)(0xc000a424b0), Once:sync.Once{done:0x1, m:sync.Mutex{state:0, sema:0x0}}, SomethingIsWritten:true, initRoutine:(func() error)(0x1b86840), err:error(nil)}"] [error="googleapi: Error 401: Anonymous caller does not have storage.objects.create access to test_backup_data/test/bikeshare.trips.000000000.sql., required"]
[2021/06/30 23:00:25.433 +00:00] [ERROR] [writer_util.go:459] ["close file failed"] [path=gcs:/test_backup_data/test/bikeshare.trips.000000000.sql] [error="googleapi: Error 401: Anonymous caller does not have storage.objects.create access to test_backup_data/test/bikeshare.trips.000000000.sql., required"]
^CGot signal interrupt (Component: dumpling ; PID: 28911)

6. Configuration of the cluster and the task
   N/A

7. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus if possible
   N/A

[dveeden]

Looking at the commits between the two hashes from the report:

$ git log --oneline 463e4cc96078fa532baeea05624d9b5ceb044fb1..292cbe6c9da0f53a262332d7711dd6ba96567411
292cbe6 *: replace storage.Create by storage.New, upgrade dumpling CI to go 1.16 (#286)
be441d7 config: Correct usage info for snapshot flag (#283)
d6e5fb4 dump: always split TiDB v4.* tables through tidb rowid to save TiDB's memory (#273)
0a1f0d3 log: replace pingcap global logger when start dumpling binary (#270)
e1cfe4c export: fix estimateTotalRowsCounter nil pointer (#267)
2781c1e export: fix a metric not using labelNames (#266)
51b465b sql: fix table sample sql generator bug (#262)
59b763e Add conn session vars (#265)
48171c9 dump: add split big tidb query to several small queries (#257)
bff60f8 dump: support new where sql generator (#260)
a88ebcb dump: fix the problem that dumpling gets stuck after the database connection is lost (#256)
fe53295 *: add logs and try fixing error log (#246)
20e81ab add estimate total rows count to metrics for dump table (#254)
d35245d export: ignore unused parameters (#250)
843aff7 *: refine user experience (#245)

I think pingcap/dumpling@292cbe6 from pingcap/dumpling#286 might be the commit that causes this.

This looks like a dumpling issue to me instead of br/lightning. So I think it should be moved/copied to there and the title needs adjustment.

[lichunzhu]

@BarDev Are you still here? I have tried to fix this on this branch. Could you please help us build a binary from this branch and try dumping data to GCS?
https://github.com/lichunzhu/dumpling/tree/v5.1.0-hotfix-gcs

[sleepymole]

This issue is closed due to not being active for a long time. If you have further questions, please create a new one and reference this in pingcap/tidb.

[lichunzhu]

already fixed on v5.3.0.
feel free to take a try. pingcap/tidb#29414