Skip to content
This repository has been archived by the owner on Aug 21, 2023. It is now read-only.

*: refine dumpling log hint system #228

Merged
merged 8 commits into from
Dec 30, 2020
Merged

Conversation

lichunzhu
Copy link
Contributor

@lichunzhu lichunzhu commented Dec 30, 2020

What problem does this PR solve?

  1. close dumpling should fail when provided positional parameters to CLI #225. Dumpling won't check unparsed arguments now. It's not easy for users to find spell errors.
  2. The speed of dumpling data from tidb to s3 drops 2+GBit/s to 200Mbit/s #223. Some logs in dumpling are not clear enough. For example, we print the estimated count without table name. It's hard to debug.
  3. Users can't get current dumpling work status and progress from logs now.

What is changed and how it works?

  1. Check unparsed arguments and return an error to users.
  2. Refine dumpling's log info and adjust some log's level.
  3. Add status printer in dumper.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  1. example for unparsed arguments:
$ ./bin/dumpling -P 4000 -T test.order_line -r 20 -F 10B where "id<5"
Release version: v4.0.8-31-gd75dc72
Git commit hash: d75dc721dd44902ea6fc72aafef88aad59977647
Git branch:      setWarnLog
Build timestamp: 2020-12-30 07:34:11Z
Go version:      go version go1.13.4 darwin/amd64

meet some unparsed arguments, please check again: [where id<5]
  1. example for status printer:
[2020/12/30 16:42:22.197 +08:00] [INFO] [status.go:37] [progress] [tables="0/1 (0.0%)"] ["finished rows"=42191631] ["finished size"=7.764GB] ["average speed(MiB/s)"=61.70014464561885]
[2020/12/30 16:44:22.197 +08:00] [INFO] [status.go:37] [progress] [tables="0/1 (0.0%)"] ["finished rows"=86052795] ["finished size"=15.83GB] ["average speed(MiB/s)"=64.1418352230345]

Related changes

  • Need to cherry-pick to the release branch

Release note

  • support check un-recognized arguments and print current progress during dumping

v4/export/status.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@tangenta tangenta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

kennytm
kennytm previously approved these changes Dec 30, 2020
tangenta
tangenta previously approved these changes Dec 30, 2020
@lichunzhu lichunzhu merged commit 22a1881 into pingcap:master Dec 30, 2020
@lichunzhu lichunzhu deleted the setWarnLog branch December 30, 2020 09:56
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
* refine log message, check unparsed arguments and add log status printer
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
* refine log message, check unparsed arguments and add log status printer
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
* refine log message, check unparsed arguments and add log status printer
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
* refine log message, check unparsed arguments and add log status printer
tisonkun pushed a commit to tisonkun/tidb that referenced this pull request Oct 20, 2021
* refine log message, check unparsed arguments and add log status printer
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dumpling should fail when provided positional parameters to CLI
3 participants