Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

avoid unnecessary fsync in ps #7616

Merged
merged 8 commits into from
Jun 9, 2023
Merged

avoid unnecessary fsync in ps #7616

merged 8 commits into from
Jun 9, 2023

Conversation

lidezhu
Copy link
Contributor

@lidezhu lidezhu commented Jun 8, 2023 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #6827 close #7736

Problem Summary:

  1. Avoid unnecessary sync when writing to wal, after optimization, the write performance is about 14MB/s(previous is about 7MB/s)
    image

  2. Add some metrics
    image
    image

  3. Add some unit tests for pipeline write

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-linked-issue size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 8, 2023
@lidezhu lidezhu force-pushed the improve-write-performance branch from d56b85d to eca4ff6 Compare June 8, 2023 03:22
@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 8, 2023
@lidezhu lidezhu force-pushed the improve-write-performance branch from eca4ff6 to 2b0339d Compare June 8, 2023 05:38
@ti-chi-bot ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 8, 2023
@lidezhu lidezhu force-pushed the improve-write-performance branch from 2b0339d to 1607a0e Compare June 8, 2023 05:39
@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 8, 2023
@lidezhu lidezhu force-pushed the improve-write-performance branch from 63336ab to f77342d Compare June 8, 2023 06:02
@lidezhu lidezhu changed the title Avoid unnecessary fsync in ps avoid unnecessary fsync in ps Jun 8, 2023
@JaySon-Huang
Copy link
Contributor

/rebuild

1 similar comment
@JaySon-Huang
Copy link
Contributor

/rebuild

@lidezhu lidezhu force-pushed the improve-write-performance branch from f33f043 to db8853a Compare June 8, 2023 10:08
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 8, 2023
@lidezhu lidezhu force-pushed the improve-write-performance branch from 6d6b130 to 6c2f9c3 Compare June 8, 2023 10:31
JaySon-Huang
JaySon-Huang approved these changes Jun 9, 2023
View reviewed changes
Copy link
Contributor

@JaySon-Huang JaySon-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jun 9, 2023
@lidezhu
Copy link
Contributor Author

lidezhu commented Jun 9, 2023

/run-all-tests

@ti-chi-bot ti-chi-bot bot added the lgtm label Jun 9, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jun 9, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [JaySon-Huang,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 9, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jun 9, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-06-09 07:59:09.547136712 +0000 UTC m=+71563.795058530: ☑️ agreed by JaySon-Huang.
  • 2023-06-09 09:56:44.591057209 +0000 UTC m=+78618.838979029: ☑️ agreed by Lloyd-Pottiger.

@ti-chi-bot ti-chi-bot bot merged commit 5b90c7e into pingcap:master Jun 9, 2023
@lidezhu lidezhu deleted the improve-write-performance branch June 9, 2023 10:07
@lidezhu lidezhu added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. labels Jul 4, 2023
@ti-chi-bot ti-chi-bot mentioned this pull request Jul 4, 2023
Merged
12 tasks
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #7733.

@ti-chi-bot ti-chi-bot mentioned this pull request Jul 4, 2023
Merged
12 tasks
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #7734.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Jul 4, 2023
@lidezhu @ti-chi-bot
This is an automated cherry-pick of pingcap#7616
ab4877f 
Signed-off-by: ti-chi-bot <[email protected]>
ti-chi-bot bot pushed a commit that referenced this pull request Jul 4, 2023
@ti-chi-bot
avoid unnecessary fsync in ps (#7616) (#7733)
de01024 
ref #6827, close #7736
ti-chi-bot bot pushed a commit that referenced this pull request Jul 4, 2023
@ti-chi-bot
avoid unnecessary fsync in ps (#7616) (#7734)
6e18968 
ref #6827, close #7736
lidezhu pushed a commit to lidezhu/tics that referenced this pull request Jul 10, 2023
@ti-chi-bot @lidezhu
avoid unnecessary fsync in ps (pingcap#7616) (pingcap#7734)
66487a8 
ref pingcap#6827, close pingcap#7736
@lidezhu lidezhu mentioned this pull request Jul 10, 2023
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Lloyd-Pottiger Lloyd-Pottiger Lloyd-Pottiger approved these changes

@JaySon-Huang JaySon-Huang JaySon-Huang approved these changes

@flowbehappy flowbehappy Awaiting requested review from flowbehappy

Assignees
No one assigned
Labels
approved lgtm needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Fix the problem that there is some unnecessary fsync in ps
4 participants
@lidezhu @JaySon-Huang @ti-chi-bot @Lloyd-Pottiger