Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

task: implement parallel concurrency limit #1420

Merged
merged 3 commits into from
Jun 15, 2021

Conversation

AstroProfundis
Copy link
Contributor

What problem does this PR solve?

Close #649

What is changed and how it works?

  • Add a concurrency threshold in context
  • Implement concurrency limiter in task.Execute for parallel tasks
  • Add a global argument to tiup-cluster and tiup-dm to let user set the concurrency

The concurrency is default to 5 as set in command line argument, and is the number of CPU threads for task if not set.

Check List

Tests

  • Manual test (add detailed scripts or steps below)

Code changes

  • Has exported function/method change
  • Has exported variable/fields change

Related changes

  • Need to update the documentation

Release notes:

task: implement parallel concurrency limit

@AstroProfundis AstroProfundis added type/new-feature Categorizes pr as related to a new feature. category/stability Categorizes issue or PR as a stability enhancement. labels Jun 10, 2021
@AstroProfundis AstroProfundis added this to the v1.6.0 milestone Jun 10, 2021
@AstroProfundis AstroProfundis requested a review from lucklove June 10, 2021 08:51
@AstroProfundis AstroProfundis self-assigned this Jun 10, 2021
@ti-chi-bot ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 10, 2021
@AstroProfundis AstroProfundis added component/cluster Issues about the tiup-cluster component component/dm Issues about the tiup-dm component labels Jun 10, 2021
@codecov-commenter
Copy link

codecov-commenter commented Jun 10, 2021

Codecov Report

Merging #1420 (194a8f9) into master (e5e1b46) will increase coverage by 15.50%.
The diff coverage is 55.76%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #1420       +/-   ##
===========================================
+ Coverage   25.39%   40.90%   +15.50%     
===========================================
  Files         266      291       +25     
  Lines       20628    22525     +1897     
===========================================
+ Hits         5238     9213     +3975     
+ Misses      14586    11828     -2758     
- Partials      804     1484      +680     
Flag Coverage Δ
dm 24.08% <54.90%> (?)
integrate 32.24% <54.90%> (+16.77%) ⬆️
playground 13.73% <0.00%> (?)
tiup 15.46% <0.00%> (-0.01%) ⬇️
unittest 21.73% <6.25%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
components/dm/command/import.go 19.04% <0.00%> (ø)
pkg/cluster/ansible/config.go 0.00% <0.00%> (ø)
pkg/cluster/ansible/service.go 3.53% <0.00%> (ø)
pkg/cluster/manager/basic.go 50.48% <0.00%> (+50.48%) ⬆️
pkg/cluster/manager/check.go 0.00% <0.00%> (ø)
pkg/cluster/manager/cleanup.go 0.00% <0.00%> (ø)
pkg/cluster/manager/deploy.go 59.80% <0.00%> (+59.80%) ⬆️
pkg/cluster/manager/destroy.go 30.98% <0.00%> (+30.98%) ⬆️
pkg/cluster/manager/patch.go 0.00% <0.00%> (ø)
pkg/cluster/manager/reload.go 60.65% <0.00%> (+60.65%) ⬆️
... and 165 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5e1b46...194a8f9. Read the comment docs.

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • lucklove

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 15, 2021
@lucklove
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: d802b89

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 15, 2021
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Jun 15, 2021
@AstroProfundis
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 194a8f9

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 15, 2021
@ti-chi-bot ti-chi-bot merged commit 6c92a87 into pingcap:master Jun 15, 2021
@AstroProfundis AstroProfundis deleted the limiter branch June 15, 2021 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category/stability Categorizes issue or PR as a stability enhancement. component/cluster Issues about the tiup-cluster component component/dm Issues about the tiup-dm component size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. type/new-feature Categorizes pr as related to a new feature.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support control concurrency in tiup-cluster
4 participants