Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reduce the high db cpu usage for tag retention #17296

Merged
merged 1 commit into from
Aug 4, 2022

Conversation

chlins
Copy link
Member

@chlins chlins commented Aug 3, 2022

  1. Add two indexes to database migrations.
  2. Skip refresh quota in middleware for requests from jobservice.
  3. Refresh quota by self in the end of tag retention job.

Closes: #14708

Signed-off-by: chlins [email protected]

Thank you for contributing to Harbor!

Comprehensive Summary of your change

Issue being fixed

Fixes #14708

Please indicate you've done the following:

  • Well Written Title and Summary of the PR
  • Label the PR as needed. "release-note/ignore-for-release, release-note/new-feature, release-note/update, release-note/enhancement, release-note/community, release-note/breaking-change, release-note/docs, release-note/infra, release-note/deprecation"
  • Accepted the DCO. Commits without the DCO will delay acceptance.
  • Made sure tests are passing and test coverage is added if needed.
  • Considered the docs impact and opened a new docs issue or PR with docs changes if needed in website repository.

@chlins chlins added area/performance area/tag-retention Issues relating to the tag retention policies features release-note/update Update or Fix labels Aug 3, 2022
@chlins chlins requested a review from a team as a code owner August 3, 2022 02:35
@codecov
Copy link

codecov bot commented Aug 3, 2022

Codecov Report

❗ No coverage uploaded for pull request base (main@bf741ad). Click here to learn what that means.
The diff coverage is 48.48%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main   #17296   +/-   ##
=======================================
  Coverage        ?   66.97%           
=======================================
  Files           ?      993           
  Lines           ?    83633           
  Branches        ?     2677           
=======================================
  Hits            ?    56011           
  Misses          ?    23742           
  Partials        ?     3880           
Flag Coverage Δ
unittests 66.97% <48.48%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/controller/quota/options.go 55.55% <0.00%> (ø)
src/server/middleware/quota/quota.go 83.49% <0.00%> (ø)
src/pkg/retention/job.go 55.22% <23.07%> (ø)
src/controller/quota/controller.go 44.26% <33.33%> (ø)
src/pkg/clients/core/client.go 75.00% <100.00%> (ø)
src/pkg/retention/dep/client.go 68.25% <100.00%> (ø)
src/server/middleware/security/utils.go 100.00% <100.00%> (ø)

Copy link
Contributor

@wy65701436 wy65701436 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@chlins chlins force-pushed the fix/tag-retention-high-cpu branch 3 times, most recently from 51f21ed to ca6ae45 Compare August 3, 2022 09:14
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.

Closes: goharbor#14708

Signed-off-by: chlins <[email protected]>
@chlins chlins force-pushed the fix/tag-retention-high-cpu branch from ca6ae45 to 2ba3a67 Compare August 3, 2022 09:38
@chlins chlins merged commit 70a95a9 into goharbor:main Aug 4, 2022
sluetze pushed a commit to sluetze/harbor that referenced this pull request Oct 29, 2022
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.

Closes: goharbor#14708

Signed-off-by: chlins <[email protected]>
mcsage pushed a commit to mcsage/harbor that referenced this pull request Feb 16, 2023
1. Add two indexes to database migrations.
2. Skip refresh quota in middleware for requests from jobservice.
3. Refresh quota by self in the end of tag retention job.

Closes: goharbor#14708

Signed-off-by: chlins <[email protected]>
Signed-off-by: Stephan Hohn <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/performance area/tag-retention Issues relating to the tag retention policies features release-note/update Update or Fix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tag deletion performance is very poor with a high number of tags
4 participants