Improve dashboard's repo list performance #18963

Gusted · 2022-03-01T22:55:37Z

Avoid "a few" database lookups for each repo that's being requested, only return minimal information which doesn't require a database operation for each repo. This is general change for the repo API, so other use-cases of this will also benefit from it.
Makes fetching these list faster.
Less CPU overhead when a user visits home page.
Unintentionally removed some jQuery code.

- Avoid a lot of database lookups for all the repo's, by adding a undocumented "minimal" mode for this specific task, which returns the data that's only needed by this list which doesn't require any database lookups. - Makes fetching these list faster. - Less CPU overhead when a user visits home page.

web_src/js/components/DashboardRepoList.js

lunny · 2022-03-02T00:41:06Z

I would like we split api/v1 and UI API to make the things simple. This is a chance to make a new route but not reuse the old one.

Gusted · 2022-03-02T10:41:43Z

I would like we split api/v1 and UI API to make the things simple. This is a chance to make a new route but not reuse the old one.

Sounds like a good idea to me, which route do you propose for it? I guess api/internal/... or api/ui/...?

- Use async in the function so we can use `await`. - Remove `archivedFilter` check for count, as it doesn't make sense to show the count of repos when you can't even see them(as they are filited away).

web_src/js/components/DashboardRepoList.js

6543 · 2022-03-03T16:14:15Z

... -> #16052

6543 · 2022-03-14T11:08:33Z

I'm for adding a new ui route - we can migrate one func at a time ... just start now

(UI routes dont have the issue to need to be/stay backwards compatible)

Gusted · 2022-03-14T13:03:27Z

just start now

I don't think that's possible, #16052 adds many new things and as well it's own file for the new API, it will result in conflict files for that PR if I try remotely to migrate it to a new route.

6543 · 2022-04-07T19:01:14Z

#19318 got merged ... please ajust this pull @Gusted

Gusted · 2022-04-11T22:20:50Z

@6543 Updated it.

routers/api/v1/repo/repo.go

codecov-commenter · 2022-04-12T15:58:57Z

Codecov Report

Merging #18963 (7319f8d) into main (deffe9e) will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@           Coverage Diff           @@
##             main   #18963   +/-   ##
=======================================
  Coverage   47.37%   47.37%           
=======================================
  Files         949      949           
  Lines      132101   132102    +1     
=======================================
+ Hits        62583    62588    +5     
+ Misses      61987    61982    -5     
- Partials     7531     7532    +1

Impacted Files	Coverage Δ
routers/api/v1/repo/repo.go	`66.55% <ø> (ø)`
routers/web/repo/repo.go	`24.37% <0.00%> (-0.06%)`	⬇️
modules/process/manager_exec.go	`85.36% <0.00%> (-7.32%)`	⬇️
models/unit/unit.go	`46.08% <0.00%> (-1.74%)`	⬇️
models/repo_list.go	`76.15% <0.00%> (+0.48%)`	⬆️
modules/queue/workerpool.go	`53.62% <0.00%> (+2.07%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update deffe9e...7319f8d. Read the comment docs.

routers/web/repo/repo.go

web_src/js/components/DashboardRepoList.js

Co-authored-by: delvh <[email protected]>

routers/web/repo/repo.go

Co-authored-by: delvh <[email protected]>

routers/web/repo/repo.go

web_src/js/components/DashboardRepoList.js

Co-authored-by: wxiaoguang <[email protected]>

wxiaoguang

CI seems down .... wait for the pass

zeripath · 2022-04-26T17:54:19Z

make lgtm work

* giteaofficial/main: (21 commits) Prevent intermittent race in attribute reader close (go-gitea#19537) Make repository file list useable on mobile (go-gitea#19515) Update image URL for Discord webhook (go-gitea#19536) [skip ci] Updated translations via Crowdin Fix 64-bit atomic operations on 32-bit machines (go-gitea#19531) Fix `upgrade.sh` script error with `su -c` (go-gitea#19483) When view _Siderbar or _Footer, just display once (go-gitea#19501) Fix migrate release from github (go-gitea#19510) Prevent dangling archiver goroutine (go-gitea#19516) Don't let repo clone URL overflow (go-gitea#19517) Add commit status popup to issuelist (go-gitea#19375) Disable unnecessary GitHooks elements (go-gitea#18485) Disable unnecessary GitHooks elements Improve dashboard's repo list performance (go-gitea#18963) By default force vertical tabs on mobile (go-gitea#19486) Refactor readme file renderer (go-gitea#19502) Allow package dump skipping (go-gitea#19506) Unset git author/committer variables when running integration tests (go-gitea#19512) Allow commit status popup on /pulls page (go-gitea#19507) Use router param for filepath in GetRawFile (go-gitea#19499) ...

* Improve dashboard's repo list performance - Avoid a lot of database lookups for all the repo's, by adding a undocumented "minimal" mode for this specific task, which returns the data that's only needed by this list which doesn't require any database lookups. - Makes fetching these list faster. - Less CPU overhead when a user visits home page. * Refactor javascript code + fix Fork icon - Use async in the function so we can use `await`. - Remove `archivedFilter` check for count, as it doesn't make sense to show the count of repos when you can't even see them(as they are filited away). * Add `count_only` * Remove uncessary code * Improve comment Co-authored-by: delvh <[email protected]> * Update web_src/js/components/DashboardRepoList.js Co-authored-by: delvh <[email protected]> * Update web_src/js/components/DashboardRepoList.js Co-authored-by: delvh <[email protected]> * By default apply minimal mode * Remove `minimal` paramater * Refactor count header * Simplify init Co-authored-by: wxiaoguang <[email protected]> Co-authored-by: delvh <[email protected]> Co-authored-by: wxiaoguang <[email protected]> Co-authored-by: zeripath <[email protected]>

Gusted added this to the 1.17.0 milestone Mar 1, 2022

Gusted added the performance/cpu label Mar 1, 2022

Gusted commented Mar 1, 2022

View reviewed changes

web_src/js/components/DashboardRepoList.js Outdated Show resolved Hide resolved

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Mar 1, 2022

Gusted added 2 commits March 2, 2022 12:10

Refactor javascript code + fix Fork icon

22acfd6

- Use async in the function so we can use `await`. - Remove `archivedFilter` check for count, as it doesn't make sense to show the count of repos when you can't even see them(as they are filited away).

Merge branch 'main' into improve-dashboard-list

931de98

wxiaoguang reviewed Mar 2, 2022

View reviewed changes

web_src/js/components/DashboardRepoList.js Outdated Show resolved Hide resolved

wxiaoguang reviewed Mar 2, 2022

View reviewed changes

web_src/js/components/DashboardRepoList.js Outdated Show resolved Hide resolved

Add count_only

c6706e6

silverwind approved these changes Mar 2, 2022

View reviewed changes

GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Mar 2, 2022

Merge branch 'main' into improve-dashboard-list

964527b

lafriks reviewed Apr 12, 2022

View reviewed changes

routers/api/v1/repo/repo.go Outdated Show resolved Hide resolved

Remove uncessary code

53199d4

delvh reviewed Apr 12, 2022

View reviewed changes

Gusted and others added 2 commits April 14, 2022 16:37

Improve comment

47926f5

Co-authored-by: delvh <[email protected]>

Update web_src/js/components/DashboardRepoList.js

1b42dd2

Co-authored-by: delvh <[email protected]>

wxiaoguang reviewed Apr 15, 2022

View reviewed changes

routers/web/repo/repo.go Outdated Show resolved Hide resolved

Gusted and others added 2 commits April 15, 2022 10:47

Update web_src/js/components/DashboardRepoList.js

1444f06

Co-authored-by: delvh <[email protected]>

By default apply minimal mode

19f811f

Gusted added 2 commits April 15, 2022 19:28

Remove minimal paramater

3b67a8c

Merge branch 'main' into improve-dashboard-list

7319f8d

delvh reviewed Apr 15, 2022

View reviewed changes

routers/web/repo/repo.go Outdated Show resolved Hide resolved

web_src/js/components/DashboardRepoList.js Show resolved Hide resolved

Gusted added 2 commits April 25, 2022 21:35

Refactor count header

3b91ad4

Merge branch 'main' into improve-dashboard-list

84bf230

delvh approved these changes Apr 26, 2022

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Apr 26, 2022

Merge branch 'main' into improve-dashboard-list

cb4f7d2

wxiaoguang reviewed Apr 26, 2022

View reviewed changes

web_src/js/components/DashboardRepoList.js Outdated Show resolved Hide resolved

Simplify init

fab1d89

Co-authored-by: wxiaoguang <[email protected]>

wxiaoguang approved these changes Apr 26, 2022

View reviewed changes

Merge branch 'main' into improve-dashboard-list

7594040

Merge branch 'main' into improve-dashboard-list

b0ded7f

techknowlogick merged commit 076eaad into go-gitea:main Apr 26, 2022

Gusted deleted the improve-dashboard-list branch April 26, 2022 20:35

go-gitea locked and limited conversation to collaborators May 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve dashboard's repo list performance #18963

Improve dashboard's repo list performance #18963

Gusted commented Mar 1, 2022 •

edited

Loading

lunny commented Mar 2, 2022

Gusted commented Mar 2, 2022

6543 commented Mar 3, 2022

6543 commented Mar 14, 2022 •

edited

Loading

Gusted commented Mar 14, 2022

6543 commented Apr 7, 2022

Gusted commented Apr 11, 2022

codecov-commenter commented Apr 12, 2022 •

edited

Loading

wxiaoguang left a comment

zeripath commented Apr 26, 2022

Improve dashboard's repo list performance #18963

Improve dashboard's repo list performance #18963

Conversation

Gusted commented Mar 1, 2022 • edited Loading

lunny commented Mar 2, 2022

Gusted commented Mar 2, 2022

6543 commented Mar 3, 2022

6543 commented Mar 14, 2022 • edited Loading

Gusted commented Mar 14, 2022

6543 commented Apr 7, 2022

Gusted commented Apr 11, 2022

codecov-commenter commented Apr 12, 2022 • edited Loading

Codecov Report

wxiaoguang left a comment

Choose a reason for hiding this comment

zeripath commented Apr 26, 2022

Gusted commented Mar 1, 2022 •

edited

Loading

6543 commented Mar 14, 2022 •

edited

Loading

codecov-commenter commented Apr 12, 2022 •

edited

Loading