Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: prevent excessive repo-server disk usage for large repos (#8845) #8897

Merged
merged 8 commits into from
Mar 29, 2022

Conversation

crenshaw-dev
Copy link
Member

@crenshaw-dev crenshaw-dev commented Mar 25, 2022

Fixes #8845

Fetching specific commits allowed us to get commits pointing to non-standard refspecs (like refs/pulls/*/head).

But it costs a lot more disk space (demonstrated in the issue) and slows down app refreshes after commits.

This change defaults to the old behavior (fetch standard refspecs, checkout commit) and then falls back to fetching a specific commit if the checkout fails.

So if someone has an Application pointing to HEAD of a repo with 100k commits, this will efficiently fetch each commit without filling up the disk.

And if someone has an application pointing to refs/pulls/123/head, this will fail on the initial fetch and then fall back to fetching the specific commit. It's higher latency, but I think it's justified to avoid filling up the disk.

I've also confirmed there is no regression by testing this app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-cd
  namespace: argocd
spec:
  destination:
    name: ""
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: manifests/core-install
    repoURL: https://github.com/argoproj/argo-cd.git
    targetRevision: refs/pull/8897/head

@crenshaw-dev crenshaw-dev changed the title fix: prevent excessive repo-server disk usage for large repos fix: prevent excessive repo-server disk usage for large repos (#8845) Mar 25, 2022
@codecov
Copy link

codecov bot commented Mar 25, 2022

Codecov Report

Merging #8897 (a595331) into master (b1ff9db) will increase coverage by 0.03%.
The diff coverage is 69.23%.

@@            Coverage Diff             @@
##           master    #8897      +/-   ##
==========================================
+ Coverage   43.10%   43.13%   +0.03%     
==========================================
  Files         186      186              
  Lines       23359    23359              
==========================================
+ Hits        10069    10077       +8     
+ Misses      11849    11841       -8     
  Partials     1441     1441              
Impacted Files Coverage Δ
reposerver/repository/repository.go 60.08% <69.23%> (+0.85%) ⬆️
util/settings/settings.go 48.10% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1ff9db...a595331. Read the comment docs.

Signed-off-by: Michael Crenshaw <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>
@crenshaw-dev crenshaw-dev force-pushed the fetch-without-origin-by-default branch from 9515384 to 03d93c0 Compare March 25, 2022 14:55
Signed-off-by: Michael Crenshaw <[email protected]>
@crenshaw-dev crenshaw-dev force-pushed the fetch-without-origin-by-default branch from 0d0208b to 8ea0cba Compare March 25, 2022 15:01
Signed-off-by: Michael Crenshaw <[email protected]>
@crenshaw-dev crenshaw-dev marked this pull request as ready for review March 25, 2022 17:03
Signed-off-by: Michael Crenshaw <[email protected]>
@crenshaw-dev
Copy link
Member Author

Note: planning to run this internally for a week to see if it solves an internal problem we faced. This is sensitive code with a lot of edge cases, so we should be careful.

@crenshaw-dev crenshaw-dev added the cherry-pick/2.3 Candidate for cherry picking into the 2.3 release branch label Mar 25, 2022
Copy link
Collaborator

@alexmt alexmt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @crenshaw-dev ! Internal testing was successful. LGTM

@alexmt alexmt merged commit 3f51d92 into argoproj:master Mar 29, 2022
alexmt pushed a commit that referenced this pull request Mar 29, 2022
…#8897)

fix: prevent excessive repo-server disk usage for large repos (#8845) (#8897)

Signed-off-by: Michael Crenshaw <[email protected]>
@crenshaw-dev crenshaw-dev deleted the fetch-without-origin-by-default branch March 29, 2022 22:59
wojtekidd pushed a commit to wojtekidd/argo-cd that referenced this pull request Apr 25, 2022
…oj#8845) (argoproj#8897)

fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

Signed-off-by: Michael Crenshaw <[email protected]>
Signed-off-by: wojtekidd <[email protected]>
sujeilyfonseca added a commit to sujeilyfonseca/argo-cd that referenced this pull request Jun 24, 2022
* fix(ui): Applications page incorrectly resets to tiles view. Fixes argoproj#8702 (argoproj#8718)

Signed-off-by: Yuan Tang <[email protected]>

* fix: correct jsonnet paths resolution (argoproj#8721)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* chore: Bump stable version of application set addon (argoproj#8744)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* fix: Retry checkbox unchecked unexpectedly; Sync up with YAML (argoproj#8682) (argoproj#8720)

Signed-off-by: Keith Chong <[email protected]>

* Bump version to 2.3.1

* Bump version to 2.3.1

* Merge pull request from GHSA-2f5v-8r3f-8pww

* fix: application resource APIs must enforce project restrictions

Signed-off-by: Alexander Matyushentsev <[email protected]>

* Fix unit tests

Signed-off-by: jannfis <[email protected]>

Co-authored-by: jannfis <[email protected]>

* chore: remove lint-docs CI task (argoproj#8722) (argoproj#8858)

* chore: remove lint-docs CI task

Signed-off-by: Alexander Matyushentsev <[email protected]>

* chore: remove not longer necessary url-allow-list

Signed-off-by: Alexander Matyushentsev <[email protected]>

Co-authored-by: Alexander Matyushentsev <[email protected]>

* chore: fix imports (argoproj#8859)

Signed-off-by: Michael Crenshaw <[email protected]>

* Bump version to 2.3.2

* Bump version to 2.3.2

* fix: Set QPS and burst rate for resource ops client (argoproj#8915)

* fix: Set QPS and burst rate for resource ops client

Signed-off-by: jannfis <[email protected]>

* fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

Signed-off-by: Michael Crenshaw <[email protected]>

* fix: bump gitops engine version to v0.6.2

Signed-off-by: Alexander Matyushentsev <[email protected]>

* docs: update v2.4+ roadmap items (argoproj#8593)

Signed-off-by: ishitasequeira <[email protected]>

* docs: reflect v2.3 release changes in roadmap.md (argoproj#8747)

docs: reflect v2.3 release changes in roadmap.md (argoproj#8747)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* Bump version to 2.3.3

* Bump version to 2.3.3

* Add manifest for OnePipeline

(cherry picked from commit 15aa211080ef020e6a2ceaee9b845eb3259db237)

* Load additional resource overrides from dedicated ConfigMap

* Run unit tests

(cherry picked from commit 7605d5b0e2e816bb1cf9a29c5910c0fd511900c2)

* Install and config Git for unit tests

(cherry picked from commit 05dda11f6adf3191712b4598c8d55fe8ca1647a6)

* Add doc for changes

* feat: Argo CD CI pipeline changes (argoproj#4)

* updated cicd image

* upadted registry region

* updated one.pipeline.yaml to use the latest scripts

* updated makefile to add required targets

* feat: Argo CD v2.3.2 (argoproj#5)

* fix(ui): Applications page incorrectly resets to tiles view. Fixes argoproj#8702 (argoproj#8718)

Signed-off-by: Yuan Tang <[email protected]>

* fix: correct jsonnet paths resolution (argoproj#8721)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* fix: Retry checkbox unchecked unexpectedly; Sync up with YAML (argoproj#8682) (argoproj#8720)

Signed-off-by: Keith Chong <[email protected]>

* chore: Bump stable version of application set addon (argoproj#8744)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* Bump version to 2.3.1

* Bump version to 2.3.1

* Merge pull request from GHSA-2f5v-8r3f-8pww

* fix: application resource APIs must enforce project restrictions

Signed-off-by: Alexander Matyushentsev <[email protected]>

* Fix unit tests

Signed-off-by: jannfis <[email protected]>

Co-authored-by: jannfis <[email protected]>

* chore: remove lint-docs CI task (argoproj#8722) (argoproj#8858)

* chore: remove lint-docs CI task

Signed-off-by: Alexander Matyushentsev <[email protected]>

* chore: remove not longer necessary url-allow-list

Signed-off-by: Alexander Matyushentsev <[email protected]>

Co-authored-by: Alexander Matyushentsev <[email protected]>

* chore: fix imports (argoproj#8859)

Signed-off-by: Michael Crenshaw <[email protected]>

* Bump version to 2.3.2

* Bump version to 2.3.2

* feat: Updated CHANGES.md

Co-authored-by: Yuan Tang <[email protected]>
Co-authored-by: Alexander Matyushentsev <[email protected]>
Co-authored-by: Keith Chong <[email protected]>
Co-authored-by: argo-bot <[email protected]>
Co-authored-by: jannfis <[email protected]>
Co-authored-by: Michael Crenshaw <[email protected]>

* feat: Add .whitesource configuration file (argoproj#6)

Co-authored-by: whitesource-ets[bot] <328400+whitesource-ets[bot]@users.noreply.github.ibm.com>

* docs: CHANGES.md

Updated the CHANGES.md file to include updated
information about changes made.

Contributes to: automation-saas/native-AWS#1413

Signed-off-by: Sujeily Fonseca <[email protected]>

Co-authored-by: Yuan Tang <[email protected]>
Co-authored-by: Alexander Matyushentsev <[email protected]>
Co-authored-by: Keith Chong <[email protected]>
Co-authored-by: argo-bot <[email protected]>
Co-authored-by: jannfis <[email protected]>
Co-authored-by: Michael Crenshaw <[email protected]>
Co-authored-by: Ishita Sequeira <[email protected]>
Co-authored-by: Nikolas McGovern <[email protected]>
Co-authored-by: Rahul Mourya <[email protected]>
Co-authored-by: whitesource-ets[bot] <328400+whitesource-ets[bot]@users.noreply.github.ibm.com>
sujeilyfonseca added a commit to sujeilyfonseca/argo-cd that referenced this pull request Jul 7, 2022
* fix(ui): Applications page incorrectly resets to tiles view. Fixes argoproj#8702 (argoproj#8718)

Signed-off-by: Yuan Tang <[email protected]>

* fix: correct jsonnet paths resolution (argoproj#8721)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* chore: Bump stable version of application set addon (argoproj#8744)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* fix: Retry checkbox unchecked unexpectedly; Sync up with YAML (argoproj#8682) (argoproj#8720)

Signed-off-by: Keith Chong <[email protected]>

* Bump version to 2.3.1

* Bump version to 2.3.1

* Merge pull request from GHSA-2f5v-8r3f-8pww

* fix: application resource APIs must enforce project restrictions

Signed-off-by: Alexander Matyushentsev <[email protected]>

* Fix unit tests

Signed-off-by: jannfis <[email protected]>

Co-authored-by: jannfis <[email protected]>

* chore: remove lint-docs CI task (argoproj#8722) (argoproj#8858)

* chore: remove lint-docs CI task

Signed-off-by: Alexander Matyushentsev <[email protected]>

* chore: remove not longer necessary url-allow-list

Signed-off-by: Alexander Matyushentsev <[email protected]>

Co-authored-by: Alexander Matyushentsev <[email protected]>

* chore: fix imports (argoproj#8859)

Signed-off-by: Michael Crenshaw <[email protected]>

* Bump version to 2.3.2

* Bump version to 2.3.2

* fix: Set QPS and burst rate for resource ops client (argoproj#8915)

* fix: Set QPS and burst rate for resource ops client

Signed-off-by: jannfis <[email protected]>

* fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

fix: prevent excessive repo-server disk usage for large repos (argoproj#8845) (argoproj#8897)

Signed-off-by: Michael Crenshaw <[email protected]>

* fix: bump gitops engine version to v0.6.2

Signed-off-by: Alexander Matyushentsev <[email protected]>

* docs: update v2.4+ roadmap items (argoproj#8593)

Signed-off-by: ishitasequeira <[email protected]>

* docs: reflect v2.3 release changes in roadmap.md (argoproj#8747)

docs: reflect v2.3 release changes in roadmap.md (argoproj#8747)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* Bump version to 2.3.3

* Bump version to 2.3.3

* fix: Fix docs build error (argoproj#8895)

* work with specific jinja version

Signed-off-by: pashavictorovich <[email protected]>

* fix: fix broken monaco editor collapse icons (argoproj#8709)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* chore: upgrade to go 1.17.8 (argoproj#8866) (argoproj#9004)

* chore: upgrade to go 1.17.8

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: use 1.17 so it's always latest in the series

Signed-off-by: Michael Crenshaw <[email protected]>

* fix: allow cli/ui to follow logs (argoproj#8987) (argoproj#9065)

Signed-off-by: Daniel Helfand <[email protected]>

* Merge pull request from GHSA-xmg8-99r8-jc2j

Signed-off-by: Michael Crenshaw <[email protected]>

Co-authored-by: Michael Crenshaw <[email protected]>

* Merge pull request from GHSA-6gcg-hp2x-q54h

* fix: do not allow symlinks from directory-type applications

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: add new util file

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: lint

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: use t.TempDir for simpler tests

Signed-off-by: Michael Crenshaw <[email protected]>

* address comments

Signed-off-by: Michael Crenshaw <[email protected]>

* Merge pull request from GHSA-r642-gv9p-2wjj

Signed-off-by: jannfis <[email protected]>

Co-authored-by: Michael Crenshaw <[email protected]>

Co-authored-by: Michael Crenshaw <[email protected]>

* Bump version to 2.3.4

* Bump version to 2.3.4

* test: fix ErrorContains (argoproj#9445)

Signed-off-by: Michael Crenshaw <[email protected]>

* fix: missing Helm params (argoproj#9565) (argoproj#9566)

* fix: missing Helm params

Signed-off-by: Michael Crenshaw <[email protected]>

* use absolute paths, fix tests

Signed-off-by: Michael Crenshaw <[email protected]>

* fix race in test

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: upgrade golangci-lint to v1.46.2 (argoproj#9448)

* chore: upgrade golangci-lint to v1.46.2

Because:

* Installation of golangci-lint v1.45.2 is currently broken and fails
  silently due to a redacted dependency
  (blizzy78/varnamelen#13)

This commit:

* Upgrades golangci-lint to v1.46.2

Signed-off-by: Tommaso Sardelli <[email protected]>

* fix: lint

Signed-off-by: Michael Crenshaw <[email protected]>

* fix: lint

Signed-off-by: Tommaso Sardelli <[email protected]>

Co-authored-by: Michael Crenshaw <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

* fix: test race (argoproj#9469)

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: lint issues

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: update golangci-lint (argoproj#8988)

* chore: update golangci-lint

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: remove obsolete repo-server unit test (argoproj#9559)

Signed-off-by: Alexander Matyushentsev <[email protected]>

* chore: Make unit tests run on platforms other than amd64 (argoproj#8995)

Signed-off-by: jannfis <[email protected]>

Co-authored-by: Michael Crenshaw <[email protected]>
Signed-off-by: Michael Crenshaw <[email protected]>

* chore: eliminate go-mpatch dependency (argoproj#9045)

* chore: eliminate go-mpatch dependency

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: abstract out resource list function

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: don't exit the program in anything but the main function

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: better error messages

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: better error messages

Signed-off-by: Michael Crenshaw <[email protected]>

* test: directory app manifest generation (argoproj#9503)

* test: directory app manifest generation

Signed-off-by: Michael Crenshaw <[email protected]>

* git doesn't support empty dirs

Signed-off-by: Michael Crenshaw <[email protected]>

* Merge pull request from GHSA-h4w9-6x78-8vrj

Signed-off-by: Michael Crenshaw <[email protected]>

* Merge pull request from GHSA-2m7h-86qq-fp4v

Signed-off-by: Michael Crenshaw <[email protected]>

fix references

Signed-off-by: Michael Crenshaw <[email protected]>

use long enough state param for oauth2

Signed-off-by: Michael Crenshaw <[email protected]>

typo

Signed-off-by: Michael Crenshaw <[email protected]>

more entropy

Signed-off-by: Michael Crenshaw <[email protected]>

fix test

Signed-off-by: Michael Crenshaw <[email protected]>

* Merge pull request from GHSA-q4w5-4gq2-98vm

Signed-off-by: Michael Crenshaw <[email protected]>

* Merge pull request from GHSA-jhqp-vf4w-rpwq

Signed-off-by: Michael Crenshaw <[email protected]>

defer instead of multiple close calls

Signed-off-by: Michael Crenshaw <[email protected]>

oops

Signed-off-by: Michael Crenshaw <[email protected]>

don't count jsonnet against max

Signed-off-by: Michael Crenshaw <[email protected]>

fix codegen

Signed-off-by: Michael Crenshaw <[email protected]>

add caveat about 300x ratio

Signed-off-by: Michael Crenshaw <[email protected]>

fix versions

Signed-off-by: Michael Crenshaw <[email protected]>

fix tests/lint

Signed-off-by: Michael Crenshaw <[email protected]>

* chore: fix docs gen

Signed-off-by: Michael Crenshaw <[email protected]>

* Bump version to 2.3.5

* Bump version to 2.3.5

* docs: Changes for v2.3.5

Documented key decision factors to use Argo CD v2.3.5.

Contributes to: automation-saas/automation-saas/native-AWS#1972

Signed-off-by: Sujeily Fonseca <[email protected]>

Co-authored-by: Yuan Tang <[email protected]>
Co-authored-by: Alexander Matyushentsev <[email protected]>
Co-authored-by: Keith Chong <[email protected]>
Co-authored-by: argo-bot <[email protected]>
Co-authored-by: jannfis <[email protected]>
Co-authored-by: Michael Crenshaw <[email protected]>
Co-authored-by: Ishita Sequeira <[email protected]>
Co-authored-by: pasha-codefresh <[email protected]>
Co-authored-by: Daniel Helfand <[email protected]>
Co-authored-by: Tommaso Sardelli <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-pick/2.3 Candidate for cherry picking into the 2.3 release branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

repo-server duplicates git packfiles, filling up disk
2 participants