feat: add flag to skip deleting obsolete referrers index #957

qweeah · 2023-05-29T02:31:26Z

What this PR does / why we need it:
This PR adds a flag to oras attach, oras cp and oras manifest push. The flag can be used to specify whether to clean up old referrers index when referrers API is not available.

There is not a breaking change and by default, mentioned commands will still do referrers cleanup.

Depends on merging of #967.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #954, fixes #952, fixes #915

Please check the following list:

Does the affected code have corresponding tests, e.g. unit test, E2E test?
Does this change require a documentation update?
Does this introduce breaking changes that would require an announcement or bumping the major version?
Do all new files have an appropriate license header?

codecov-commenter · 2023-05-29T03:58:32Z

Codecov Report

Merging #957 (757f2ae) into main (f9aec91) will increase coverage by 0.30%.
The diff coverage is 85.71%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##             main     #957      +/-   ##
==========================================
+ Coverage   81.08%   81.39%   +0.30%     
==========================================
  Files          53       54       +1     
  Lines        2792     2816      +24     
==========================================
+ Hits         2264     2292      +28     
+ Misses        358      355       -3     
+ Partials      170      169       -1

Impacted Files	Coverage Δ
cmd/oras/internal/option/referrers.go	`55.55% <55.55%> (ø)`
cmd/oras/internal/errors/errors.go	`100.00% <100.00%> (ø)`
cmd/oras/root/attach.go	`85.48% <100.00%> (+2.98%)`	⬆️
cmd/oras/root/cp.go	`86.86% <100.00%> (+0.49%)`	⬆️
cmd/oras/root/manifest/push.go	`77.27% <100.00%> (+4.40%)`	⬆️

cmd/oras/internal/option/referrers.go

Signed-off-by: Billy Zha <[email protected]>

shizhMSFT

What about the manifest delete path?

cmd/oras/internal/option/referrers.go

qweeah · 2023-06-29T10:28:46Z

What about the manifest delete path?

Referrers index deletion happens after manifest deletion right? If the registry already supports deletion and the token used covers manifest deletion, why would referrers index fail?

Also why would user want to skip referrers index deletion when manifest deletion is successful?

Signed-off-by: Billy Zha <[email protected]>

shizhMSFT · 2023-06-30T06:52:10Z

Referrers index deletion happens after manifest deletion right? If the registry already supports deletion and the token used covers manifest deletion, why would referrers index fail?

LoL. Makes sense.

cmd/oras/internal/option/referrers.go

Signed-off-by: Billy Zha <[email protected]>

shizhMSFT

LGTM

shizhMSFT

LGTM

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <[email protected]>

…t#957) Signed-off-by: Billy Zha <[email protected]> Co-authored-by: Terry Howe <[email protected]>

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <[email protected]>

…as-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <[email protected]>

…)" (#1046) Signed-off-by: Billy Zha <[email protected]>

…as-project#957)" (oras-project#1046) Signed-off-by: Billy Zha <[email protected]>

qweeah force-pushed the gc-referrers branch from 5ebe86e to 3819f7d Compare May 29, 2023 03:55

qweeah force-pushed the gc-referrers branch from 193980f to 3151255 Compare May 30, 2023 07:25

qweeah marked this pull request as ready for review June 1, 2023 00:33

qweeah requested review from sajayantony, shizhMSFT, SteveLasker and TerryHowe as code owners June 1, 2023 00:33

qweeah force-pushed the gc-referrers branch from d37ec9b to b16519e Compare June 1, 2023 00:33

Wwwsylvia reviewed Jun 2, 2023

View reviewed changes

cmd/oras/internal/option/referrers.go Show resolved Hide resolved

qweeah changed the title ~~feat!: add flag for referrers garbage collection~~ feat!: add flag to skip deleting obsolete referrers index Jun 2, 2023

qweeah added 15 commits June 6, 2023 01:32

feat!: add flag for referrers GC

8b22600

Signed-off-by: Billy Zha <[email protected]>

update go mod

7b7e5a4

Signed-off-by: Billy Zha <[email protected]>

fix e2e

e27d723

Signed-off-by: Billy Zha <[email protected]>

add e2e

57b4e75

Signed-off-by: Billy Zha <[email protected]>

update deps

adfcca1

Signed-off-by: Billy Zha <[email protected]>

update dep

cacd6bb

Signed-off-by: Billy Zha <[email protected]>

update

f0ecebf

Signed-off-by: Billy Zha <[email protected]>

update version

ffbd8bc

Signed-off-by: Billy Zha <[email protected]>

mod tidy

d5f1937

Signed-off-by: Billy Zha <[email protected]>

fix e2e

abae524

Signed-off-by: Billy Zha <[email protected]>

rename flag

06d382d

Signed-off-by: Billy Zha <[email protected]>

fix: default to true

86f3b1d

Signed-off-by: Billy Zha <[email protected]>

fix e2e

19c4516

Signed-off-by: Billy Zha <[email protected]>

change default value to false

623dc99

Signed-off-by: Billy Zha <[email protected]>

fix e2e

aa73bc9

Signed-off-by: Billy Zha <[email protected]>

qweeah force-pushed the gc-referrers branch from 1e65493 to aa73bc9 Compare June 6, 2023 01:32

fix e2e

aa91493

Signed-off-by: Billy Zha <[email protected]>

qweeah changed the title ~~feat!: add flag to skip deleting obsolete referrers index~~ feat: add flag to skip deleting obsolete referrers index Jun 6, 2023

refactor: simplify e2e scripts and github action

06af37b

Signed-off-by: Billy Zha <[email protected]>

qweeah added 2 commits June 28, 2023 01:37

Merge branch 'main' into gc-referrers

bec6661

Merge branch 'main' into gc-referrers

a69435d

shizhMSFT added this to the v1.1.0 milestone Jun 28, 2023

qweeah mentioned this pull request Jun 28, 2023

Improve the error msg for OCI v1.0 registries or registries that don't support deletion API #915

Closed

1 task

shizhMSFT reviewed Jun 29, 2023

View reviewed changes

cmd/oras/internal/option/referrers.go Outdated Show resolved Hide resolved

qweeah added 2 commits June 29, 2023 15:28

Merge remote-tracking branch 'origin-upstream/main' into gc-referrers

fdc59b8

bug fix

6b92395

Signed-off-by: Billy Zha <[email protected]>

shizhMSFT reviewed Jun 30, 2023

View reviewed changes

cmd/oras/internal/option/referrers.go Outdated Show resolved Hide resolved

update comment

757f2ae

Signed-off-by: Billy Zha <[email protected]>

shizhMSFT approved these changes Jul 3, 2023

View reviewed changes

Merge branch 'main' into gc-referrers

474c798

shizhMSFT approved these changes Jul 3, 2023

View reviewed changes

qweeah merged commit ba60b99 into oras-project:main Jul 3, 2023

This was referenced Jul 5, 2023

Vote for releasing v1.1.0-rc.1 #1002

Closed

bump: tag and release ORAS CLI v1.1.0-rc.1 #1016

Merged

shizhMSFT mentioned this pull request Jul 26, 2023

Request to adjust the release plan #1038

Closed

qweeah added a commit to qweeah/oras that referenced this pull request Aug 3, 2023

Revert "feat: add flag to skip deleting obsolete referrers index (ora…

078dd8e

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <[email protected]>

shizhMSFT pushed a commit to shizhMSFT/oras that referenced this pull request Aug 3, 2023

Revert "feat: add flag to skip deleting obsolete referrers index (ora…

fb3dd72

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <[email protected]>

shizhMSFT pushed a commit to shizhMSFT/oras that referenced this pull request Aug 3, 2023

feat: add flag to skip deleting obsolete referrers index (oras-projec…

5e6d3bf

…t#957) Signed-off-by: Billy Zha <[email protected]> Co-authored-by: Terry Howe <[email protected]>

shizhMSFT pushed a commit to shizhMSFT/oras that referenced this pull request Aug 3, 2023

Revert "feat: add flag to skip deleting obsolete referrers index (ora…

6f6c9c1

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <[email protected]>

qweeah added a commit to qweeah/oras that referenced this pull request Aug 3, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

d25f652

…as-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <[email protected]>

qweeah added a commit to qweeah/oras that referenced this pull request Aug 3, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

73b531d

…as-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <[email protected]>

qweeah added a commit to qweeah/oras that referenced this pull request Aug 3, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

06700fb

…as-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <[email protected]>

qweeah added a commit that referenced this pull request Aug 3, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (#957…

43e4832

…)" (#1046) Signed-off-by: Billy Zha <[email protected]>

ZhongRuoyu mentioned this pull request Aug 7, 2023

oras 1.0.1 Homebrew/homebrew-core#138923

Merged

qweeah added a commit to qweeah/oras that referenced this pull request Aug 9, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

b3ac818

…as-project#957)" (oras-project#1046) Signed-off-by: Billy Zha <[email protected]>

qweeah added a commit to qweeah/oras that referenced this pull request Aug 9, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

44c61b9

…as-project#957)" (oras-project#1046) Signed-off-by: Billy Zha <[email protected]>

shizhMSFT mentioned this pull request Jan 8, 2024

manifest delete is leaking indexes when deleting client-managed referrer #1228

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add flag to skip deleting obsolete referrers index #957

feat: add flag to skip deleting obsolete referrers index #957

qweeah commented May 29, 2023 •

edited

Loading

codecov-commenter commented May 29, 2023 •

edited

Loading

shizhMSFT left a comment

qweeah commented Jun 29, 2023 •

edited

Loading

shizhMSFT commented Jun 30, 2023

shizhMSFT left a comment

shizhMSFT left a comment

feat: add flag to skip deleting obsolete referrers index #957

feat: add flag to skip deleting obsolete referrers index #957

Conversation

qweeah commented May 29, 2023 • edited Loading

codecov-commenter commented May 29, 2023 • edited Loading

Codecov Report

shizhMSFT left a comment

Choose a reason for hiding this comment

qweeah commented Jun 29, 2023 • edited Loading

shizhMSFT commented Jun 30, 2023

shizhMSFT left a comment

Choose a reason for hiding this comment

shizhMSFT left a comment

Choose a reason for hiding this comment

qweeah commented May 29, 2023 •

edited

Loading

codecov-commenter commented May 29, 2023 •

edited

Loading

qweeah commented Jun 29, 2023 •

edited

Loading