Releases are deleted when mirror repository is synchronized #22066

kotovalexarian · 2022-12-07T23:46:47Z

Description

By some reason the created releases are deleted when mirror repository is synchronized even when the tags have not been recreated.

Steps

Sign-in to Gitea
Click "+"
Click "New Migration"
Click "GitHub"
Put https://github.com/tailix/libkernaux in the field "Migrate / Clone From URL"
Check "This repository will be a mirror"
Click the button "Create repository"
Click "7 tags" in the bar (the number may be different)
Click "New release" on tag "v0.6.1"
Put foobar in the field "Title"
Click "Publish release"
Click on the tab "Settings"
Click the button "Synchronize Now"
Wait a few (2-5) minutes
Refresh the page

Expected

You should see the label "Releases [1]" (the number as a badge) among the tabs.

Got

You see the label "Releases" (without a number) among the tabs.

Gitea Version

1.17.3 (self-hosted), 1.19.0+dev-166-g0a85537c7 (try.gitea.io)

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

No response

Screenshots

No response

Git Version

2.37.2

Operating System

Ubuntu 22.10

How are you running Gitea?

https://try.gitea.io & self-hosted

Database

PostgreSQL

The text was updated successfully, but these errors were encountered:

lunny · 2022-12-08T05:09:53Z

Hm, maybe we should not allow to create release on a mirror repository?

kotovalexarian · 2022-12-09T07:31:35Z

Why not keep them on sync?

iarp · 2022-12-15T21:31:22Z

When setting up a migration from github and selecting the mirroring option, this disables the ability to sync labels, issues, pull requests, releases, milestones. Why would those be touched anyways during a sync when they're not configurable?

I just setup a script to do the mirroring of releases and their assets on a whole bunch of repos only to find everything gone the next day. The attachments still exist in the attachments directory but are no longer associated with any release.

dboreham · 2023-04-12T13:17:01Z

Also interested in addressing this problem which arises when you use the new CI features in Gitea on a mirrored repository, with CI publishing local releases.

mrexodia · 2023-07-19T00:38:43Z

Any pointers where to start fixing this in the code?

lunny · 2023-07-19T03:46:46Z

My point is to prevent users to create releases on a mirror repository currently.
The second method is to not delete releases when syncing tags.

dboreham · 2023-07-19T03:58:56Z

Please don't prevent release creation on a mirror -- this is how we test/debug our CI processes.

lunny · 2023-07-19T04:05:03Z

Please don't prevent release creation on a mirror -- this is how we test/debug our CI processes.

OK. Then you would like the second option. But how should Gitea handle when remote tags are deleted from your view point?

dboreham · 2023-07-19T04:09:51Z

Do you mean local tags?

lunny · 2023-07-19T04:13:01Z

Do you mean local tags?

I mean as a mirror repository, if the source repository in the remote deleted some tags, when syncing the mirror repository, whether Gitea need to delete the release after the tags deleted?

mrexodia · 2023-07-19T08:06:09Z

I would suggest to handle it the same as GitHub does. I think if you delete a tag for a release that release gets deleted or converted to draft (not 100% sure). Deleting the release in all cases is undesirable for me.

In my mirror structure I want to manually (with the API) re-create the releases and mirror the binaries too.

dboreham · 2023-07-19T14:33:54Z

Do you mean local tags?

I mean as a mirror repository, if the source repository in the remote deleted some tags, when syncing the mirror repository, whether Gitea need to delete the release after the tags deleted?

Not sure I care much about tags being deleted in the source repo since I'd never do that. I thought this problem was about this:

Release created in mirror repo -- a release has a 1:1 relationship with a tag, implying that a tag must be created in the mirror in order to create a release there.
Now sync from the source repo.
The mirror is supposed to be read-only, but in step 1 we create a tag locally, hence it is not read-only.
What to do about that? Delete the locally created tags, so the repo matches the source? But that means deleting the release.

lunny · 2023-07-24T06:02:15Z

Yes, that's why I think we need to disable creating releases in a mirror repository.

dboreham · 2023-07-24T13:59:43Z

Ok, we're on same page then.

For my purposes, I don't need tags associated with locally-created releases to be preserved in the mirror.
I would like to have releases created in the mirror be preserved.
If, as @mrexodia suggests above it is already the case that a release on a non-mirror repo will survive if its associated tag is deleted, then I'd like that same behavior on a mirror.

mrexodia · 2023-07-28T07:39:11Z

Yes, that's why I think we need to disable creating releases in a mirror repository.

In my case this forces me to use a non-mirror repository and manually mirror + recreate releases. My use case is hosting a mirror with release binaries in another region. The GitHub repository I mirror from is the main development place. Currently I’m forced to use a non-mirror already so it’s fine, but in general disabling releases altogether prevents my use case of a mirror completely…

mrexodia · 2023-07-28T07:39:56Z

And again, I’m happy to try to contribute to the code. I just need some pointers where to start.

lunny · 2023-07-28T15:37:28Z

Because #20311 will support mirror releases from upstream. So I think release creation on mirror repositories must be disabled. Otherwise, it will result in possible conflicts.

lkubb · 2023-08-15T10:05:27Z

I think there are two separate use cases for mirrored repositories:

Migrating a project from GitHub etc.
Keeping a local mirror of code dependencies (does not protect from force-pushed deletions ofc).

Mine is the latter. In addition, I'm building my own signed releases from those mirrors. Because of this bug, I have to manually sync a fork instead of having automatic releases.

Two proposals regarding #20311:

When release syncing is available, allow to deselect it to be able to have local releases. When activating it later, warn that all local releases will be deleted if there are any.
Or introduce another repo/migration type that specifically addresses (2) (code mirroring only) and allow local releases there without being able to mirror remote releases/issues etc.

Not sure about the difficulty of those proposals, just throwing it out there. I really appreciate your hard work on Gitea!

dnviti · 2023-08-30T12:52:26Z

This is clearly a bug.
If i mirror a repository it disables all the other options (such as releases) so i understand that it ignores them.
I needed to attach new releases in a public read-only organization and every time it syncs any release uploaded on the mirrored repo will be deleted.

lunny · 2023-09-06T06:03:54Z

Or maybe we can convert a release as draft automatically if the related tag has been removed when mirroring.

xpufx · 2023-11-03T18:32:12Z

I am not sure my issue is the same since most people seem to be talking about mirroring from github initially and/or creating the releases on the gitea side.

My use case is a private gitea repo that mirrors to github. The commits sync fine to the remote github repo. I want to take advantage of the Actions on Github so I create the release there - manually for now. Everything works fine until I push another commit to my private gitea instance which then syncs to Github. At this point the releases disappear because they've been converted to drafts. Is this the same issue everyone is having?

Is it happening because the gitea side does not have the tag I have to create on the github side? How should I proceed if I want to keep the same workflow?

EDIT: I just tested this. If I create a tag on the gitea side first, it gets mirrored to github and the release does not get deleted.

Jinnrry · 2024-01-16T08:17:07Z

Any results? This bug has been around for a year now. I want to use Gitea to backup Github data, including binaries from releases, but this bug makes it impossible to backup releases at all.

lunny · 2024-01-16T08:22:55Z

This bug was introduced by #19125.
Ref: https://github.com/go-gitea/gitea/pull/19125/files#diff-2aa04998a791c30e5a02b49a97c07fcd93d50e8b31640ce2ddb1afeebf605d02R481

I think we need a method to not delete all previous releases but also keep better performance.

The bottleneck is upset is very slow and the batch insert is not.
Or maybe we can load all tag names into memory first.

Jinnrry · 2024-01-16T08:56:05Z

Also, although the Release list is lost, the uploaded attachment files are not deleted. And every time you upload again, a copy of the attachment will be saved again, and no de-duplication will be done. This had caused me to store 500GB+ of invalid data on my disk. Is there any way I can delete this invalid data now?

Jinnrry · 2024-01-16T09:15:42Z

This bug was introduced by #19125. Ref: https://github.com/go-gitea/gitea/pull/19125/files#diff-2aa04998a791c30e5a02b49a97c07fcd93d50e8b31640ce2ddb1afeebf605d02R481

I think we need a method to not delete all previous releases but also keep better performance.

The bottleneck is upset is very slow and the batch insert is not. Or maybe we can load all tag names into memory first.

I'm not familiar with Gitea code. I've looked over the logic of this code and maybe changed this here to if _, err := db.DeleteByBean(ctx, &repo_model.Release{RepoID: repo.ID, IsTag: true}); err ! = nil { and that fixes the bug? I think the Tag data is just expected to be deleted here, right?

lunny · 2024-01-16T09:17:13Z

You can fix them with ./gitea doctor storages commands.

Yes, I think you are right. That's why I added this to 1.21.5.

lunny · 2024-01-16T14:32:06Z

I sent #28817. It will not remove mirror releases when mirroring.

Fix #22066 # Purpose This PR fix the releases will be deleted when mirror repository sync the tags. # The problem In the previous implementation of #19125. All releases record in databases of one mirror repository will be deleted before sync. Ref: https://github.com/go-gitea/gitea/pull/19125/files#diff-2aa04998a791c30e5a02b49a97c07fcd93d50e8b31640ce2ddb1afeebf605d02R481 # The Pros This PR introduced a new method which will load all releases from databases and all tags on git data into memory. And detect which tags needs to be inserted, which tags need to be updated or deleted. Only tags releases(IsTag=true) which are not included in git data will be deleted, only tags which sha1 changed will be updated. So it will not delete any real releases include drafts. # The Cons The drawback is the memory usage will be higher than before if there are many tags on this repository. This PR defined a special release struct to reduce columns loaded from database to memory.

…a#28817) Fix go-gitea#22066 # Purpose This PR fix the releases will be deleted when mirror repository sync the tags. # The problem In the previous implementation of go-gitea#19125. All releases record in databases of one mirror repository will be deleted before sync. Ref: https://github.com/go-gitea/gitea/pull/19125/files#diff-2aa04998a791c30e5a02b49a97c07fcd93d50e8b31640ce2ddb1afeebf605d02R481 # The Pros This PR introduced a new method which will load all releases from databases and all tags on git data into memory. And detect which tags needs to be inserted, which tags need to be updated or deleted. Only tags releases(IsTag=true) which are not included in git data will be deleted, only tags which sha1 changed will be updated. So it will not delete any real releases include drafts. # The Cons The drawback is the memory usage will be higher than before if there are many tags on this repository. This PR defined a special release struct to reduce columns loaded from database to memory.

@lunny

…#28939) Backport #28817 by @lunny Fix #22066 # Purpose This PR fix the releases will be deleted when mirror repository sync the tags. # The problem In the previous implementation of #19125. All releases record in databases of one mirror repository will be deleted before sync. Ref: https://github.com/go-gitea/gitea/pull/19125/files#diff-2aa04998a791c30e5a02b49a97c07fcd93d50e8b31640ce2ddb1afeebf605d02R481 # The Pros This PR introduced a new method which will load all releases from databases and all tags on git data into memory. And detect which tags needs to be inserted, which tags need to be updated or deleted. Only tags releases(IsTag=true) which are not included in git data will be deleted, only tags which sha1 changed will be updated. So it will not delete any real releases include drafts. # The Cons The drawback is the memory usage will be higher than before if there are many tags on this repository. This PR defined a special release struct to reduce columns loaded from database to memory. --------- Co-authored-by: Lunny Xiao <[email protected]>

…a#28817) Fix go-gitea#22066 # Purpose This PR fix the releases will be deleted when mirror repository sync the tags. # The problem In the previous implementation of go-gitea#19125. All releases record in databases of one mirror repository will be deleted before sync. Ref: https://github.com/go-gitea/gitea/pull/19125/files#diff-2aa04998a791c30e5a02b49a97c07fcd93d50e8b31640ce2ddb1afeebf605d02R481 # The Pros This PR introduced a new method which will load all releases from databases and all tags on git data into memory. And detect which tags needs to be inserted, which tags need to be updated or deleted. Only tags releases(IsTag=true) which are not included in git data will be deleted, only tags which sha1 changed will be updated. So it will not delete any real releases include drafts. # The Cons The drawback is the memory usage will be higher than before if there are many tags on this repository. This PR defined a special release struct to reduce columns loaded from database to memory.

github-actions · 2024-02-29T00:02:35Z

Automatically locked because of our CONTRIBUTING guidelines

kotovalexarian added the type/bug label Dec 7, 2022

lunny mentioned this issue Dec 9, 2022

Pull mirror repository will delete all release info when sync #22076

Closed

dboreham mentioned this issue Apr 12, 2023

Tracking issue for Gitea CI bugs/limitations cerc-io/hosting#25

Closed

7 tasks

lng2020 mentioned this issue Nov 7, 2023

The release is lost after mirror synchronization #27936

Closed

lunny added this to the 1.21.5 milestone Jan 16, 2024

lunny mentioned this issue Jan 16, 2024

Don't remove all mirror repository's releases when mirroring #28817

Merged

lunny closed this as completed in #28817 Jan 26, 2024

GiteaBot mentioned this issue Jan 26, 2024

Don't remove all mirror repository's releases when mirroring (#28817) #28939

Merged

github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Releases are deleted when mirror repository is synchronized #22066

Releases are deleted when mirror repository is synchronized #22066

kotovalexarian commented Dec 7, 2022 •

edited

Loading

lunny commented Dec 8, 2022

kotovalexarian commented Dec 9, 2022

iarp commented Dec 15, 2022

dboreham commented Apr 12, 2023

mrexodia commented Jul 19, 2023

lunny commented Jul 19, 2023

dboreham commented Jul 19, 2023

lunny commented Jul 19, 2023

dboreham commented Jul 19, 2023

lunny commented Jul 19, 2023

mrexodia commented Jul 19, 2023

dboreham commented Jul 19, 2023

lunny commented Jul 24, 2023

dboreham commented Jul 24, 2023

mrexodia commented Jul 28, 2023

mrexodia commented Jul 28, 2023

lunny commented Jul 28, 2023

lkubb commented Aug 15, 2023

dnviti commented Aug 30, 2023

lunny commented Sep 6, 2023

xpufx commented Nov 3, 2023 •

edited

Loading

Jinnrry commented Jan 16, 2024

lunny commented Jan 16, 2024 •

edited

Loading

Jinnrry commented Jan 16, 2024

Jinnrry commented Jan 16, 2024

lunny commented Jan 16, 2024 •

edited

Loading

lunny commented Jan 16, 2024

github-actions bot commented Feb 29, 2024

Releases are deleted when mirror repository is synchronized #22066

Releases are deleted when mirror repository is synchronized #22066

Comments

kotovalexarian commented Dec 7, 2022 • edited Loading

Description

Steps

Expected

Got

Gitea Version

Can you reproduce the bug on the Gitea demo site?

Log Gist

Screenshots

Git Version

Operating System

How are you running Gitea?

Database

lunny commented Dec 8, 2022

kotovalexarian commented Dec 9, 2022

iarp commented Dec 15, 2022

dboreham commented Apr 12, 2023

mrexodia commented Jul 19, 2023

lunny commented Jul 19, 2023

dboreham commented Jul 19, 2023

lunny commented Jul 19, 2023

dboreham commented Jul 19, 2023

lunny commented Jul 19, 2023

mrexodia commented Jul 19, 2023

dboreham commented Jul 19, 2023

lunny commented Jul 24, 2023

dboreham commented Jul 24, 2023

mrexodia commented Jul 28, 2023

mrexodia commented Jul 28, 2023

lunny commented Jul 28, 2023

lkubb commented Aug 15, 2023

dnviti commented Aug 30, 2023

lunny commented Sep 6, 2023

xpufx commented Nov 3, 2023 • edited Loading

Jinnrry commented Jan 16, 2024

lunny commented Jan 16, 2024 • edited Loading

Jinnrry commented Jan 16, 2024

Jinnrry commented Jan 16, 2024

lunny commented Jan 16, 2024 • edited Loading

lunny commented Jan 16, 2024

github-actions bot commented Feb 29, 2024

kotovalexarian commented Dec 7, 2022 •

edited

Loading

xpufx commented Nov 3, 2023 •

edited

Loading

lunny commented Jan 16, 2024 •

edited

Loading

lunny commented Jan 16, 2024 •

edited

Loading