Mirror issues along with the repo

[IzzySoft]

• Gitea version (or commit ref): 1.1.1
• Git version: 2.1.4
• Operating system: Bananian (based on Debian Jessie)
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant (not a bug)
• Log gist:

Description

I've started mirroring my repos hosted on Github. I was glad the Wikis came along, but missed the issues. Would it be possible to include them with the mirror, as well as any PRs? Not speaking of a two-way mirror here (I wouldn't edit/add issues in my mirror, and neither approve PRs), "read-only" would be fine with me.

[bkcsoft]

This has been discussed in length previously. Issue/PRs are implementation specific and Gitea does not know OR care what you're mirroring from...

[IzzySoft]

I saw that issue on PRs. Didn't draw the conclusion to "regular issues" then. Now that you mention it: it indeed seems like GH treats PRs as "a special form of issue", indeed. Not sure why I didn't consider that.

Still, especially based on Githubs current behavior towards FOSS projects (see e.g. this Heise article), I know of several FOSS projects wanting to "migrate away" or at least have a "local mirror" to backup their "stuff". While repo (code) and wiki are dealt fine with by Gitea, Issues (and their history) would be lost. Of course, same for PRs and releases, as it stands.

I understand your hesitation in this context, so I won't argue that point (made clear in the other issue on PRs, IIRC). Let's check another approach: Say I already had a script to "copy" my project's issues from (wherever). Would there be a way to feed them into Gitea, with Gitea taking care of "duplicates"? What I had in mind was: Github API AFAIK offers to "pull issues". So one could use a script to facilitate that, and then feed it to an API offered by Gitea. Which means: just one interface to maintain for the Gitea team, and that interface wouldn't need to care where the input came from. Users could create a script to wrap around that, and fire it via its own Cron job.

[bkcsoft]

This would work better if written as a separate service that listened on GH-wehbooks and passed data along to Gitea.

Reasons:

1. Less code in Gitea makes for less bugs.
2. This is a "niche" thing that most users don't need or ever use.
3. (lastly) Consider how many issue-trackers there are out there, do we expect Gitea to support them all? The code would be "huge". And what if someone suddenly changes it's API (like GitLab did in 9.0 going from v3 to v4, though still maintaining v3 for a while longer), then Gitea would have to be updated, and this would have to be backported since not everyone wants to upgrade their instance (with the downtime and bugs that comes with that)

All in all, I'd rather see this as a separate service. I know that many people (myself included) like the "single binary" thinking, but please don't turn Gitea into yet another unmaintainable monolith for the sake of single binary. 🙂

[IzzySoft]

Agreed, all good reasons (and I remember 3. from some other issue, most likely the PR discussion). It might work with a webhook, or with an external script called by the mirror process (to be more flexible: if you speak of "how many issue trackers", not all of them might have webhooks). For GH, there's an API which can be used by that "separate service".

But still, we'd need an interface on Giteas end to talk to and feed it the issues/PRs. With that available, folks with the appropriate knowledge could write the service-specific scripts, and maybe contribute them to Gitea. Not to be included into the "single binary", but to be downloaded separately (if there's a place for such "external tools" in the repo).

[bkcsoft]

That is available, both webhooks and issue/PR API, aaand they are GH compatible 😉

[IzzySoft]

Oh! Any pointers where to find it? Must have missed it on my search. Checked the entire website, and also tried finding things here in the repo, but didn't succeed.

[bkcsoft]

Unfortunately the only documentation we have currently is reading https://github.com/go-gitea/go-sdk 😞 But we are creating real documentation using Swagger https://try.gitea.io/swagger

[lunny]

Issues migration from Github to Gitea is needed by Gitea hosted Gitea.

[bkcsoft]

@lunny We don't need it in Gitea. As I've stated above, it should be a separate application (either one-off or service).

[IzzySoft]

@bkcsoft in the light of TeaHub, it should be available from within Gitea. Whether it's built-in or comes as a plugin is a different question.

Important from the user perspective: it shouldn't require a GO environment. If I write PHP code and want to mirror my repos from {put_hosting_service_here}, I'm not necessarily able to deal with GO code nor do I want to (or have the time to) make myself familiar with it (I'm not, for example).

So this should be easily accessible, and support at least Gitea/Gogs, Github and GitLab as "sources".

[techknowlogick]

@IzzySoft perhaps Teahub could create a web interface for https://git.jonasfranz.software/JonasFranzDEV/gitea-github-migrator and provide it as a service to Teahub users. @JonasFranzDEV already has an issue open for a web interface.

[IzzySoft]

@techknowlogick that might solve it for TeaHub, agreed. But what for the mirrors on my own machine at home? The migrator from Jonas would need a GO environment, which I'd not be willing to cope with (time wise and resource wise, as my Gitea runs on a Pi). If i'd be a simple plug-in which I'd just need to copy into some directory (e.g. inside the Gitea-created tree) and it would be picked up, fine with me. I don't insist it being in "the core". But it should be easy on non-GO-devs.

[techknowlogick]

@IzzySoft Jonas provides compiled releases for many different architectures (incl. rpi), which means you don't need a go environment.

[IzzySoft]

@techknowlogick and that works for (permanent) mirroring (as this issue asks for), or only for initial import of a project? Because the readme just mentions the latter. Further: "This migration tool does not work with Gitea instances using a SQLite database." Which is what's often used for smaller environments (like mine, on a Pi). Also read the other 2 problems (which are no deal-breakers but nasty, though I'm not sure whether they could be avoided at all).

[jonasfranz]

@IzzySoft Currently the tool only supports the initial import. If you want to have a mirroing feature at it, please open up an issue at https://git.jonasfranz.software/JonasFranzDEV/gitea-github-migrator . You could try to use the tool with a SQLite database it won't refuse to work. But it might happen that not every issue get's migrated due to database limitations.

The other two problems are getting solved currently since I've started a PR which adds a migration API to gitea which could allow it to migrate issues in the name of other users.

[IzzySoft]

@JonasFranzDEV I would, but login (at least with Github) fails.

[IzzySoft]

Yes, the two are closely related, @lunny 😄

[lunny]

@IzzySoft but I think it's not duplicated now.

[IzzySoft]

@lunny no, definitely not. Though I wouldn't wonder if solving one would solve the other as a "side effect", I wouldn't count on that.

[lunny]

This is some different from #479

[GlassedSilver]

Wouldn't mind throwing some bounty at this if someone else wants to join. If there isn't interest anymore, I'd wait with parking my money at bountysource, because I very much think that this should be in Gitea instead of a hacky external service that wouldn't nearly get the attention nor servicing. (not a big plug-in fanin general)

[techknowlogick]

While we can migrate issues & PRs right now, mirroring them would require a tasks that checks all content in the remote repo to see if anything has changed (edits, reactions, new issues, deletions, comments, etc...). And that already eats the ratelimit to even migrate for the Gitea repo itself, if we were to do this it'd use all the API quota and then some.

[GlassedSilver]

While we can migrate issues & PRs right now, mirroring them would require a tasks that checks all content in the remote repo to see if anything has changed (edits, reactions, new issues, deletions, comments, etc...). And that already eats the ratelimit to even migrate for the Gitea repo itself, if we were to do this it'd use all the API quota and then some.

So maybe not mirroring, but how about scheduled partial checks, prioritizing possible new issues?
Reason is that it could still function as backup method.