-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Design Discussion: ActivityPub support + ForgeFed vocabulary #14186
Comments
I also opened a discussion on Forgefed where Bill helpfully pointed out I did not reference #9045. That issue is locked to collaborators so I am not able to post there, but I think a lot of the technical discussion over there is relevant to this issue, so please check it out. Wishing everyone a happy New Year and guten Rutsch ins neue Jahr! |
Posted on behalf of Loic, as explained in the post scriptum (original message on the Fedeproxy forum). Bonjour, TL;DR: what would it take to put federation on the Gitea 1.16.x roadmap? The gitea roadmap for 1.1.15 is now locked and the next release cycle will start right after it is released. However the roadmap for 1.1.16 does not include the implementation of ActivityPub. What would it take to make that happen? It is of course a matter of who is interested to implement it and has time and skills to actually do it. An area where the fedeproxy project could help by redirecting 5,000€ of its funds (total 75K€). This is not much but it's not nothing and could be used to make the first baby step(s). It is also a matter of defining the first baby steps that would lead Gitea in the direction of federation. At the moment there only is a very high level discussion that can hardly be translated into actionable items. In your expert opinion, what would be the first minimal tasks that would make most sense? Cheers P.S: I'm very motivated to help move forward and in case you're curious why I don't post myself (thank you Pierre-Louis for being my proxy :-) ), feel free to read why I deleted my GItHub account a few years ago. |
I should note that Loic had reached out to me (and likely a few other forge maintainers) about this in the past via email, and I apologize for not responding. To quote Jeff Goldblum, life uhh.. gets in the way (I know this isn't the actual quote)
Technically 1.16.x work has already started, but that's due to the feature freeze with 1.15.x. Although the sooner a PR is created the more likely it is to get into the 1.16.x milestone (see note about creating the PR below).
To be frank, it's a matter of finding a developer to work on this (be it a maintainer of Gitea, or someone from outside the project). As you do have funding, I suspect that might be easier, as in the past few months over 1K USD has been collected as bounties working on various PRs (I personally have put up some funds as bounties). So there are developers who would likely be willing (and capable) to take on this work. If you were to use these funds I would recommend giving them directly to the developer working on this PR, and the maintainers reviewing it, rather than to the project itself (although of course speaking on behalf of the project we'd be happy/thankful to accept funds, but donations to the project don't necessarily guarantee that specific tasks would be completed). A note: as this is likely a large PR, I recommend the developer who works on it first discuss the approach of integrating this into gitea with the project (this is to prevent a large PR from being completed and perhaps some foundational work of the PR would be suggested to be done differently, so the developer would then potentially need to change a significant amount of it). cc: @KN4CK3R and @adelowo in case you'd be interested in paid work on Gitea |
FYI. On fediverse this recent development was enthusiastically shared and on Lemmy (a federated Reddit) a core dev said they'd move issue tracking from Github with this in place, while someone else said 'where can I donate?'. This last bit is not really clear, plus there may be additional opportunities to get more funding to realize this functionality. And also ForgeFed may still have funds available. |
Posted on behalf of Loic. @techknowlogick thanks for your reply and guidance :-) It's good to know the timing is right and I'm hopeful someone will be interested.
I'll follow your advice. Since fedeproxy is horizontal (no organization) the funds originate from individuals (Pierre-Louis and myself, 50% each) and it will be possible to pay the person(s) doing the work directly.
Maybe it can be broken down in smaller tasks / PRs? It would be easier to review and implement. And it will also be easier to prioritize which task should be worked on first and which ones can fit is the modest budget there is. |
There was a small meeting today between zeripath, myself, and Loic around building towards a future where federation work has a grant fund people to do it (to be clear: not necessarily funding us three, it could be funding one or two of us + others in the community). On the socialhub was the agenda. In the interest of transparency, the meeting was recorded which should also shortly be available there as well. Since the scope of work is fairly large and the goals rather ambitious, we definitely want to be as transparent as possible w/ the wider Gitea community and welcome folks to feel free to step up & participate if you, dear reader, would like. |
If somebody want to start the work from v1.16, it's a good start point to add comment on #16429 . |
Posted on behalf of Loic Bonjour, While working on the grant application today, I ran into a question that is probably worth discussing before moving forward. Go-fed is AGPLv3+, and Gitea is MIT. Adding Go-fed as a dependency of Gitea means that Gitea, as a whole (meaning Gitea+Go-fed+other dependencies), can only be released under the AGPLv3+. Or not released at all. To be more precise, here is the minimal set of actions required to distribute a gitea executable that contains Go-fed:
This is not the only possible course of action, only the simplest. What do people think about this? |
Damn that makes integrating go-fed not possible. |
Slight correction: The implementation of APCore is what is licensed AGPL, which is itself an extension of the standalone Go-Fed Activity libraries, which are licensed under the BSD 3-clause license instead. So what we would have to do would be to reimplement at least the relevant parts of APCore. |
I think there's no real issue here, as apcore is meant as the opinionated batteries-included server framework you'd use when creating a stand-alone AP server and save you a bunch of work. Since this issue was created, a new AP server project GoToSocial was created with Go-Fed and they chose to build on top of Note that both @cjslep and @tsmethurst (of GoToSocial) can be found in the Go-Fed Matrix chatroom at https://matrix.to/#/#go-fed:feneas.org |
Posted on behalf of Loic Oh… my bad, thanks for the clarification! |
I want to confirm what @aschrijver said. The I think Gitea's infrastructure is sufficiently different from APCore that there's nothing really lost by not using it. Since they have different technical approaches, I think Gitea's integration with |
In the last few weeks I looked throught the initial comment in this issue and tried to create a plan outlining what the implementation of ForgeFed could look like. The idea was to aid the discussion of the actual code that needs to be written, and not to have a 100% correct roadmap. I hope my proposals are at least somewhat sensible, as I do not have too much experience with either AP/ForgeFed or the Gitea codebase. Mapping ForgeFed concepts to GiteaBy comparing the ForgeFed Actor guidance to Giteas modules, I came to the conclusion that the following mapping should be reasonable:
Preparing the data modelAll modules mentioned in the table above should have the following fields added:
An instance-wide list of federated/blocked instances needs to be added to the database aswell. The "translation layer" mentioned by @cjslep should be rather straightforward, as the forgefed spec is minimal and gitea has most of the fields already. Some things like "Description" -> "summary" need to be kept in mind though. Implementing ActivityPub behaviourFrom this point onward there are two possible routes: Implementing it all by handWhen not using a library to abstract the ActivityPub functions, this would be my proposed way of adding federated behaviour to Gitea:
The following is an example of a repos json representation: {
"@context": [
"https://www.w3.org/ns/activitystreams",
"https://w3id.org/security/v1",
"https://forgefed.peers.community/ns"
],
"id": "https://dev.example/aviva/treesim",
"type": "Repository",
"publicKey": {
"id": "https://dev.example/aviva/treesim#main-key",
"owner": "https://dev.example/aviva/treesim",
"publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhki....."
},
"inbox": "https://dev.example/aviva/treesim/inbox",
"outbox": "https://dev.example/aviva/treesim/outbox",
"followers": "https://dev.example/aviva/treesim/followers",
"team": "https://dev.example/aviva/treesim/team",
"name": "Tree Growth 3D Simulation",
"summary": "<p>Tree growth 3D simulator for my nature exploration game</p>"
} Using Go-FedThe equivalent of writing an actor interface would be implementing the db interface from go-fed. The other two points would then be met by implementing the FederatingProtocol for all actor classes (http signing is possible using go-fed/httpsig). Fetching ActivityStreamsIn whatever way it is implemented, at this point Gitea would be able to send and receive Activities as they happen, but it would be nice to also see all the Actors and their activities that came before. To do this, Gitea would have to do the following when a user requests Data about an Actor:
It might be a good idea to add a UI element to force a fetch of remote data, so users can get up-to-date information when needed. |
As an aside, from what I read, it seems like SharedInbox would help in generating a timeline of all public activities. To me this seems like a desirable feature for a ForgeFed implementation. My knowledge of AP is not deep enough to have a really meaningful opinion on the topic though and I would love to find out more! |
Thanks for your elaboration. It is great to see the growing interest in federating Gitea!
The most appropriate location would be the SocialHub community forum, where there's some prior discussion already, like this topic. |
Posted on behalf of Loic Loic wrote:
@zeripath @cjslep here is an idea for the first baby step: creating a keypair for every user that will be used to sign http requests (see also the IETF draft). Although signing http requests is not required by ActivityPub or ActivityStreams, it is mentioned in the ActivityPub wikii. Since mastodon, bookwyrm and other ActivityPub implementations verify and expect a signature, it is de facto required. If it sounds sensible to you, I think the creation of the corresponding issue as well as its implementation is eligible to be funded by the 5,000€ grant made available by the fedeproxy project a month ago:
|
Any news on this? So exited to have this when it is done! |
@thebiblelover7 please see the pinned issue #16827 |
…-ap (#19133) * go.mod: add go-fed/{httpsig,activity/pub,activity/streams} dependency go get github.com/go-fed/activity/streams@master go get github.com/go-fed/activity/pub@master go get github.com/go-fed/httpsig@master * activitypub: implement /api/v1/activitypub/user/{username} (#14186) Return informations regarding a Person (as defined in ActivityStreams https://www.w3.org/TR/activitystreams-vocabulary/#dfn-person). Refs: #14186 Signed-off-by: Loïc Dachary <[email protected]> * activitypub: add the public key to Person (#14186) Refs: #14186 Signed-off-by: Loïc Dachary <[email protected]> * activitypub: go-fed conformant Clock instance Signed-off-by: Loïc Dachary <[email protected]> * activitypub: signing http client Signed-off-by: Loïc Dachary <[email protected]> * activitypub: implement the ReqSignature middleware Signed-off-by: Loïc Dachary <[email protected]> * activitypub: hack_16834 Signed-off-by: Loïc Dachary <[email protected]> * Fix CI checks-backend errors with go mod tidy Signed-off-by: Anthony Wang <[email protected]> * Change 2021 to 2022, properly format package imports Signed-off-by: Anthony Wang <[email protected]> * Run make fmt and make generate-swagger Signed-off-by: Anthony Wang <[email protected]> * Use Gitea JSON library, add assert for pkp Signed-off-by: Anthony Wang <[email protected]> * Run make fmt again, fix err var redeclaration Signed-off-by: Anthony Wang <[email protected]> * Remove LogSQL from ActivityPub person test Signed-off-by: Anthony Wang <[email protected]> * Assert if json.Unmarshal succeeds Signed-off-by: Anthony Wang <[email protected]> * Cleanup, handle invalid usernames for ActivityPub person GET request Signed-off-by: Anthony Wang <[email protected]> * Rename hack_16834 to user_settings Signed-off-by: Anthony Wang <[email protected]> * Use the httplib module instead of http for GET requests * Clean up whitespace with make fmt * Use time.RFC1123 and make the http.Client proxy-aware * Check if digest algo is supported in setting module * Clean up some variable declarations * Remove unneeded copy * Use system timezone instead of setting.DefaultUILocation * Use named constant for httpsigExpirationTime * Make pubKey IRI #main-key instead of /#main-key * Move /#main-key to #main-key in tests * Implemented Webfinger endpoint. * Add visible check. * Add user profile as alias. * Add actor IRI and remote interaction URL to WebFinger response * fmt * Fix lint errors * Use go-ap instead of go-fed * Run go mod tidy to fix missing modules in go.mod and go.sum * make fmt * Convert remaining code to go-ap * Clean up go.sum * Fix JSON unmarshall error * Fix CI errors by adding @context to Person() and making sure types match * Correctly decode JSON in api_activitypub_person_test.go * Force CI rerun * Fix TestActivityPubPersonInbox segfault * Fix lint error * Use @mariusor's suggestions for idiomatic go-ap usage * Correctly add inbox/outbox IRIs to person * Code cleanup * Remove another LogSQL from ActivityPub person test * Move httpsig algos slice to an init() function * Add actor IRI and remote interaction URL to WebFinger response * Update TestWebFinger to check for ActivityPub IRI in aliases * make fmt * Force CI rerun * WebFinger: Add CORS header and fix Href -> Template for remote interactions The CORS header is needed due to https://datatracker.ietf.org/doc/html/rfc7033#section-5 and fixes some Peertube <-> Gitea federation issues * make lint-backend * Make sure Person endpoint has Content-Type application/activity+json and includes PreferredUsername, URL, and Icon Setting the correct Content-Type is essential for federating with Mastodon * Use UTC instead of GMT * Rename pkey to pubKey * Make sure HTTP request Date in GMT * make fmt * dont drop err * Make sure API responses always refer to username in original case Copied from what I wrote on #19133 discussion: Handling username case is a very tricky issue and I've already encountered a Mastodon <-> Gitea federation bug due to Gitea considering Ta180m and ta180m to be the same user while Mastodon thinks they are two different users. I think the best way forward is for Gitea to only use the original case version of the username for federation so other AP software don't get confused. * Move httpsig algs constant slice to modules/setting/federation.go * Add new federation settings to app.example.ini and config-cheat-sheet * Return if marshalling error * Make sure Person IRIs are generated correctly This commit ensures that if the setting.AppURL is something like "http://127.0.0.1:42567" (like in the integration tests), a trailing slash will be added after that URL. * If httpsig verification fails, fix Host header and try again This fixes a very rare bug when Gitea and another AP server (confirmed to happen with Mastodon) are running on the same machine, Gitea fails to verify incoming HTTP signatures. This is because the other AP server creates the sig with the public Gitea domain as the Host. However, when Gitea receives the request, the Host header is instead localhost, so the signature verification fails. Manually changing the host header to the correct value and trying the veification again fixes the bug. * Revert "If httpsig verification fails, fix Host header and try again" This reverts commit f53e46c. The bug was actually caused by nginx messing up the Host header when reverse-proxying since I didn't have the line `proxy_set_header Host $host;` in my nginx config for Gitea. * Go back to using ap.IRI to generate inbox and outbox IRIs * use const for key values * Update routers/web/webfinger.go * Use ctx.JSON in Person response to make code cleaner * Revert "Use ctx.JSON in Person response to make code cleaner" This doesn't work because the ctx.JSON() function already sends the response out and it's too late to edit the headers. This reverts commit 95aad98. * Use activitypub.ActivityStreamsContentType for Person response Content Type * Limit maximum ActivityPub request and response sizes to a configurable setting * Move setting key constants to models/user/setting_keys.go * Fix failing ActivityPubPerson integration test by checking the correct field for username * Add a warning about changing settings that can break federation * Add better comments * Don't multiply Federation.MaxSize by 1<<20 twice * Add more better comments * Fix failing ActivityPubMissingPerson test We now use ctx.ContextUser so the message printed out when a user does not exist is slightly different * make generate-swagger For some reason I didn't realize that /templates/swagger/v1_json.tmpl was machine-generated by make generate-swagger... I've been editing it by hand for three months! 🤦 * Move getting the RFC 2616 time to a separate function * More code cleanup * Update go-ap to fix empty liked collection and removed unneeded HTTP headers * go mod tidy * Add ed25519 to httpsig algorithms * Use go-ap/jsonld to add @context and marshal JSON * Change Gitea user agent from the default to Gitea/Version * Use ctx.ServerError and remove all remote interaction code from webfinger.go
…-ap (go-gitea#19133) * go.mod: add go-fed/{httpsig,activity/pub,activity/streams} dependency go get github.com/go-fed/activity/streams@master go get github.com/go-fed/activity/pub@master go get github.com/go-fed/httpsig@master * activitypub: implement /api/v1/activitypub/user/{username} (go-gitea#14186) Return informations regarding a Person (as defined in ActivityStreams https://www.w3.org/TR/activitystreams-vocabulary/#dfn-person). Refs: go-gitea#14186 Signed-off-by: Loïc Dachary <[email protected]> * activitypub: add the public key to Person (go-gitea#14186) Refs: go-gitea#14186 Signed-off-by: Loïc Dachary <[email protected]> * activitypub: go-fed conformant Clock instance Signed-off-by: Loïc Dachary <[email protected]> * activitypub: signing http client Signed-off-by: Loïc Dachary <[email protected]> * activitypub: implement the ReqSignature middleware Signed-off-by: Loïc Dachary <[email protected]> * activitypub: hack_16834 Signed-off-by: Loïc Dachary <[email protected]> * Fix CI checks-backend errors with go mod tidy Signed-off-by: Anthony Wang <[email protected]> * Change 2021 to 2022, properly format package imports Signed-off-by: Anthony Wang <[email protected]> * Run make fmt and make generate-swagger Signed-off-by: Anthony Wang <[email protected]> * Use Gitea JSON library, add assert for pkp Signed-off-by: Anthony Wang <[email protected]> * Run make fmt again, fix err var redeclaration Signed-off-by: Anthony Wang <[email protected]> * Remove LogSQL from ActivityPub person test Signed-off-by: Anthony Wang <[email protected]> * Assert if json.Unmarshal succeeds Signed-off-by: Anthony Wang <[email protected]> * Cleanup, handle invalid usernames for ActivityPub person GET request Signed-off-by: Anthony Wang <[email protected]> * Rename hack_16834 to user_settings Signed-off-by: Anthony Wang <[email protected]> * Use the httplib module instead of http for GET requests * Clean up whitespace with make fmt * Use time.RFC1123 and make the http.Client proxy-aware * Check if digest algo is supported in setting module * Clean up some variable declarations * Remove unneeded copy * Use system timezone instead of setting.DefaultUILocation * Use named constant for httpsigExpirationTime * Make pubKey IRI #main-key instead of /#main-key * Move /#main-key to #main-key in tests * Implemented Webfinger endpoint. * Add visible check. * Add user profile as alias. * Add actor IRI and remote interaction URL to WebFinger response * fmt * Fix lint errors * Use go-ap instead of go-fed * Run go mod tidy to fix missing modules in go.mod and go.sum * make fmt * Convert remaining code to go-ap * Clean up go.sum * Fix JSON unmarshall error * Fix CI errors by adding @context to Person() and making sure types match * Correctly decode JSON in api_activitypub_person_test.go * Force CI rerun * Fix TestActivityPubPersonInbox segfault * Fix lint error * Use @mariusor's suggestions for idiomatic go-ap usage * Correctly add inbox/outbox IRIs to person * Code cleanup * Remove another LogSQL from ActivityPub person test * Move httpsig algos slice to an init() function * Add actor IRI and remote interaction URL to WebFinger response * Update TestWebFinger to check for ActivityPub IRI in aliases * make fmt * Force CI rerun * WebFinger: Add CORS header and fix Href -> Template for remote interactions The CORS header is needed due to https://datatracker.ietf.org/doc/html/rfc7033#section-5 and fixes some Peertube <-> Gitea federation issues * make lint-backend * Make sure Person endpoint has Content-Type application/activity+json and includes PreferredUsername, URL, and Icon Setting the correct Content-Type is essential for federating with Mastodon * Use UTC instead of GMT * Rename pkey to pubKey * Make sure HTTP request Date in GMT * make fmt * dont drop err * Make sure API responses always refer to username in original case Copied from what I wrote on go-gitea#19133 discussion: Handling username case is a very tricky issue and I've already encountered a Mastodon <-> Gitea federation bug due to Gitea considering Ta180m and ta180m to be the same user while Mastodon thinks they are two different users. I think the best way forward is for Gitea to only use the original case version of the username for federation so other AP software don't get confused. * Move httpsig algs constant slice to modules/setting/federation.go * Add new federation settings to app.example.ini and config-cheat-sheet * Return if marshalling error * Make sure Person IRIs are generated correctly This commit ensures that if the setting.AppURL is something like "http://127.0.0.1:42567" (like in the integration tests), a trailing slash will be added after that URL. * If httpsig verification fails, fix Host header and try again This fixes a very rare bug when Gitea and another AP server (confirmed to happen with Mastodon) are running on the same machine, Gitea fails to verify incoming HTTP signatures. This is because the other AP server creates the sig with the public Gitea domain as the Host. However, when Gitea receives the request, the Host header is instead localhost, so the signature verification fails. Manually changing the host header to the correct value and trying the veification again fixes the bug. * Revert "If httpsig verification fails, fix Host header and try again" This reverts commit f53e46c. The bug was actually caused by nginx messing up the Host header when reverse-proxying since I didn't have the line `proxy_set_header Host $host;` in my nginx config for Gitea. * Go back to using ap.IRI to generate inbox and outbox IRIs * use const for key values * Update routers/web/webfinger.go * Use ctx.JSON in Person response to make code cleaner * Revert "Use ctx.JSON in Person response to make code cleaner" This doesn't work because the ctx.JSON() function already sends the response out and it's too late to edit the headers. This reverts commit 95aad98. * Use activitypub.ActivityStreamsContentType for Person response Content Type * Limit maximum ActivityPub request and response sizes to a configurable setting * Move setting key constants to models/user/setting_keys.go * Fix failing ActivityPubPerson integration test by checking the correct field for username * Add a warning about changing settings that can break federation * Add better comments * Don't multiply Federation.MaxSize by 1<<20 twice * Add more better comments * Fix failing ActivityPubMissingPerson test We now use ctx.ContextUser so the message printed out when a user does not exist is slightly different * make generate-swagger For some reason I didn't realize that /templates/swagger/v1_json.tmpl was machine-generated by make generate-swagger... I've been editing it by hand for three months! 🤦 * Move getting the RFC 2616 time to a separate function * More code cleanup * Update go-ap to fix empty liked collection and removed unneeded HTTP headers * go mod tidy * Add ed25519 to httpsig algorithms * Use go-ap/jsonld to add @context and marshal JSON * Change Gitea user agent from the default to Gitea/Version * Use ctx.ServerError and remove all remote interaction code from webfinger.go
…-ap (go-gitea#19133) * go.mod: add go-fed/{httpsig,activity/pub,activity/streams} dependency go get github.com/go-fed/activity/streams@master go get github.com/go-fed/activity/pub@master go get github.com/go-fed/httpsig@master * activitypub: implement /api/v1/activitypub/user/{username} (go-gitea#14186) Return informations regarding a Person (as defined in ActivityStreams https://www.w3.org/TR/activitystreams-vocabulary/#dfn-person). Refs: go-gitea#14186 Signed-off-by: Loïc Dachary <[email protected]> * activitypub: add the public key to Person (go-gitea#14186) Refs: go-gitea#14186 Signed-off-by: Loïc Dachary <[email protected]> * activitypub: go-fed conformant Clock instance Signed-off-by: Loïc Dachary <[email protected]> * activitypub: signing http client Signed-off-by: Loïc Dachary <[email protected]> * activitypub: implement the ReqSignature middleware Signed-off-by: Loïc Dachary <[email protected]> * activitypub: hack_16834 Signed-off-by: Loïc Dachary <[email protected]> * Fix CI checks-backend errors with go mod tidy Signed-off-by: Anthony Wang <[email protected]> * Change 2021 to 2022, properly format package imports Signed-off-by: Anthony Wang <[email protected]> * Run make fmt and make generate-swagger Signed-off-by: Anthony Wang <[email protected]> * Use Gitea JSON library, add assert for pkp Signed-off-by: Anthony Wang <[email protected]> * Run make fmt again, fix err var redeclaration Signed-off-by: Anthony Wang <[email protected]> * Remove LogSQL from ActivityPub person test Signed-off-by: Anthony Wang <[email protected]> * Assert if json.Unmarshal succeeds Signed-off-by: Anthony Wang <[email protected]> * Cleanup, handle invalid usernames for ActivityPub person GET request Signed-off-by: Anthony Wang <[email protected]> * Rename hack_16834 to user_settings Signed-off-by: Anthony Wang <[email protected]> * Use the httplib module instead of http for GET requests * Clean up whitespace with make fmt * Use time.RFC1123 and make the http.Client proxy-aware * Check if digest algo is supported in setting module * Clean up some variable declarations * Remove unneeded copy * Use system timezone instead of setting.DefaultUILocation * Use named constant for httpsigExpirationTime * Make pubKey IRI #main-key instead of /#main-key * Move /#main-key to #main-key in tests * Implemented Webfinger endpoint. * Add visible check. * Add user profile as alias. * Add actor IRI and remote interaction URL to WebFinger response * fmt * Fix lint errors * Use go-ap instead of go-fed * Run go mod tidy to fix missing modules in go.mod and go.sum * make fmt * Convert remaining code to go-ap * Clean up go.sum * Fix JSON unmarshall error * Fix CI errors by adding @context to Person() and making sure types match * Correctly decode JSON in api_activitypub_person_test.go * Force CI rerun * Fix TestActivityPubPersonInbox segfault * Fix lint error * Use @mariusor's suggestions for idiomatic go-ap usage * Correctly add inbox/outbox IRIs to person * Code cleanup * Remove another LogSQL from ActivityPub person test * Move httpsig algos slice to an init() function * Add actor IRI and remote interaction URL to WebFinger response * Update TestWebFinger to check for ActivityPub IRI in aliases * make fmt * Force CI rerun * WebFinger: Add CORS header and fix Href -> Template for remote interactions The CORS header is needed due to https://datatracker.ietf.org/doc/html/rfc7033#section-5 and fixes some Peertube <-> Gitea federation issues * make lint-backend * Make sure Person endpoint has Content-Type application/activity+json and includes PreferredUsername, URL, and Icon Setting the correct Content-Type is essential for federating with Mastodon * Use UTC instead of GMT * Rename pkey to pubKey * Make sure HTTP request Date in GMT * make fmt * dont drop err * Make sure API responses always refer to username in original case Copied from what I wrote on go-gitea#19133 discussion: Handling username case is a very tricky issue and I've already encountered a Mastodon <-> Gitea federation bug due to Gitea considering Ta180m and ta180m to be the same user while Mastodon thinks they are two different users. I think the best way forward is for Gitea to only use the original case version of the username for federation so other AP software don't get confused. * Move httpsig algs constant slice to modules/setting/federation.go * Add new federation settings to app.example.ini and config-cheat-sheet * Return if marshalling error * Make sure Person IRIs are generated correctly This commit ensures that if the setting.AppURL is something like "http://127.0.0.1:42567" (like in the integration tests), a trailing slash will be added after that URL. * If httpsig verification fails, fix Host header and try again This fixes a very rare bug when Gitea and another AP server (confirmed to happen with Mastodon) are running on the same machine, Gitea fails to verify incoming HTTP signatures. This is because the other AP server creates the sig with the public Gitea domain as the Host. However, when Gitea receives the request, the Host header is instead localhost, so the signature verification fails. Manually changing the host header to the correct value and trying the veification again fixes the bug. * Revert "If httpsig verification fails, fix Host header and try again" This reverts commit f53e46c. The bug was actually caused by nginx messing up the Host header when reverse-proxying since I didn't have the line `proxy_set_header Host $host;` in my nginx config for Gitea. * Go back to using ap.IRI to generate inbox and outbox IRIs * use const for key values * Update routers/web/webfinger.go * Use ctx.JSON in Person response to make code cleaner * Revert "Use ctx.JSON in Person response to make code cleaner" This doesn't work because the ctx.JSON() function already sends the response out and it's too late to edit the headers. This reverts commit 95aad98. * Use activitypub.ActivityStreamsContentType for Person response Content Type * Limit maximum ActivityPub request and response sizes to a configurable setting * Move setting key constants to models/user/setting_keys.go * Fix failing ActivityPubPerson integration test by checking the correct field for username * Add a warning about changing settings that can break federation * Add better comments * Don't multiply Federation.MaxSize by 1<<20 twice * Add more better comments * Fix failing ActivityPubMissingPerson test We now use ctx.ContextUser so the message printed out when a user does not exist is slightly different * make generate-swagger For some reason I didn't realize that /templates/swagger/v1_json.tmpl was machine-generated by make generate-swagger... I've been editing it by hand for three months! 🤦 * Move getting the RFC 2616 time to a separate function * More code cleanup * Update go-ap to fix empty liked collection and removed unneeded HTTP headers * go mod tidy * Add ed25519 to httpsig algorithms * Use go-ap/jsonld to add @context and marshal JSON * Change Gitea user agent from the default to Gitea/Version * Use ctx.ServerError and remove all remote interaction code from webfinger.go
is this still active? |
I don't think any of the Gitea developers are actively working on federation, but there's been some progress in implementing it in Forgejo. I've personally been busy with other stuff for the past year or so, but I'm hoping to getting back to working on federation sometime in the next month. |
@Ta180m I'm still sending PRs following my original plan, but as this is a large undertaking it's being done in small parts over time as large PRs make things difficult to review. |
#1612 discusses Federation in general but I wanted to open an issue for the ActivityPub + ForgeFed solution specifically and concretize this unit of work. Let's keep to discussing ActivityPub+ForgeFed design specifics here, and have the question "whether it should be ActivityPub+ForgeFed at all" discussed in the other issue (#1612).
Background
I work on the go-fed suite of ActivityPub related libraries, sites, and tools. So my knowledge is more centered on the AP/ForgeFed angle, but as I've only spent light amount of time with the Gitea code I'm not comfortable making the changes required, especially w/o without serious design discussions. And I'm not here to shill go-fed as being the solution, it just provides a solution, as there are reasons to pick it entirely, partly, or not at all. I'll try to keep the evangelization to a minimum and self-contained, at the very end.
(Optional, Time-Sensitive) Grant Opportunity
There is a limited-time opportunity for this work to be submitted as a grant to the NLNet folks via the NGI Search & Discovery grant (5-50k euros) or DAPSI grants (50k+ euros?). There is never a guarantee a grant will be selected to receive money, but given the slate of other Fediverse projects that have gotten funding via the NGI S&D, I think this is a great exercise. There is the possibility that the EU will extend certain NGI funding periods for further cycles, but it is not guaranteed.
Concretely, would need 1 or more Gitea community member volunteers interested in taking the lead on those. I personally am applying separately for other projects, so I don't have time / energy to push this aspect forward, but happy to provide guidance where possible.
Community Standards
I believe the ForgeFed work is still ongoing. An outcome could be that this effort allows whoever wants to help to pioneer additional ForgeFed behavior and be a voice there. Additionally, depending how Gitea embraces ActivityPub in general, it may also have opportunities to create Fediverse Enhancement Proposals, so no matter what the volunteers will definitely get open-source community leadership opportunities.
Design
Overview
ActivityPub is based on the concept of actors exchanging activities. These activities tell federated peers how to update their view of the "Federated Web", which is a linked-data graph composed of different RDF ontologies. That's a jargon-y way of saying "data types are flexible, and have pointers to other pieces of data". ForgeFed is just one ontology focusing on Forge behaviors and entities. Peers are not expected to know how to interpret every single ontology on this Federated Web, so Gitea can just focus on a narrow one -- ForgeFed -- in addition to the basic ActivityStreams vocabulary that acts as a common language.
This does not prevent Gitea from adopting different ontologies later, if the project decided to support viewing/interacting with the other kinds of activities going on in the wider web. It is just not a requirement right now, in the spirit of keeping scope limited.
The ForgeFed spec outlines the actors and the data being exchanged. A subset of that data are activites which are shareable between actors. One actor can
Create
aTicket
(issue) and give it to a peer, who knows: "this data -- which happens to be an activity -- is aCreate
so I'll invoke my create behavior/function with the payload".So if REST is...
POST
to/issues/new
with the body containingpayload
and asession_id
containing authenticated credentials. This results in invoking the server'sCreateIssue
function withpayload
based on theuser
calling it....where REST scales by just creating more endpoints and using more HTTP verbs:
POST
andGET
andPATCH
to/repo
,/merge-request
,/repo
Then ActivityPub is...
POST
to/actors/cj/inbox
with anhttp_signature
header and Activitypayload
. This results in invoking the server'sWhatActivityIsThis
function which determines the Activity is aCreate
, so it calls theCreateIssue
function with the rest of thepayload
information based on thefederated_peer_user
calling it.... where ActivityPub scales by just having new types of Activities (
Create
,Update
,Offer
, etc) and new data types that are acted upon (Person
,Repository
,Commit
,Note
, etc).This means Gitea adopting ActivityPub will require a little bit of a different philosophical mindset than perhaps is common. Rectifying that, or isolating that, with the existing codebase is a core engineering challenge.
Therefore, at a high level, Gitea would need to support the following concepts:
...then can "First Federated Behavior" be reasoned about, as a penultimate section.
Let's concretely dive into what is required to do them. The "why"s might not come together until the "Fetching ActivityStreams" section. These sections are the things I can think off of the top of my head, it may be incomplete.
Note: This only goes into S2S federation and not C2S federation
Actors
ForgeFed only has suggested guidance for actors:
Person
Project
Repository
Group
/Organization
/Team
To support any of these, the following would need to be tackled:
/repository/actor/{id}
IRI. Note: Existing IRIs can be re-used, so long as they respect the wholeAccept
/Content-Type
headers for ActivityStreams content.inbox
andoutbox
ordered collections. This one is the big doozy. Each actor basically has a "sent" and "received" queue, which means:following
andfollowers
collections for actors. In addition to the previous concerns with theinbox
andoutbox
:Follow
andAccept
/Reject
flowliked
collection, if a "Team" or "Project" wants to star or favorite other items seen on the Fediverse. I'm unsure what this would look like in practice, but this could be an opportunity for someone wanting to play with UI/UX to do so.Sending Activities
Sending activities is described in the ActivityPub spec. Unfortunately, it also explicitly relies on the C2S addressing, which must be kept in mind while implementing.
This both unblocks Gitea's ability to do any federated flow in the future, but alone is insufficient to do any specific federated flow.
To keep it succinct:
outbox
to a concrete IRI, ex:/users/{id}/outbox
, from which it can serve the outbox collection.Create
activity") (C2S)sharedInbox
, alleviates the peer server if they're a massive instanceinbox
andPOST
ing to there. See later on Dereferencing.Receiving Activities
Receiving activities is the second half:
inbox
to a concrete IRI, ex:/users/{id}/inbox
, from which it can serve the inbox collection for aGET
request (if desired), or receive federated peerPOST
requests.federated_view
table of data that is just a cached version of the peer's data (but can always be authoritatively gotten from a peer, see "Fetching ActivityStreams below")sharedInbox
support. This is a way to optimize your Gitea instance if you have thousands of users on one instance and tons of activities flying about constantly, and don't want to DoS that server instance. I personally dislikesharedInbox
but won't go into those reasons here.Serving ActivityStreams
As a consequence of the aforementioned section of "having Actors do things", they will generate data that needs to have an ActivityStreams representation so that peers, upon looking at what an actor has been up to, can natively understand what is going on. For Gitea specifically, this gets into the ForgeFed examples.
/repo/{id}
. Note: as before, existing IRIs can be re-used, so long as they respect the wholeAccept
/Content-Type
headers for ActivityStreams content.http.Handler
. Again, "translation layer" is in quotes because there may be a better design choice.This unblocks the next bit...
Fetching ActivityStreams
If I ever use the term "dereferencing", this is what I mean.
A Gitea instance will be able to fetch a peer Gitea instance's ActivityStreams, thanks to the work outlined in the previous section. This allows you on
foo.gitea.io
to fetch myPerson
actor onbar.gitea.io
but, say, render it on a webpage to yourself natively onfoo.gitea.io
. Dereferencing is needed for other operations, in particular the Delivery and Addressing portions of "Sending an Activity".Fetching ActivityStreams data also allows
foo.gitea.io
to potentially display all the information of a Repository shown onbar.gitea.io
, without having to actually navigate tobar.gitea.io
. Any actions done by users would spawn new Activities and resulting in invoking the "Sending Activities" section. Again, that's up to the concrete design.http.Client
)Accept
header appropriatelyRepository
,Person
,Team
) in Gitea's UIFirst Federated Behavior
All of the above, and we haven't yet discussed the behaviors unlocked by ForgeFed yet. They list several:
I would propose just aiming for one initially. Even aiming for none of these, but doing the other sections above, is a large enough feat worth celebrating: Getting to the point where the followers flow works is a celebratory moment, if Gitea wants to have the concepts of "followers"/"following" (and I think it does?).
This first federated behavior would involve:
Whew, done. :)
Go-Fed
I promised to keep this at the end and self-contained. :) The
go-fed/activity
library focuses on being middleware. You implement several interfaces likeDatabase
which it will use. You then use the resulting library calls in ahttp.Handler
to deal with "Actors sending Activities" or "serve this ActivityStreams representation".Since it is middleware, it only solves some of the problems I listed above. Big picture, the main problems remaining unsolved by go-fed are the integration ones:
More specifically, going section-by-section and listing the bullets that are addressed by
go-fed
:json
is woefully insufficient. I didn't mention this aspect at all before, but I won't dive into this unless someone wants to do so. Highly caution against deserializing ActivityStreams on one's own,go-fed/activity/streams
is there to help to go from[]byte
toActivityStreamsCreate
. Scales across all sorts of RDF vocabularies, too.go-fed/activity/pub
sharedInbox
go-fed/httpsig
for HTTP Signatures signing and verifyingpayload
to your Activity-specific behavior (Create => createFn()
, etc).Create{Note}
will attempt to create the federatedNote
via the localDatabase
interface, to try to help make bootstrapping quicker.go-fed/activity/pub
provides an optional transport to dereference using HTTP signatures, but doesn't tell you how to use the resulting data nor does it have any idea what it is for.The
go-fed/activity
library does not solve:Finally, some downsides of
go-fed
:go-fed/activity/pub
, which comes with its own risks. WriteFreely usesgo-fed/activity/streams
.I hope this kicks off a productive discussion. Thanks for reading this far, if you made it. :)
The text was updated successfully, but these errors were encountered: