From 5dfd5a4b281f8cd5300f77e1e6246ac80584d763 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 24 Feb 2020 00:05:42 +0000 Subject: [PATCH 01/21] MSC2444: peeking over federation via /peek --- .../2444-peeking-over-federation-peek-api.md | 104 ++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 proposals/2444-peeking-over-federation-peek-api.md diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md new file mode 100644 index 00000000000..a371e6a8e88 --- /dev/null +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -0,0 +1,104 @@ +# Proposal for implementing peeking over federation (peek API) + +## Problem + +Currently you can't peek over federation, as it was never designed or +implemented due to time constraints when peeking was originally added to Matrix +in 2016. + +As well as stopping users from previewing rooms before joining, the fact that +servers can't participate in remote rooms without joining them first is +inconvenient in many other ways: + + * You can't reliably participate in E2E encryption in rooms you're invited to + unless the server is actually participating in the room + (https://github.com/vector-im/riot-web/issues/2713) + * You can't use rooms as generic pubsub mechanisms for synchronising data like + profiles, groups, device-lists etc if you can't peek into them remotely. + * Search engines can't work if they can't peek remote rooms. + +## Solution + +We let servers participate in peekable rooms (i.e. those with `world_readable` +`m.room.history_visibility`) without having actually joined them. + +We do this by subscribing to a room on one or more servers via a new `/peek` +S2S API, which lets users on a given server declare their interest in viewing +events in that room. Having started peeking into the room, the server(s) +being peeked will relay *all* events it sees in that room to the peeking +server (including ones from other servers). It will also service the backfill +and event-retrieval APIs as if the peeking server was in the room. + +This continues until the peeking server calls DELETE on the peek it initiated. + +To start peeking, firstly the peeking server must pick server(s) to peek via. +This is typically the same server you would use to try to join the room via +(i.e. one taken from the alias, or the via param on a matrix.to URL). The +server could also call S2S /state on m.room.members to find other servers +participating in the room and try to peek them from too. + +The peeking server starts to peek by PUTting to `/peek` on the peeked server. +The request takes an empty object as a body as a placeholder for future (where +we might put filters). The peeking server selects an ID for the peeking +subscription for the purposes of idempotency. The ID must be 8 or less bytes +of ASCII and should be unique for a given peeking & peeked server. + +``` +PUT /_matrix/federation/v1/peek/{roomId}/{peekId} +{} + +{ + "peek_id": "12345", +} +``` + +``` +DELETE /_matrix/federation/v1/peek/{roomId}/{peekId} +{} +``` + +If the peeking server hasn't heard any events from the peeked server for a +while, it should attempt to re-PUT the /peek. If the peeked server is +unavailable, it should retry via other servers from the room's members until +it can reestablish. + +## Security considerations + +The peeked server becomes a centralisation point which could conspire against +the peeking server to withhold events. This is not that dissimilar to trying +to join a room via a malicious server, however, and can be mitigated somewhat +if the peeking server tries to query missing events from other servers. +The peeking server could also peek to multiple servers for resilience against +this sort of attack. + +The peeked server will be able to track the metadata surrounding which servers +are peeking into which of its rooms, and when. This could be particularly +sensitive for single-person servers peeking at profile rooms. + +## Design considerations + +This doesn't solve the problem that rooms wink out of existence when all +participants leave (https://github.com/matrix-org/matrix-doc/issues/534), +unlike other approaches to peeking (e.g. MSC1777) + +Do we allow filtering the peek? (e.g. if you only care about particular +events, or particular servers - e.g. if load-balancing peeking via multiple +servers). Similarly, is it concerning that this significantly overlaps with +the /sync CS API? + +How do we handle backpressure or rate limiting on the event stream (if at +all?) + +## Dependencies + +This unblocks MSC1769 (profiles as rooms) and MSC1772 (groups as rooms) +and is required for MSC1776 (peeking via /sync) to be of any use. + +## History + +This would close https://github.com/matrix-org/matrix-doc/issues/913 + +An earlier rejected solution is MSC1777, which proposed joining a pseudouser +(`@:server`) to a room in order to peek into it. However, being forced to write +to a room DAG (by joining such a user) in order to perform a read-only operation +(peeking) was deemed inefficient and rejected. From 8ef23c344cda9d9cfb8ca0efa9c0fdd201c3b571 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 24 Feb 2020 00:10:33 +0000 Subject: [PATCH 02/21] fix API --- proposals/2444-peeking-over-federation-peek-api.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index a371e6a8e88..c94534a7f87 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -43,18 +43,22 @@ we might put filters). The peeking server selects an ID for the peeking subscription for the purposes of idempotency. The ID must be 8 or less bytes of ASCII and should be unique for a given peeking & peeked server. +PUT and DELETE return 200 OK with an empty `{}` on success. + ``` PUT /_matrix/federation/v1/peek/{roomId}/{peekId} {} -{ - "peek_id": "12345", -} +200 OK +{} ``` ``` DELETE /_matrix/federation/v1/peek/{roomId}/{peekId} {} + +200 OK +{} ``` If the peeking server hasn't heard any events from the peeked server for a From fcfb491acd1a623fcb1f95f266af9d3434938c36 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 24 Feb 2020 00:38:41 +0000 Subject: [PATCH 03/21] clarify backfill & event retrieval for peekable rooms --- proposals/2444-peeking-over-federation-peek-api.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index c94534a7f87..85fe3af7484 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -26,8 +26,9 @@ We do this by subscribing to a room on one or more servers via a new `/peek` S2S API, which lets users on a given server declare their interest in viewing events in that room. Having started peeking into the room, the server(s) being peeked will relay *all* events it sees in that room to the peeking -server (including ones from other servers). It will also service the backfill -and event-retrieval APIs as if the peeking server was in the room. +server (including ones from other servers). Backfill +and event-retrieval APIs should be changed to be queryable from servers not +in the room if the room is peekable. This continues until the peeking server calls DELETE on the peek it initiated. From a2b53ce2531b1e02c191adbd807a2ddd0c9a3f57 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sun, 30 Aug 2020 19:13:02 +0300 Subject: [PATCH 04/21] Update proposals/2444-peeking-over-federation-peek-api.md Co-authored-by: Travis Ralston --- proposals/2444-peeking-over-federation-peek-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 85fe3af7484..31ac11b9da1 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -47,7 +47,7 @@ of ASCII and should be unique for a given peeking & peeked server. PUT and DELETE return 200 OK with an empty `{}` on success. ``` -PUT /_matrix/federation/v1/peek/{roomId}/{peekId} +PUT /_matrix/federation/v2/peek/{roomId}/{peekId} {} 200 OK From a438e9fdaa8d404467775a130a338f26ffac7059 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sat, 5 Sep 2020 00:54:48 +0100 Subject: [PATCH 05/21] refine based on implementation adventures --- .../2444-peeking-over-federation-peek-api.md | 60 ++++++++++++++++--- 1 file changed, 53 insertions(+), 7 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 31ac11b9da1..4d18e1ac876 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -39,19 +39,62 @@ server could also call S2S /state on m.room.members to find other servers participating in the room and try to peek them from too. The peeking server starts to peek by PUTting to `/peek` on the peeked server. +The peeked server is determined from the `server_names` parameter of the CS API +`/peek` command (from [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753)), +or failing that the domain of the room_alias or room_id being peeked. The request takes an empty object as a body as a placeholder for future (where we might put filters). The peeking server selects an ID for the peeking subscription for the purposes of idempotency. The ID must be 8 or less bytes -of ASCII and should be unique for a given peeking & peeked server. +of ASCII and should be unique for a given `{ origin_server, room_id, target_server }` +tuple. -PUT and DELETE return 200 OK with an empty `{}` on success. +We don't just use the `room_id` for idempotency because we may want to add +filtering in future to the /peek invocation, and this lets us distinguish +between different peeks which are filtering on different things for the +same room between the same pair of servers. Until filtering is added to the API, +implementors can just go use `room_id` as a `peek_id` for convenience. + +PUT returns 200 OK with the current state of the room being peeked on success, +using roughly the same response shape as the /state SS API. + +The response also includes a field called `renewal_interval` which specifies +after how many milliseconds of inactivity the peeked server requires the +peeking server to re-PUT the /peek in order for it to stay active. If the +peeked server is unavailable, it should retry via other servers from the +room's members until it can reestablish. + +PUT returns 403 if the user does not have permission to peek into the room, +and 404 if the room ID is not known to the peeked server. + +DELETE return 200 OK with an empty `{}` on success, and 404 if the room ID is +not known to the peeked server. ``` PUT /_matrix/federation/v2/peek/{roomId}/{peekId} {} 200 OK -{} +{ + "auth_chain": [ + { + "type": "m.room.minimal_pdu", + "room_id": "!somewhere:example.org", + "content": { + "see_room_version_spec": "The event format changes depending on the room version." + } + } + ], + "state": [ + { + "type": "m.room.minimal_pdu", + "room_id": "!somewhere:example.org", + "content": { + "see_room_version_spec": "The event format changes depending on the room version." + } + } + ], + "renewal_interval": 3600000 +} ``` ``` @@ -62,10 +105,13 @@ DELETE /_matrix/federation/v1/peek/{roomId}/{peekId} {} ``` -If the peeking server hasn't heard any events from the peeked server for a -while, it should attempt to re-PUT the /peek. If the peeked server is -unavailable, it should retry via other servers from the room's members until -it can reestablish. +The state block returned by /peek should be validated just as the one returned +by the /send_join API. + +When the user joins the peeked room, the server should just emit the right +membership event rather than calling /make_join or /send_join, to avoid the +unnecessary burden of a full room join, given the server is already participating +in the room. ## Security considerations From d2c67458fca231ea82c147372feca4ad1b20344f Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sat, 5 Sep 2020 00:56:58 +0100 Subject: [PATCH 06/21] 1776->2753 --- proposals/2444-peeking-over-federation-peek-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 4d18e1ac876..115156f59e8 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -143,7 +143,7 @@ all?) ## Dependencies This unblocks MSC1769 (profiles as rooms) and MSC1772 (groups as rooms) -and is required for MSC1776 (peeking via /sync) to be of any use. +and is required for MSC2753 (peeking via /sync) to be of any use. ## History From 7df4e7c09fd1846d227762b0ecd106538fb2eb40 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 7 Sep 2020 13:12:54 +0100 Subject: [PATCH 07/21] incorporate review --- .../2444-peeking-over-federation-peek-api.md | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 115156f59e8..5c3976d4f07 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -15,7 +15,7 @@ inconvenient in many other ways: (https://github.com/vector-im/riot-web/issues/2713) * You can't use rooms as generic pubsub mechanisms for synchronising data like profiles, groups, device-lists etc if you can't peek into them remotely. - * Search engines can't work if they can't peek remote rooms. + * Matrix-speaking search engines can't work if they can't peek remote rooms. ## Solution @@ -26,9 +26,9 @@ We do this by subscribing to a room on one or more servers via a new `/peek` S2S API, which lets users on a given server declare their interest in viewing events in that room. Having started peeking into the room, the server(s) being peeked will relay *all* events it sees in that room to the peeking -server (including ones from other servers). Backfill -and event-retrieval APIs should be changed to be queryable from servers not -in the room if the room is peekable. +server (including ones from other servers) via `/send` as if it were joined. +Backfill and event-retrieval APIs should be changed to be queryable from +servers not in the room if the room is peekable. This continues until the peeking server calls DELETE on the peek it initiated. @@ -58,15 +58,19 @@ PUT returns 200 OK with the current state of the room being peeked on success, using roughly the same response shape as the /state SS API. The response also includes a field called `renewal_interval` which specifies -after how many milliseconds of inactivity the peeked server requires the -peeking server to re-PUT the /peek in order for it to stay active. If the -peeked server is unavailable, it should retry via other servers from the -room's members until it can reestablish. +how often the peeked server requires the peeking server to re-PUT the /peek in +order for it to stay active. If the peeked server is unavailable, it should +retry via other servers from the room's members until it can reestablish. +We require `/peek`s to be regularly renewed even if the server has been accepting +peeked events, as the fact a server happens to accept peeked events doesn't +mean that it was necessarily deliberately peeking. PUT returns 403 if the user does not have permission to peek into the room, -and 404 if the room ID is not known to the peeked server. +and 404 if the room ID or peek ID is not known to the peeked server. +If the server implementation doesn't wish to honour the peek request due to +server load, it may respond with 429. -DELETE return 200 OK with an empty `{}` on success, and 404 if the room ID is +DELETE return 200 OK with an empty `{}` on success, and 404 if the room ID or peek ID is not known to the peeked server. ``` From b6afa4efcc68610ff091b303bedf8f7824574d8f Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sat, 12 Sep 2020 00:27:09 +0100 Subject: [PATCH 08/21] add room version support and use /v1 ns --- proposals/2444-peeking-over-federation-peek-api.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 5c3976d4f07..cc5c164c7f5 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -46,7 +46,8 @@ The request takes an empty object as a body as a placeholder for future (where we might put filters). The peeking server selects an ID for the peeking subscription for the purposes of idempotency. The ID must be 8 or less bytes of ASCII and should be unique for a given `{ origin_server, room_id, target_server }` -tuple. +tuple. The request takes `?ver=` querystring parameters with the same behaviour +as `/make_join` to advertise the room versions the peeking server supports. We don't just use the `room_id` for idempotency because we may want to add filtering in future to the /peek invocation, and this lets us distinguish @@ -68,13 +69,15 @@ mean that it was necessarily deliberately peeking. PUT returns 403 if the user does not have permission to peek into the room, and 404 if the room ID or peek ID is not known to the peeked server. If the server implementation doesn't wish to honour the peek request due to -server load, it may respond with 429. +server load, it may respond with 429. If the room version of the room being +peeked isn't supported by the peeking server, it responds with 400 and +`M_INCOMPATIBLE_ROOM_VERSION`. DELETE return 200 OK with an empty `{}` on success, and 404 if the room ID or peek ID is not known to the peeked server. ``` -PUT /_matrix/federation/v2/peek/{roomId}/{peekId} +PUT /_matrix/federation/v1/peek/{roomId}/{peekId}?ver=5&ver=6 {} 200 OK From a069d1a0da6ba76399197d0214976bca403fb577 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sat, 12 Sep 2020 12:40:29 +0100 Subject: [PATCH 09/21] clarify interaction with encryption --- .../2444-peeking-over-federation-peek-api.md | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index cc5c164c7f5..3cd64326ef2 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -8,15 +8,20 @@ in 2016. As well as stopping users from previewing rooms before joining, the fact that servers can't participate in remote rooms without joining them first is -inconvenient in many other ways: +inconvenient in other ways: - * You can't reliably participate in E2E encryption in rooms you're invited to - unless the server is actually participating in the room - (https://github.com/vector-im/riot-web/issues/2713) * You can't use rooms as generic pubsub mechanisms for synchronising data like - profiles, groups, device-lists etc if you can't peek into them remotely. + profiles, groups, reputation lists, device-lists etc if you can't peek into + them remotely. * Matrix-speaking search engines can't work if they can't peek remote rooms. +A related problem (not solved by this MSC) is that servers can't participate +in E2E encryption before joining a room, given the other users in the +room do not know to encrypt for the peeking device. This also impacts invited +users (https://github.com/vector-im/riot-web/issues/2713), given the invited +server doesn't currently tell the inviter about devicelist changes unless +the inviter is joined. + ## Solution We let servers participate in peekable rooms (i.e. those with `world_readable` @@ -120,6 +125,12 @@ membership event rather than calling /make_join or /send_join, to avoid the unnecessary burden of a full room join, given the server is already participating in the room. +It is considered a feature that you cannot peek into encrypted rooms, given +the act of peeking would leak the identity of the peeker to the joined users +in the room (as they'd need to encrypt for the peeker). This also feels +acceptable given there is little point in encrypting something intended to be +world-readable. + ## Security considerations The peeked server becomes a centralisation point which could conspire against From 14844e262ae03cb5ca854ac5314a7804cb8bf767 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Sat, 12 Sep 2020 13:40:02 +0100 Subject: [PATCH 10/21] more thoughts about invited users peeking into e2ee rooms --- proposals/2444-peeking-over-federation-peek-api.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 3cd64326ef2..de6bf6c6163 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -16,11 +16,15 @@ inconvenient in other ways: * Matrix-speaking search engines can't work if they can't peek remote rooms. A related problem (not solved by this MSC) is that servers can't participate -in E2E encryption before joining a room, given the other users in the -room do not know to encrypt for the peeking device. This also impacts invited -users (https://github.com/vector-im/riot-web/issues/2713), given the invited -server doesn't currently tell the inviter about devicelist changes unless -the inviter is joined. +in E2E encryption when peeking into a room, given the other users in the +room do not know to encrypt for the peeking device. + +Another related problem (not solved by this MSC) is that invited users can't +reliably participate in E2E encryption before joining a room, given the invited +server doesn't currently have a way to know about new users/devices in the room +without peeking, and so doesn't tell them if the invited user's devices changes. +(https://github.com/vector-im/element-web/issues/2713#issuecomment-691480736 +outlines a fix to this, not covered by this MSC). ## Solution From f021d504541bc28e7a4ec38782b8eb86ac2d845d Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 14 Sep 2020 01:11:47 +0100 Subject: [PATCH 11/21] add `latest_event_id` --- proposals/2444-peeking-over-federation-peek-api.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index de6bf6c6163..1ae69f15f78 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -75,6 +75,14 @@ We require `/peek`s to be regularly renewed even if the server has been acceptin peeked events, as the fact a server happens to accept peeked events doesn't mean that it was necessarily deliberately peeking. +Finally, the response also includes a `latest_event_id` field which provides +the most recent event (as ordered by stream id) at the point of the `/peek`. +This is the same event_id that you would pass to `/state` to get the same +response as this `/peek`. + +If the peek is renewed without having lapsed, `PUT /peek` simply returns `{}` +on success. + PUT returns 403 if the user does not have permission to peek into the room, and 404 if the room ID or peek ID is not known to the peeked server. If the server implementation doesn't wish to honour the peek request due to @@ -109,7 +117,8 @@ PUT /_matrix/federation/v1/peek/{roomId}/{peekId}?ver=5&ver=6 } } ], - "renewal_interval": 3600000 + "renewal_interval": 3600000, + "latest_event_id": "$6omAdDmGphrSAE_zH5RhMPAJzUQWAYEYPwr6wnl4ptz" } ``` From 005d0efb3ee4a41c082d00469ba36f88abb712d3 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Mon, 21 Sep 2020 13:43:55 +0100 Subject: [PATCH 12/21] clarify latest_event_id a bit more --- proposals/2444-peeking-over-federation-peek-api.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 1ae69f15f78..a3499b948d6 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -78,7 +78,8 @@ mean that it was necessarily deliberately peeking. Finally, the response also includes a `latest_event_id` field which provides the most recent event (as ordered by stream id) at the point of the `/peek`. This is the same event_id that you would pass to `/state` to get the same -response as this `/peek`. +response as this `/peek`, and identifies the precise point in time that the +given state refers to. If the peek is renewed without having lapsed, `PUT /peek` simply returns `{}` on success. From e04e4c3f797f80736029937dd03eac4c574253f8 Mon Sep 17 00:00:00 2001 From: Matthew Hodgson Date: Tue, 22 Sep 2020 23:43:08 +0100 Subject: [PATCH 13/21] switch to returning `latest_event` itself to avoid a roundtrip --- .../2444-peeking-over-federation-peek-api.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index a3499b948d6..4ca84d5a88f 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -75,11 +75,12 @@ We require `/peek`s to be regularly renewed even if the server has been acceptin peeked events, as the fact a server happens to accept peeked events doesn't mean that it was necessarily deliberately peeking. -Finally, the response also includes a `latest_event_id` field which provides +Finally, the response also includes a `latest_event` field which provides the most recent event (as ordered by stream id) at the point of the `/peek`. -This is the same event_id that you would pass to `/state` to get the same +This is the event whose id you would pass to `/state` to get the same response as this `/peek`, and identifies the precise point in time that the -given state refers to. +given state refers to. We return a full event rather than an eventid to +avoid an unnecessary roundtrip. If the peek is renewed without having lapsed, `PUT /peek` simply returns `{}` on success. @@ -119,7 +120,14 @@ PUT /_matrix/federation/v1/peek/{roomId}/{peekId}?ver=5&ver=6 } ], "renewal_interval": 3600000, - "latest_event_id": "$6omAdDmGphrSAE_zH5RhMPAJzUQWAYEYPwr6wnl4ptz" + "latest_event": { + "type": "m.room.minimal_pdu", + "room_id": "!somewhere:example.org", + "content": { + "see_room_version_spec": "The event format changes depending on the room version." + } + } + } } ``` From 9ec656c55c21ac101616eee7a5f82139e10ad753 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Wed, 18 Nov 2020 17:00:52 +0000 Subject: [PATCH 14/21] various updates --- .../2444-peeking-over-federation-peek-api.md | 268 +++++++++++------- 1 file changed, 172 insertions(+), 96 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 4ca84d5a88f..928ad9a7015 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -31,121 +31,183 @@ outlines a fix to this, not covered by this MSC). We let servers participate in peekable rooms (i.e. those with `world_readable` `m.room.history_visibility`) without having actually joined them. -We do this by subscribing to a room on one or more servers via a new `/peek` -S2S API, which lets users on a given server declare their interest in viewing -events in that room. Having started peeking into the room, the server(s) -being peeked will relay *all* events it sees in that room to the peeking -server (including ones from other servers) via `/send` as if it were joined. -Backfill and event-retrieval APIs should be changed to be queryable from -servers not in the room if the room is peekable. - -This continues until the peeking server calls DELETE on the peek it initiated. - -To start peeking, firstly the peeking server must pick server(s) to peek via. -This is typically the same server you would use to try to join the room via -(i.e. one taken from the alias, or the via param on a matrix.to URL). The -server could also call S2S /state on m.room.members to find other servers -participating in the room and try to peek them from too. - -The peeking server starts to peek by PUTting to `/peek` on the peeked server. -The peeked server is determined from the `server_names` parameter of the CS API -`/peek` command (from [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753)), -or failing that the domain of the room_alias or room_id being peeked. -The request takes an empty object as a body as a placeholder for future (where -we might put filters). The peeking server selects an ID for the peeking -subscription for the purposes of idempotency. The ID must be 8 or less bytes -of ASCII and should be unique for a given `{ origin_server, room_id, target_server }` -tuple. The request takes `?ver=` querystring parameters with the same behaviour -as `/make_join` to advertise the room versions the peeking server supports. - -We don't just use the `room_id` for idempotency because we may want to add -filtering in future to the /peek invocation, and this lets us distinguish -between different peeks which are filtering on different things for the -same room between the same pair of servers. Until filtering is added to the API, -implementors can just go use `room_id` as a `peek_id` for convenience. - -PUT returns 200 OK with the current state of the room being peeked on success, -using roughly the same response shape as the /state SS API. - -The response also includes a field called `renewal_interval` which specifies -how often the peeked server requires the peeking server to re-PUT the /peek in -order for it to stay active. If the peeked server is unavailable, it should -retry via other servers from the room's members until it can reestablish. -We require `/peek`s to be regularly renewed even if the server has been accepting -peeked events, as the fact a server happens to accept peeked events doesn't -mean that it was necessarily deliberately peeking. - -Finally, the response also includes a `latest_event` field which provides -the most recent event (as ordered by stream id) at the point of the `/peek`. -This is the event whose id you would pass to `/state` to get the same -response as this `/peek`, and identifies the precise point in time that the -given state refers to. We return a full event rather than an eventid to -avoid an unnecessary roundtrip. - -If the peek is renewed without having lapsed, `PUT /peek` simply returns `{}` -on success. - -PUT returns 403 if the user does not have permission to peek into the room, -and 404 if the room ID or peek ID is not known to the peeked server. -If the server implementation doesn't wish to honour the peek request due to -server load, it may respond with 429. If the room version of the room being -peeked isn't supported by the peeking server, it responds with 400 and -`M_INCOMPATIBLE_ROOM_VERSION`. - -DELETE return 200 OK with an empty `{}` on success, and 404 if the room ID or peek ID is -not known to the peeked server. +Firstly, this means that a number of federation endpoints should be updated to +allow inspection of `world_readable` rooms. This includes: + + * [`GET /_matrix/federation/v1/event_auth/{roomId}/{eventId}`](https://matrix.org/docs/spec/server_server/r0.1.4#get-matrix-federation-v1-event-auth-roomid-eventid) + * [`GET /_matrix/federation/v1/backfill/{roomId}`](https://matrix.org/docs/spec/server_server/r0.1.4#get-matrix-federation-v1-backfill-roomid) + * [`POST /_matrix/federation/v1/get_missing_events/{roomId}`](https://matrix.org/docs/spec/server_server/r0.1.4#post-matrix-federation-v1-get-missing-events-roomid) + * [`GET /_matrix/federation/v1/state/{roomId}`](https://matrix.org/docs/spec/server_server/r0.1.4#get-matrix-federation-v1-state-roomid) + * [`GET /_matrix/federation/v1/state_ids/{roomId}`](https://matrix.org/docs/spec/server_server/r0.1.4#get-matrix-federation-v1-state-ids-roomid) + * [`GET /_matrix/federation/v1/event/{eventId}`](https://matrix.org/docs/spec/server_server/r0.1.4#get-matrix-federation-v1-event-eventid) + +(Of course, these apis should only allow access to `world_readable` parts of +the history.) + +Secondly, we introduce a new API allowing servers to subscribe to new events. + +### Initiating a peek + +To start peeking, firstly the peeking server must pick server(s) to peek +via. It can do this based on the `servers` parameter of the CS API `/peek` +command (from [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753)), +or failing that the domain of the room alias being peeked. + +The peeking server then makes a `/peek` request to the target server. An +example request and response might look like: ``` -PUT /_matrix/federation/v1/peek/{roomId}/{peekId}?ver=5&ver=6 +PUT /_matrix/federation/v1/peek/{roomId}/{peekId}?ver=5&ver=6 HTTP/1.1 {} 200 OK { - "auth_chain": [ - { - "type": "m.room.minimal_pdu", - "room_id": "!somewhere:example.org", - "content": { - "see_room_version_spec": "The event format changes depending on the room version." - } - } + "latest_event_state_ids": { + "$fwd_extremity_1": [ + "$state_event_3", + "$state_event_4" ], - "state": [ - { - "type": "m.room.minimal_pdu", - "room_id": "!somewhere:example.org", - "content": { - "see_room_version_spec": "The event format changes depending on the room version." - } - } - ], - "renewal_interval": 3600000, - "latest_event": { - "type": "m.room.minimal_pdu", - "room_id": "!somewhere:example.org", - "content": { - "see_room_version_spec": "The event format changes depending on the room version." - } - } + "$fwd_extremity_2": [ + "$state_event_5", + "$state_event_6" + ] + }, + "common_state_ids": [ + "$state_event_1", + "$state_event_2", + ], + "events": [ + { + "type": "m.room.member", + "room_id": "!somewhere:example.org", + "content": { /* ... */ } } + ], + "renewal_interval": 3600000 } ``` +The request takes an empty object as a body as a placeholder for future +extension. + +The peeking server selects an ID for the peeking subscription for the purposes +of idempotency. The ID must be unique for a given `{ origin_server, room_id, +target_server }` tuple, and should be a string consisting of the characters +`[0-9a-zA-Z.=_-]`. Its length must not exceed 8 characters and it should not be +empty. + +The request takes `?ver=` querystring parameters with the same behaviour as +`/make_join` to advertise the room versions the peeking server supports. + +If the request is successful, the target server retuns a 200 response with the +following fields: + * `latest_event_state_ids`: a map whose keys are the IDs of the events forming + the target server's current forward extremities in the room. The values are + lists of the IDs of the events forming the room state after the event in + question, excluding any events in `common_state_ids`. + + TBD: would the state *before* the extremity event be more useful? + + * `common_state_ids`: A list of the IDs of any events which are common to the room + states after *all* of the forward extremities in the room. + + * `events`: The bodies of any events whose IDs are: + * listed in the keys of `latest_event_state_ids`, or: + * listed in the values of `latest_event_state_ids`, or: + * listed in the values of `common_state_ids`, or: + * listed in the `auth_events` field of any of the above events, or: + * listed in the `auth_events` of the `auth_events`, recursively. + + * `renewal_interval`: a duration in milliseconds after which the target server + will expire the peek. The peeking server must renew the peek before that + time to be sure of continuing to receive events. + +If the room is not peekable, the target server should return a 403 error with + `M_FORBIDDEN`. + +If the room is not known to the target server, it should return a 404 error +with `M_NOT_FOUND`. + +If the peek ID is not valid, the target server responds with 400 and `M_UNRECOGNIZED`. + +If the room version of the room being peeked isn't supported by the peeking +server, the target server responds with 400 and `M_INCOMPATIBLE_ROOM_VERSION`. + +If the target server doesn't wish to honour the peek request due to server load +or rate-limiting, it may respond with 429 and `M_LIMIT_EXCEEDED`, including a +`retry_after_ms` value indicating when the request could be retried. + +The room states returned by `/peek` should be validated just as the one +returned by the `/send_join` API. If the peeking server finds the response +unacceptable, it should cancel the peek with a `DELETE` request (see below). + +XXX: it might be better to split this into two operations: first fetch the +state data, then begin the peek operation by sending your idea of the forward +extremities, to bring you up to date with anything you missed. This would +reduce the chance of having to immediately cancel a peek, and would be more +efficient in the case of rapid `peek-unpeek-peek` switches. + +While a peek subscription is active, the target server must relay any events +received in that room over the [`PUT +/_matrix/federation/v1/send/{txnId}`](https://matrix.org/docs/spec/server_server/r0.1.4#put-matrix-federation-v1-send-txnid) +API. + +### Renewing a peek + +The target server will eventually expire a peek if it is not renewed. The +peeking server can renew a peek by calling `POST +/_matrix/federation/v1/peek/{roomId}/{peekId}/renew`: + ``` -DELETE /_matrix/federation/v1/peek/{roomId}/{peekId} +POST /_matrix/federation/v1/peek/{roomId}/{peekId}/renew HTTP/1.1 {} +200 OK +{ + "renewal_interval": 3600000 +} +``` + +The target server simply returns the new `renewal_interval`. + +If the peek ID is not known for the `{ origin_server, room_id, target_server }` +tuple, the target server returns a 404 error with `M_NOT_FOUND`. + +### Deleting a peek + +The peeking server may terminate a peek by calling `DELETE +/_matrix/federation/v1/peek/{roomId}/{peekId}`: + +``` +DELETE /_matrix/federation/v1/peek/{roomId}/{peekId} HTTP/1.1 +Content-Length: 0 + 200 OK {} ``` -The state block returned by /peek should be validated just as the one returned -by the /send_join API. +The request has no body [1](#f1). On success, the target +server returns a 200 with an empty json object. -When the user joins the peeked room, the server should just emit the right -membership event rather than calling /make_join or /send_join, to avoid the -unnecessary burden of a full room join, given the server is already participating -in the room. +If the peek ID is not known for the `{ origin_server, room_id, target_server }` +tuple, the target server returns a 404 error with `M_NOT_FOUND`. + +### Expiring a peek + +The target server should expire any peek which is not renewed before the +`renewal_interval` elapses. + +XXX how to tell the peeking server? + +### Joining a room + +When the user joins the peeked room, the peeking server should just emit the +right membership event rather than calling `/make_join` or `/send_join`, to +avoid the unnecessary burden of a full room join, given the server is already +participating in the room. It should also send a `DELETE` request to cancel +any active peeks. + +### Encrypted rooms It is considered a feature that you cannot peek into encrypted rooms, given the act of peeking would leak the identity of the peeker to the joined users @@ -153,6 +215,13 @@ in the room (as they'd need to encrypt for the peeker). This also feels acceptable given there is little point in encrypting something intended to be world-readable. +## Alternatives + + * simply use `room_id` for idempotency rather than requiring a separate + `peek_id`. One reason not to do this is to allow a future extension where + there are multiple subscriptions active, each filtering out different event + types. In the meantime, implementers can use a hard-coded constant. + ## Security considerations The peeked server becomes a centralisation point which could conspire against @@ -193,3 +262,10 @@ An earlier rejected solution is MSC1777, which proposed joining a pseudouser (`@:server`) to a room in order to peek into it. However, being forced to write to a room DAG (by joining such a user) in order to perform a read-only operation (peeking) was deemed inefficient and rejected. + + +## Footnotes + +[1]: per +https://www.ietf.org/archive/id/draft-ietf-httpbis-semantics-12.html#name-delete: +"A client SHOULD NOT generate a body in a DELETE request." [↩](#a1) From 721e3fba547373d42dd1b2e3b16252bdc71222c0 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 19 Nov 2020 12:26:42 +0000 Subject: [PATCH 15/21] notes on DoS attacks --- .../2444-peeking-over-federation-peek-api.md | 35 +++++++++++-------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 928ad9a7015..fec7506e06c 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -91,7 +91,7 @@ The request takes an empty object as a body as a placeholder for future extension. The peeking server selects an ID for the peeking subscription for the purposes -of idempotency. The ID must be unique for a given `{ origin_server, room_id, +of idempotency. The ID must be unique for a given `{ peeking_server, room_id, target_server }` tuple, and should be a string consisting of the characters `[0-9a-zA-Z.=_-]`. Its length must not exceed 8 characters and it should not be empty. @@ -150,7 +150,9 @@ efficient in the case of rapid `peek-unpeek-peek` switches. While a peek subscription is active, the target server must relay any events received in that room over the [`PUT /_matrix/federation/v1/send/{txnId}`](https://matrix.org/docs/spec/server_server/r0.1.4#put-matrix-federation-v1-send-txnid) -API. +API. If a peeking server has multiple peeks active for a given room and target +server, the target server should still only send one copy of each event, rather +than duplicating the event for each peek. ### Renewing a peek @@ -170,7 +172,7 @@ POST /_matrix/federation/v1/peek/{roomId}/{peekId}/renew HTTP/1.1 The target server simply returns the new `renewal_interval`. -If the peek ID is not known for the `{ origin_server, room_id, target_server }` +If the peek ID is not known for the `{ peeking_server, room_id, target_server }` tuple, the target server returns a 404 error with `M_NOT_FOUND`. ### Deleting a peek @@ -189,7 +191,7 @@ Content-Length: 0 The request has no body [1](#f1). On success, the target server returns a 200 with an empty json object. -If the peek ID is not known for the `{ origin_server, room_id, target_server }` +If the peek ID is not known for the `{ peeking_server, room_id, target_server }` tuple, the target server returns a 404 error with `M_NOT_FOUND`. ### Expiring a peek @@ -224,16 +226,21 @@ world-readable. ## Security considerations -The peeked server becomes a centralisation point which could conspire against -the peeking server to withhold events. This is not that dissimilar to trying -to join a room via a malicious server, however, and can be mitigated somewhat -if the peeking server tries to query missing events from other servers. -The peeking server could also peek to multiple servers for resilience against -this sort of attack. - -The peeked server will be able to track the metadata surrounding which servers -are peeking into which of its rooms, and when. This could be particularly -sensitive for single-person servers peeking at profile rooms. + * A malicious server could set up multiple peeks to multiple target servers by + way of attempting a denial-of-service attack. Server implementations should + rate-limit requests to establish peeks, as well as limiting the absolute + number of active peeks each server may have, to mitigate this. + + * The peeked server becomes a centralisation point which could conspire + against the peeking server to withhold events. This is not that dissimilar + to trying to join a room via a malicious server, however, and can be + mitigated somewhat if the peeking server tries to query missing events from + other servers. The peeking server could also peek to multiple servers for + resilience against this sort of attack. + + * The peeked server will be able to track the metadata surrounding which + servers are peeking into which of its rooms, and when. This could be + particularly sensitive for single-person servers peeking at profile rooms. ## Design considerations From bf667ad6fe46a22902f852bc07db04653aeed027 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 19 Nov 2020 12:31:14 +0000 Subject: [PATCH 16/21] filtering is descoped --- .../2444-peeking-over-federation-peek-api.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index fec7506e06c..9638e23ed2c 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -224,6 +224,17 @@ world-readable. there are multiple subscriptions active, each filtering out different event types. In the meantime, implementers can use a hard-coded constant. +## Future extensions + +These features are explicitly descoped for the purposes of this MSC. + + * It may be useful to allow peeking servers to "filter" the events to be + returned - for example, if you only care about particular events, or + particular servers - e.g. if load-balancing peeking via multiple servers. + + It's worth noting that this would make it very hard for peeking servers to + reliably track state changes and detect missing events. + ## Security considerations * A malicious server could set up multiple peeks to multiple target servers by @@ -246,12 +257,7 @@ world-readable. This doesn't solve the problem that rooms wink out of existence when all participants leave (https://github.com/matrix-org/matrix-doc/issues/534), -unlike other approaches to peeking (e.g. MSC1777) - -Do we allow filtering the peek? (e.g. if you only care about particular -events, or particular servers - e.g. if load-balancing peeking via multiple -servers). Similarly, is it concerning that this significantly overlaps with -the /sync CS API? +unlike other approaches to peeking (e.g. MSC1777). How do we handle backpressure or rate limiting on the event stream (if at all?) From 3868d0ffb94f282231a34249f27768b1bd919740 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 19 Nov 2020 12:41:58 +0000 Subject: [PATCH 17/21] expand comparison with #1777 --- .../2444-peeking-over-federation-peek-api.md | 47 ++++++++++++++----- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 9638e23ed2c..eab3ec1792b 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -224,6 +224,35 @@ world-readable. there are multiple subscriptions active, each filtering out different event types. In the meantime, implementers can use a hard-coded constant. + * An earlier rejected solution is + [MSC1777](https://github.com/matrix-org/matrix-doc/pull/1777), which + proposed joining a pseudouser (`@:server`) to a room in order to peek into + it. However: + + * being forced to write to a room DAG (by joining such a user) in order to + perform a read-only operation (peeking) is somewhat inefficient. + + * it also constitutes a privacy violation - tantamount to a tracking + pixel. If I'm on a smallish server and I happen to briefly peek into + `#nsfw:matrix.org`, I don't want every random server in that room to know + that I did so. It's even worse than sending read-receipts. + + * In the interests of empowering the user, it's actually quite nice to + (theoretically at least) have some control over which servers you trust to + peek via. + + * That said, a P2P world is going to need a totally different approach, + which might be back towards MSC1777 but using + [MSC1228](https://github.com/matrix-org/matrix-doc/pull/1228)-style IDs + to protect privacy. We need to solve scalability of "which nodes are + participating in the room" irrespectively of whether those nodes are + active or passive. So it's worth noting this solution (MSC2444) is very + much for today's federated architecture. + + * MSC1777 offers a solution for EDU transmission which this MSC does not, + given we don't currently have any data flows for mirroring other servers' + EDUs. + ## Future extensions These features are explicitly descoped for the purposes of this MSC. @@ -264,18 +293,14 @@ all?) ## Dependencies -This unblocks MSC1769 (profiles as rooms) and MSC1772 (groups as rooms) -and is required for MSC2753 (peeking via /sync) to be of any use. - -## History - -This would close https://github.com/matrix-org/matrix-doc/issues/913 - -An earlier rejected solution is MSC1777, which proposed joining a pseudouser -(`@:server`) to a room in order to peek into it. However, being forced to write -to a room DAG (by joining such a user) in order to perform a read-only operation -(peeking) was deemed inefficient and rejected. +This unblocks [MSC1769](https://github.com/matrix-org/matrix-doc/pull/1769) +(profiles as rooms) and +[MSC1772](https://github.com/matrix-org/matrix-doc/pull/1772) (Matrix Spaces), +and is required for +[MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) (peeking via +/sync) to be of any use. +This would close https://github.com/matrix-org/matrix-doc/issues/913. ## Footnotes From 6ced97680f2d62e8eddd742bbfff5196414b06c8 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 19 Nov 2020 12:44:16 +0000 Subject: [PATCH 18/21] formatting fix --- .../2444-peeking-over-federation-peek-api.md | 21 +++++++++---------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index eab3ec1792b..339aeb328ad 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -241,17 +241,16 @@ world-readable. (theoretically at least) have some control over which servers you trust to peek via. - * That said, a P2P world is going to need a totally different approach, - which might be back towards MSC1777 but using - [MSC1228](https://github.com/matrix-org/matrix-doc/pull/1228)-style IDs - to protect privacy. We need to solve scalability of "which nodes are - participating in the room" irrespectively of whether those nodes are - active or passive. So it's worth noting this solution (MSC2444) is very - much for today's federated architecture. - - * MSC1777 offers a solution for EDU transmission which this MSC does not, - given we don't currently have any data flows for mirroring other servers' - EDUs. + * That said, a P2P world is going to need a totally different approach, + which might be back towards MSC1777 but using + [MSC1228](https://github.com/matrix-org/matrix-doc/pull/1228)-style IDs to + protect privacy. We need to solve scalability of "which nodes are + participating in the room" irrespectively of whether those nodes are + active or passive. So it's worth noting this solution (MSC2444) is very + much for today's federated architecture. + + * MSC1777 offers a solution for EDU transmission which this MSC does not, + given we don't currently have any data flows for mirroring other servers' EDUs. ## Future extensions From 696aa6e7f95196cbdd2dfaf79ed8752cf36d37c8 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 19 Nov 2020 12:48:45 +0000 Subject: [PATCH 19/21] add proxying as an alternative --- proposals/2444-peeking-over-federation-peek-api.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 339aeb328ad..7725a9d9cb5 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -250,7 +250,15 @@ world-readable. much for today's federated architecture. * MSC1777 offers a solution for EDU transmission which this MSC does not, - given we don't currently have any data flows for mirroring other servers' EDUs. + given we don't currently have any data flows for mirroring other servers' + EDUs. + + * Rather than attempting to maintain a local replica of peeked traffic, have + the peeking server proxy any peek requests from the client-server API onto + the target server, somehow. This couples room availability to the peeked + server and means we don't have a local replica we can index or serve + independently etc - and also means you could have problems deduplicating + peeks between local clients. ## Future extensions From 1ae5f0fbb1d4a3b90c101a21e164f14c9d4ca3c2 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 19 Nov 2020 12:55:13 +0000 Subject: [PATCH 20/21] notes on peek_id --- proposals/2444-peeking-over-federation-peek-api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 7725a9d9cb5..587e42d4fd4 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -222,7 +222,9 @@ world-readable. * simply use `room_id` for idempotency rather than requiring a separate `peek_id`. One reason not to do this is to allow a future extension where there are multiple subscriptions active, each filtering out different event - types. In the meantime, implementers can use a hard-coded constant. + types. Another reason that `peek_id` is useful is to improve idempotency + with rapid peek/unpeek/peek cycles, to ensure we aren't `DELETE`ing the + wrong peek. * An earlier rejected solution is [MSC1777](https://github.com/matrix-org/matrix-doc/pull/1777), which From 6fe3b99939a1516a723983ae98e1dd3f43eee6fb Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 19 Nov 2020 18:25:30 +0000 Subject: [PATCH 21/21] expired peeks --- .../2444-peeking-over-federation-peek-api.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/proposals/2444-peeking-over-federation-peek-api.md b/proposals/2444-peeking-over-federation-peek-api.md index 587e42d4fd4..22a3444ca22 100644 --- a/proposals/2444-peeking-over-federation-peek-api.md +++ b/proposals/2444-peeking-over-federation-peek-api.md @@ -199,7 +199,21 @@ tuple, the target server returns a 404 error with `M_NOT_FOUND`. The target server should expire any peek which is not renewed before the `renewal_interval` elapses. -XXX how to tell the peeking server? +It should indicate the expiry to the peeking server via `PUT +/_matrix/federation/v1/send/`, via a new `expired_peeks` key: + +``` +PUT /_matrix/federation/v1/send/S0meTransacti0nId HTTP/1.1 + +{ + "expired_peeks": [ + { + "room_id": "{roomId}", + "peek_id": "{peekId}" + } + ] +} +``` ### Joining a room