MSC4186: Simplified Sliding Sync #4186

erikjohnston · 2024-08-30T08:31:04Z

Gnuxie · 2024-08-30T12:18:34Z

proposals/4186-simplified-sliding-sync.md

+      "initial": true|null,
+
+      // Same as in sync v2. Indicates whether there are more events to fetch than those in the timeline.
+      "limited:" true|null,


Are the options in the request and response supposed to be in a union with null?

Isn't it the convention in the spec that the property is omitted entirely rather than given the value null?

(For clarity, this is something that jumps out at me since frequently people don't realise omitting a property and giving the value null are distinct things in JSON. Again, I'm not sure what the convention is for the CS-API though.)

Hmm, seems like limited is already optional in the timeline portion of sync? https://github.com/matrix-org/matrix-spec/blob/7f2f100420a355cf8ea50d08329e6f98989fc975/data/api/client-server/definitions/timeline_batch.yaml#L15-L36

But this does not imply a union with null (it's not explicit). It could if there is another section of the specification that says something to the effect "all unrequired properties in responses and requests should be treated as though their values are in a union with null" somewhere?

It might be better to rewrite this section using JSONSchema so that the intent is clearer in either case.

turt2live · 2024-08-30T15:11:54Z

proposals/4186-simplified-sliding-sync.md

Implementation requirements:

Client (ideally multiple)

Server

Gnuxie · 2024-08-30T15:58:57Z

proposals/4186-simplified-sliding-sync.md

+        // Flag which only returns rooms which have an `m.room.encryption` state event. If unset,
+        // both encrypted and unencrypted rooms are returned. If false, only unencrypted rooms
+        // are returned. If true, only encrypted rooms are returned.
+        "is_invite": true|false|null,


documentation is copied across from is_encrypted

Gnuxie · 2024-08-30T16:25:56Z

proposals/4186-simplified-sliding-sync.md

+The format of returned state in `required_state` is a list of events. This does now allow the server to indicate if a
+"state reset" has happened which removed an entry from the state entirely (rather than it being replaced with another
+event).
+
+This is particularly problematic if the user gets "state reset" out of the room, where the server has no mechanism to
+indicate to the client that the user has effectively left the room (the server has no leave event to return).
+
+We may want to allow special entries in the `required_state` list of the form
+`{"type": .., "state_key": .., content: null}` to indicate that the state entry has been removed.


This is pretty serious and I don't think we can leave it up to clients to figure out somehow. Since that requires client authors to both be extremely knowledgeable about how room state works and also write perfect code. The chances of both of those happening is slim. For example, I tried to write some code to track room state from legacy /sync (by also using /state) in Draupnir and it seems like I haven't even been able to do that right. https://github.com/Gnuxie/matrix-protection-suite/blob/main/src/StateTracking/StateChangeType.ts#L13-L64, https://github.com/Gnuxie/matrix-protection-suite/blob/main/src/StateTracking/StateChangeType.ts#L83-L123. The situation where a state event can be removed entirely via a state reset is not even accounted for. Partially because I was under the impression that this was not something that could happen, as I was a dumbass.

You can read more about the way Draupnir collects state here https://marewolf.me/posts/draupnir/2402.html#development-room-state-tracking. But it would be good if clients didn't need to calculate state deltas at all and there was more specialized API support for that. There'd need to be consideration for how such an API would interact with timeline gaps if it was done alongside /sync.

Johennes · 2024-09-03T14:05:22Z

proposals/4186-simplified-sliding-sync.md

+
+The core differences between sync v2 and simplified sliding sync are:
+
+- The server initially only sends the most recent N rooms to the client (where N is specified by the lcient), which then


Suggested change

- The server initially only sends the most recent N rooms to the client (where N is specified by the lcient), which then

- The server initially only sends the most recent N rooms to the client (where N is specified by the client), which then

clokep · 2024-09-03T14:13:47Z

proposals/4186-simplified-sliding-sync.md

+
+The current `/sync` endpoint scales badly as the number of rooms on an account increases. It scales badly because all
+rooms are returned to the client, incremental syncs are unbounded and slow down based on how long the user has been
+offline, and clients cannot opt-out of a large amount of extraneous data such as receipts. On large accounts with


Can't data be removed with filters? (See RoomFilter)

clokep · 2024-09-03T14:21:20Z

proposals/4186-simplified-sliding-sync.md

+- Time from launch to confident usability should be as low as possible.
+- Time from login on existing accounts to usability should be as low as possible.


I don't really get the difference here. What is "launch"? Does this just mean an initial login vs an already logged in user?

clokep · 2024-09-03T14:22:30Z

proposals/4186-simplified-sliding-sync.md

+- The client can configure which information the server will return for different sets of rooms (e.g. a smaller timeline
+  limit for older rooms).
+- The client can filter what rooms it is interested in
+- The client can maintain multiple sync loops (with some caveats)


Why is this useful? When do clients want to use this?

Primarily for iOS which has 2 processes (a push process (NSE) and the main app process).

clokep · 2024-09-03T14:23:40Z

proposals/4186-simplified-sliding-sync.md

+The core component of a sliding sync request is “lists”, which specify what information to return about which rooms.
+Each list specifies some filters on rooms (e.g. ignore spaces), the range of filtered rooms to select (e.g. the most
+recent 20 filtered rooms), and the config for the data to return for those rooms (e.g. the required state, timeline
+limit, etc). The order of rooms is always done based on when the server received the most recent event for the room.


"Stream order" in synapse parlance?

Yes, should standardize this type of language.

Suggested change

limit, etc). The order of rooms is always done based on when the server received the most recent event for the room.

limit, etc). The order of rooms is always done based on the arrival time of the most recent event for the room on the homeserver.

clokep · 2024-09-03T14:29:58Z

proposals/4186-simplified-sliding-sync.md

+Multiple lists and subscriptions can be specified in a request. If a room matches multiple lists/subscriptions then the
+config is “combined” to be the superset of all configs (e.g. take the maximum timeline limit). See below for the exact
+algorithm.


So lists + subscriptions replace filters?

clokep · 2024-09-03T14:51:22Z

proposals/4186-simplified-sliding-sync.md

+
+The client can use room subscriptions, with a `timeline_limit` of 20, to preload history for the top rooms. This means
+that if the user clicks on one of the top rooms the app can immediately display a screens worth of history. (An
+alternative would be to have a second list with a static range of `[0, 19]` and a `timeline_limit` of 20. The downside)


I think this sentence was unfinished.

clokep · 2024-09-03T14:51:56Z

proposals/4186-simplified-sliding-sync.md

+The client then increases the range (in the next request) to `[0, 99]`, which will return the next 80 rooms. The server
+may sort the rooms differently than they are returned by the server (e.g. they may ignore reactions for sorting
+purposes).


I'm confused at how the server now only gives 80 rooms and not 100. The definition above seems to imply it would be 100 rooms?

I'll make this clearer, but the idea is that 100 rooms are in scope due to [0, 99], but we've already sent down the first 20 rooms and so they aren't sent down again

clokep · 2024-09-03T14:53:11Z

proposals/4186-simplified-sliding-sync.md

+rooms. If there are more rooms to send to the client (beyond `page_size`), then the client can request to "paginate" in
+these missed updates in subsequent updates.
+
+Since this would require client side changes, this should be explored in a separate MSC.


Doesn't all of this require client side changes?

clokep · 2024-09-03T14:54:43Z

proposals/4186-simplified-sliding-sync.md

+We could implement a bulk `/messages` endpoint, where the client would specify multiple rooms and `prev_batch` tokens.
+We can also add a flag to disable attempting to backfill over pagination (to match the behaviour of the sync timeline).
+
+## `required_state` response format


This section seems to contradict the opening that this MSC solves the issue of clients having incorrect state.

clokep · 2024-09-03T14:56:21Z

proposals/4186-simplified-sliding-sync.md

+  // The position to use as the `since` token in the next sliding sync request.
+  // c.f. Connections.
+  "pos": "<opaque string>",


I think there was an effort to standardize pagination terminology?

proposals/4186-simplified-sliding-sync.md

MadLittleMods · 2024-09-05T18:05:17Z

proposals/4186-simplified-sliding-sync.md

+      // The "heroes" for the room, if there is no room name. Only sent initially and when it changes.
+      "heroes": [
+        {"user_id":"@alice:example.com","displayname":"Alice","avatar_url":"mxc://..."},
+      ],


Do the heroes membership state events need to be included in required_state response when requesting required_state: ["m.room.member", "$LAZY"] (lazy-loading room members)?

Sync v2 says this:

When lazy-loading room members is enabled, the membership events for the heroes MUST be included in the state, unless they are redundant. When the list of users changes, the server notifies the client by sending a fresh list of heroes. If there are no changes since the last sync, this field may be omitted.

But I think that's because m.heroes in Sync v2 is only a list of user ID's. Whereas, in the sliding sync response here, we already have all of the info necessary in these stripped events.

One alternative is to match Sync v2 and only list the user ID's in heroes and include the membership events in required_state.

MadLittleMods · 2024-09-05T18:09:16Z

proposals/4186-simplified-sliding-sync.md

+  // the room appears in multiple lists and/or room subscriptions.
+  "rooms": {
+    "!foo:example.com": {
+      // The room name, if one exists. Only sent initially and when it changes.


Probably worth stating that this is strictly the value from the m.room.name event given the previous MSC3575 was previously trying to do a bunch of room name calculation stuff.

Suggested change

// The room name, if one exists. Only sent initially and when it changes.

// The room name (`m.room.name` -> `content.name`), if one exists. Only sent initially and when it changes.

MadLittleMods · 2024-09-05T18:21:43Z

proposals/4186-simplified-sliding-sync.md

+      // The list of events, sorted least to most recent.
+      "timeline": [ ... ],
+      // The current state of the room as a list of events
+      "required_state": [ ... ],


We should call out that the required_state, timeline, limited, num_live, prev_batch fields will be omitted for invite/knock rooms. Only invite_state/knock_state is provided.

The reason these can't be included is because we don't have any of that information for remote invites and the user isn't participating in the room yet so we shouldn't leak anything to them. We can only rely on the information in the invite event.

Related previous conversation: #3575 (comment)

MadLittleMods · 2024-09-05T18:31:27Z

proposals/4186-simplified-sliding-sync.md

+      "heroes": [
+        {"user_id":"@alice:example.com","displayname":"Alice","avatar_url":"mxc://..."},
+      ],


One weird corner with the current spec: If the room doesn't have a name set and the user is invited/knock to the room, we don't have any heroes to provide here. This is because heroes membership isn't included in the stripped state that the server receives when they are invited/knocked over federation.

Related to matrix-org/matrix-spec#380

This was solved for partial-joins via MSC3943

Not necessarily a blocker but just calling out something that won't work completely until that spec issue is solved.

@kfiven

# Synapse 1.114.0 (2024-09-02) This release enables support for [MSC4186](matrix-org/matrix-spec-proposals#4186) — Simplified Sliding Sync. This allows using the upcoming releases of the Element X mobile apps without having to run a Sliding Sync Proxy. ### Features - Enable native sliding sync support ([MSC3575](matrix-org/matrix-spec-proposals#3575) and [MSC4186](matrix-org/matrix-spec-proposals#4186)) by default. ([\#17648](element-hq/synapse#17648)) # Synapse 1.114.0rc3 (2024-08-30) ### Bugfixes - Fix regression in v1.114.0rc2 that caused workers to fail to start. ([\#17626](element-hq/synapse#17626)) # Synapse 1.114.0rc2 (2024-08-30) ### Features - Improve cross-signing upload when using [MSC3861](matrix-org/matrix-spec-proposals#3861) to use a custom UIA flow stage, with web fallback support. ([\#17509](element-hq/synapse#17509)) - Make `hash_password` script accept password input from stdin. ([\#17608](element-hq/synapse#17608)) ### Bugfixes - Fix hierarchy returning 403 when room is accessible through federation. Contributed by Krishan (@kfiven). ([\#17194](element-hq/synapse#17194)) - Fix content-length on federation `/thumbnail` responses. ([\#17532](element-hq/synapse#17532)) - Fix authenticated media responses using a wrong limit when following redirects over federation. ([\#17543](element-hq/synapse#17543)) ### Internal Changes - MSC3861: load the issuer and account management URLs from OIDC discovery. ([\#17407](element-hq/synapse#17407)) - Refactor sliding sync class into multiple files. ([\#17595](element-hq/synapse#17595)) - Store sliding sync per-connection state in the database. ([\#17599](element-hq/synapse#17599)) - Make the sliding sync `PerConnectionState` class immutable. ([\#17600](element-hq/synapse#17600)) - Add support to `@tag_args` for standalone functions. ([\#17604](element-hq/synapse#17604)) - Speed up incremental syncs in sliding sync by adding some more caching. ([\#17606](element-hq/synapse#17606)) - Always return the user's own read receipts in sliding sync. ([\#17617](element-hq/synapse#17617)) - Replace `isort` and `black` with `ruff`. ([\#17620](element-hq/synapse#17620)) - Refactor sliding sync code to move room list logic out into a separate class. ([\#17622](element-hq/synapse#17622)) ### Updates to locked dependencies * Bump attrs from 23.2.0 to 24.2.0. ([\#17609](element-hq/synapse#17609)) * Bump cryptography from 42.0.8 to 43.0.0. ([\#17584](element-hq/synapse#17584)) * Bump phonenumbers from 8.13.43 to 8.13.44. ([\#17610](element-hq/synapse#17610)) * Bump pygithub from 2.3.0 to 2.4.0. ([\#17612](element-hq/synapse#17612)) * Bump pyyaml from 6.0.1 to 6.0.2. ([\#17611](element-hq/synapse#17611)) * Bump sentry-sdk from 2.12.0 to 2.13.0. ([\#17585](element-hq/synapse#17585)) * Bump serde from 1.0.206 to 1.0.208. ([\#17581](element-hq/synapse#17581)) * Bump serde from 1.0.208 to 1.0.209. ([\#17613](element-hq/synapse#17613)) * Bump serde_json from 1.0.124 to 1.0.125. ([\#17582](element-hq/synapse#17582)) * Bump serde_json from 1.0.125 to 1.0.127. ([\#17614](element-hq/synapse#17614)) * Bump types-jsonschema from 4.23.0.20240712 to 4.23.0.20240813. ([\#17583](element-hq/synapse#17583)) * Bump types-setuptools from 71.1.0.20240726 to 71.1.0.20240818. ([\#17586](element-hq/synapse#17586)) # Synapse 1.114.0rc1 (2024-08-20) ### Features - Add a flag to `/versions`, `org.matrix.simplified_msc3575`, to indicate whether experimental sliding sync support has been enabled. ([\#17571](element-hq/synapse#17571)) - Handle changes in `timeline_limit` in experimental sliding sync. ([\#17579](element-hq/synapse#17579)) - Correctly track read receipts that should be sent down in experimental sliding sync. ([\#17575](element-hq/synapse#17575), [\#17589](element-hq/synapse#17589), [\#17592](element-hq/synapse#17592)) ### Bugfixes - Start handlers for new media endpoints when media resource configured. ([\#17483](element-hq/synapse#17483)) - Fix timeline ordering (using `stream_ordering` instead of topological ordering) in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17510](element-hq/synapse#17510)) - Fix experimental sliding sync implementation to remember any updates in rooms that were not sent down immediately. ([\#17535](element-hq/synapse#17535)) - Better exclude partially stated rooms if we must await full state in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17538](element-hq/synapse#17538)) - Handle lower-case http headers in `_Mulitpart_Parser_Protocol`. ([\#17545](element-hq/synapse#17545)) - Fix fetching federation signing keys from servers that omit `old_verify_keys`. Contributed by @tulir @ Beeper. ([\#17568](element-hq/synapse#17568)) - Fix bug where we would respond with an error when a remote server asked for media that had a length of 0, using the new multipart federation media endpoint. ([\#17570](element-hq/synapse#17570)) ### Improved Documentation - Clarify default behaviour of the [`auto_accept_invites.worker_to_run_on`](https://element-hq.github.io/synapse/develop/usage/configuration/config_documentation.html#auto-accept-invites) option. ([\#17515](element-hq/synapse#17515)) - Improve docstrings for profile methods. ([\#17559](element-hq/synapse#17559)) ### Internal Changes - Add more tracing to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17514](element-hq/synapse#17514)) - Fixup comment in sliding sync implementation. ([\#17531](element-hq/synapse#17531)) - Replace override of deprecated method `HTTPAdapter.get_connection` with `get_connection_with_tls_context`. ([\#17536](element-hq/synapse#17536)) - Fix performance of device lists in `/key/changes` and sliding sync. ([\#17537](element-hq/synapse#17537), [\#17548](element-hq/synapse#17548)) - Bump setuptools from 67.6.0 to 72.1.0. ([\#17542](element-hq/synapse#17542)) - Add a utility function for generating random event IDs. ([\#17557](element-hq/synapse#17557)) - Speed up responding to media requests. ([\#17558](element-hq/synapse#17558), [\#17561](element-hq/synapse#17561), [\#17564](element-hq/synapse#17564), [\#17566](element-hq/synapse#17566), [\#17567](element-hq/synapse#17567), [\#17569](element-hq/synapse#17569)) - Test github token before running release script steps. ([\#17562](element-hq/synapse#17562)) - Reduce log spam of multipart files. ([\#17563](element-hq/synapse#17563)) - Refactor per-connection state in experimental sliding sync handler. ([\#17574](element-hq/synapse#17574)) - Add histogram metrics for sliding sync processing time. ([\#17593](element-hq/synapse#17593)) ### Updates to locked dependencies * Bump bytes from 1.6.1 to 1.7.1. ([\#17526](element-hq/synapse#17526)) * Bump lxml from 5.2.2 to 5.3.0. ([\#17550](element-hq/synapse#17550)) * Bump phonenumbers from 8.13.42 to 8.13.43. ([\#17551](element-hq/synapse#17551)) * Bump regex from 1.10.5 to 1.10.6. ([\#17527](element-hq/synapse#17527)) * Bump sentry-sdk from 2.10.0 to 2.12.0. ([\#17553](element-hq/synapse#17553)) * Bump serde from 1.0.204 to 1.0.206. ([\#17556](element-hq/synapse#17556)) * Bump serde_json from 1.0.122 to 1.0.124. ([\#17555](element-hq/synapse#17555)) * Bump sigstore/cosign-installer from 3.5.0 to 3.6.0. ([\#17549](element-hq/synapse#17549)) * Bump types-pyyaml from 6.0.12.20240311 to 6.0.12.20240808. ([\#17552](element-hq/synapse#17552)) * Bump types-requests from 2.31.0.20240406 to 2.32.0.20240712. ([\#17524](element-hq/synapse#17524)) # Synapse 1.113.0 (2024-08-13) No significant changes since 1.113.0rc1. # Synapse 1.113.0rc1 (2024-08-06) ### Features - Track which rooms have been sent to clients in the experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17447](element-hq/synapse#17447)) - Add Account Data extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17477](element-hq/synapse#17477)) - Add receipts extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17489](element-hq/synapse#17489)) - Add typing notification extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17505](element-hq/synapse#17505)) ### Bugfixes - Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to handle invite/knock rooms when filtering. ([\#17450](element-hq/synapse#17450)) - Fix a bug introduced in v1.110.0 which caused `/keys/query` to return incomplete results, leading to high network activity and CPU usage on Matrix clients. ([\#17499](element-hq/synapse#17499)) ### Improved Documentation - Update the [`allowed_local_3pids`](https://element-hq.github.io/synapse/v1.112/usage/configuration/config_documentation.html#allowed_local_3pids) config option's msisdn address to a working example. ([\#17476](element-hq/synapse#17476)) ### Internal Changes - Change sliding sync to use their own token format in preparation for storing per-connection state. ([\#17452](element-hq/synapse#17452)) - Ensure we don't send down negative `bump_stamp` in experimental sliding sync endpoint. ([\#17478](element-hq/synapse#17478)) - Do not send down empty room entries down experimental sliding sync endpoint. ([\#17479](element-hq/synapse#17479)) - Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`. ([\#17481](element-hq/synapse#17481), [\#17482](element-hq/synapse#17482)) - Add some opentracing tags and logging to the experimental sliding sync implementation. ([\#17501](element-hq/synapse#17501)) - Split and move Sliding Sync tests so we have some more sane test file sizes. ([\#17504](element-hq/synapse#17504)) - Update the `limited` field description in the Sliding Sync response to accurately describe what it actually represents. ([\#17507](element-hq/synapse#17507)) - Easier to understand `timeline` assertions in Sliding Sync tests. ([\#17511](element-hq/synapse#17511)) - Reset the sliding sync connection if we don't recognize the per-connection state position. ([\#17529](element-hq/synapse#17529)) ### Updates to locked dependencies * Bump bcrypt from 4.1.3 to 4.2.0. ([\#17495](element-hq/synapse#17495)) * Bump black from 24.4.2 to 24.8.0. ([\#17522](element-hq/synapse#17522)) * Bump phonenumbers from 8.13.39 to 8.13.42. ([\#17521](element-hq/synapse#17521)) * Bump ruff from 0.5.4 to 0.5.5. ([\#17494](element-hq/synapse#17494)) * Bump serde_json from 1.0.120 to 1.0.121. ([\#17493](element-hq/synapse#17493)) * Bump serde_json from 1.0.121 to 1.0.122. ([\#17525](element-hq/synapse#17525)) * Bump towncrier from 23.11.0 to 24.7.1. ([\#17523](element-hq/synapse#17523)) * Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722. ([\#17496](element-hq/synapse#17496)) * Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726. ([\#17497](element-hq/synapse#17497))

deepbluev7 · 2024-09-19T13:12:16Z

proposals/4186-simplified-sliding-sync.md

+        // Flag which only returns rooms which have an `m.room.encryption` state event. If unset,
+        // both encrypted and unencrypted rooms are returned. If false, only unencrypted rooms
+        // are returned. If true, only encrypted rooms are returned.
+        "is_invite": true|false|null,


How will knocks and left rooms be handled? Maybe this should be a string and be the current membership of the user?

MadLittleMods · 2024-09-19T16:08:04Z

proposals/4186-simplified-sliding-sync.md

+      // Same as in sync v2. Indicates whether there are more events to fetch than those in the timeline.
+      "limited:" true|null,


Clarify what limited means when complicating it with gaps, newly joined, newly joined a room where you can't see the history, etc

Sync v2 in Synapse is a bit crazy with the logic probably because the spec isn't clear.

The existing definition seems accurate "Indicates whether there are more events to fetch than those in the timeline." Could maybe clarify that you can be limited if there is timeline gap and you stop at the gap and say limited: true so the client paginates over the gap themselves. And maybe also need to say that being newly joined doesn't factor in.

MadLittleMods · 2024-09-19T16:52:44Z

proposals/4186-simplified-sliding-sync.md

+      // Same as in sync v2. Indicates whether there are more events to fetch than those in the timeline.
+      "limited:" true|null,


Clarify when to add m.relations in unsigned (bundled_aggregation logic).

Currently, it's done in Sync v2 and Sliding Sync in Synapse when limited: true but @erikjohnston probably has the right inclination to not do it given that sync is returning new events and any relations to those events would come after it by their nature (and already be in the response or the next one).

@erikjohnston unearthed some more context in MSC2675 and relevant Synapse PR

MadLittleMods · 2024-09-19T19:27:55Z

proposals/4186-simplified-sliding-sync.md

+Care must be taken, as with sync v2, to ensure that only the data that the user is authorized to see is returned in the
+response.


This should be expanded to explain what this means.

In terms of state:

join: Access to all of the room state.

invite/knock: Can only derive state from the stripped state events in the event itself (invite_room_state, knock_room_state)

leave/ban: Depends on the previous membership.

If the previous membership was join, access to the state at the time of the leave/ban (historical state)

If the previous membership was invite, fallback to the stripped state on the invite event

If the previous membership was leave/ban, repeat this logic

If no previous membership, no access to anything.

In terms of timeline events, this probably depends solely on m.room.history_visibility

MadLittleMods · 2024-09-19T19:34:27Z

proposals/4186-simplified-sliding-sync.md

+## Timeline event trickling
+
+If the `timeline_limit` is increased then the server will send down historic data (c.f. "Room config changes"), which
+allows the clients to easily preload more history in recent rooms.
+
+This mechanism is fiddly to implement, and ends up resending down events that we have previously sent to the client.
+
+A simpler alternative is to use `/messages` to fetch the history. This has two main problems: 1) clients generally want
+to preload history for multiple rooms at once, and 2) `/messages` can be slow if it tries to backfill over federation.
+
+We could implement a bulk `/messages` endpoint, where the client would specify multiple rooms and `prev_batch` tokens.
+We can also add a flag to disable attempting to backfill over pagination (to match the behaviour of the sync timeline).


Please kill unstable_expanded_timeline with one of these alternatives. It was implemented in Simplified Sliding Sync solely to match what the Sliding Sync proxy did. And the proxy behavior was a bug not a feature (self-described)

Previous conversation:

Sliding Sync: Handle room_subscriptions that increase the timeline_limit element-hq/synapse#17503 (comment)

Sliding Sync: Handle timeline limit changes (take 2) element-hq/synapse#17579 (comment)

MadLittleMods · 2024-09-19T19:51:04Z

proposals/4186-simplified-sliding-sync.md

+  // Aggregated rooms from lists and room subscriptions. There will be one entry per room, even if
+  // the room appears in multiple lists and/or room subscriptions.
+  "rooms": {


Clarify which rooms should be showing up.

We want to return every room where the user has membership except rooms that the user has left themselves; unless "newly left" (left since the last incremental sync pos token) because we want everything that's still relevant to the user. We include "newly left" rooms because the last event that the user should see is their own leave event.

A leave != kick (leave events where the sender is not the same user). Kicks should always be included.

To say it another way, here is a list:

join

invite

knock

ban

Kicks (leave where the sender doesn't match the state_key)

Newly left rooms (leave membership that happened since the last incremental sync pos token)

Should you only include ban if there is a previous join, invite, or knock? :) (Maybe scope creep.)

I also had that in mind but I'm not sure (could go either way). It's kinda the same as an unsolicited invite (optionally followed by a kick/ban).

Ideally, we probably wouldn't show those kind of ban

MadLittleMods · 2024-09-19T20:56:31Z

proposals/4186-simplified-sliding-sync.md

+  "room_subscriptions": {
+    // The key is the room to subscribe to.
+    "!foo:example.com": {


Re-adding this discussion from the other MSC3575:

What should happen if you try to subscribe to a room you are not part of (or not allowed)?

The easy option would be to ignore it but then clients have no feedback about what they did wrong.

Perhaps entries in the rooms map could also have an errcode/error field. We could also just blow up the whole request with an error and point out which room_id is wrong although that isn't very machine-readable.

Can you subscribe to a public world_readable room that you're not part of?

Hywan · 2024-09-24T08:12:45Z

proposals/4186-simplified-sliding-sync.md

+      // Whether the room is a DM room.
+      "is_dm": true|null,
+
+      // An opaque integer that can be used to sort the rooms by "Bump Stamp"


Suggested change

// An opaque integer that can be used to sort the rooms by "Bump Stamp"

// An opaque unsigned 64-bits integer that can be used to sort the rooms by "Bump Stamp"

We may also precise that bump_stamp has a total order.

ara4n · 2024-09-24T10:20:57Z

proposals/4186-simplified-sliding-sync.md

+This is particularly problematic if the user gets "state reset" out of the room, where the server has no mechanism to
+indicate to the client that the user has effectively left the room (the server has no leave event to return).
+
+We may want to allow special entries in the `required_state` list of the form


Surely this isn't a nice-to-have but a requirement, otherwise we end up back in matrix-org/matrix-spec#1209

Hywan · 2024-09-25T08:14:36Z

proposals/4186-simplified-sliding-sync.md

+      "bump_stamp": 1,
+
+      // These are the same as sync v2.
+      "joined_count": 1,


Should be precised that joined_count is given only if the room has been joined. For invite room, this value will be missing.

This test was failing since the migration from the sliding sync proxy to Synapse. This patch fixes the test. The failing part was: ```rust assert_eq!(notification.joined_members_count, 1); ``` This patch changes the value from 1 to 0. Indeed, Synapse doesn't share this data for the sake of privacy because the room is not joined. A comment has been made on MSC4186 to precise this behaviour: matrix-org/matrix-spec-proposals#4186 (comment). Moreover, this test was asserting a bug (which is alright), but now a bug report has been made. The patch contains the link to this bug report. The code has been a bit rewritten to make it simpler, and more comments have been added.

Hywan · 2024-09-25T12:30:16Z

proposals/4186-simplified-sliding-sync.md

+      },
+
+      // The maximum number of timeline events to return per response.
+      "timeline_limit": 10,


MSC3575 (at least its implementation inside the sliding sync proxy) allowed timeline_limit to be null. This is NOT the case with MSC4186 (as implemented inside Synapse). I believe the MSC should clarify this value is required.

This test was failing since the migration from the sliding sync proxy to Synapse. This patch fixes the test. The failing part was: ```rust assert_eq!(notification.joined_members_count, 1); ``` This patch changes the value from 1 to 0. Indeed, Synapse doesn't share this data for the sake of privacy because the room is not joined. A comment has been made on MSC4186 to precise this behaviour: matrix-org/matrix-spec-proposals#4186 (comment). Moreover, this test was asserting a bug (which is alright), but now a bug report has been made. The patch contains the link to this bug report. The code has been a bit rewritten to make it simpler, and more comments have been added.

jmartinesp · 2024-10-16T15:03:54Z

I've made a suggestion in Synapse to implement a membership field for Room objects. This would simplify the logic for the clients a lot, avoiding checks and assumptions, and make sure they'll have consistent behaviours since it'll be the server who explicitly sets this value.

jmartinesp · 2024-10-17T06:53:09Z

proposals/4186-simplified-sliding-sync.md

+      // For invites this is the stripped state of the room at the time of invite
+      "invite_state": [ .. ],
+
+      // For knocks this is the stripped state of the room at time of knock
+      "knock_state": [ .. ],


As mentioned in a comment below it would be super useful to have a separate field indicating the user's membership state in a room instead of relying on some rules around required_state and invite_state and finding the membership event in the client-side.

Also note the current simplified sliding sync version implementation in Synapse uses a single invite_state array instead of separate ones for invite and knock. I think it was discussed to rename it to stripped_state or something along the lines?

HarHarLinks · 2024-11-02T14:02:55Z

proposals/4186-simplified-sliding-sync.md

+
+# Unstable prefix
+
+The unstable URL for simplified sliding sync is `/org.matrix.simplified_msc3575/sync`. The flag in `/versions` is


Suggested change

The unstable URL for simplified sliding sync is `/org.matrix.simplified_msc3575/sync`. The flag in `/versions` is

The unstable URL for simplified sliding sync is `/_matrix/client/unstable/org.matrix.simplified_msc3575/sync`. The flag in `/_matrix/client/versions` is

HarHarLinks · 2024-11-02T14:18:45Z

proposals/4186-simplified-sliding-sync.md

+**TODO** explain how these interact with the room lists, this is the same as in
+[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575)
+
+## Request format


which of these are required and which optional?

MadLittleMods · 2024-11-19T21:09:38Z

proposals/4186-simplified-sliding-sync.md

+      // event _except_ for @alice:example.com, and include every other state event.
+      // In addition, ["*", "*"], ["m.space.child", "*"] is an error, the m.space.child filter is not
+      // required as it would have been returned anyway.
+      "required_state": [ ... ],


Clarify that $LAZY loading room members should also include invite, ban, kick, membership targets alongside the senders of any events in the timeline.

This ensures that we send down membership updates whenever we see it change in the timeline. This allows clients to cache the membership list for as long as it doesn't get a gappy sync, but still ensures for large gaps the server doesn't need to send down all membership changes.

Reference: element-hq/synapse#17929

Simplified sliding sync

8cc34df

erikjohnston force-pushed the erikj/sss branch from 806e455 to 8cc34df Compare August 30, 2024 09:25

erikjohnston added 2 commits August 30, 2024 10:52

Add example usage

456db3b

Add security and unstable prefix

6fe4ba7

erikjohnston marked this pull request as ready for review August 30, 2024 09:59

Gnuxie reviewed Aug 30, 2024

View reviewed changes

anoadragon453 added feature Suggestion for a significant extension which needs considerable consideration proposal A matrix spec change proposal kind:core MSC which is critical to the protocol's success labels Aug 30, 2024

uhoreg added the client-server Client-Server API label Aug 30, 2024

turt2live added the needs-implementation This MSC does not have a qualifying implementation for the SCT to review. The MSC cannot enter FCP. label Aug 30, 2024

turt2live reviewed Aug 30, 2024

View reviewed changes

proposals/4186-simplified-sliding-sync.md

Copy link

Member

turt2live Aug 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation requirements:

Client (ideally multiple)

Server

Gnuxie reviewed Aug 30, 2024

View reviewed changes

erebion mentioned this pull request Sep 3, 2024

Sliding Sync does not seem to work element-hq/synapse#17651

Closed

Johennes reviewed Sep 3, 2024

View reviewed changes

clokep reviewed Sep 3, 2024

View reviewed changes

MadLittleMods reviewed Sep 5, 2024

View reviewed changes

proposals/4186-simplified-sliding-sync.md Show resolved Hide resolved

MadLittleMods reviewed Sep 5, 2024

View reviewed changes

MadLittleMods mentioned this pull request Sep 5, 2024

MSC3575: Sliding Sync (aka Sync v3) #3575

Open

MadLittleMods reviewed Sep 5, 2024

View reviewed changes

This was referenced Sep 9, 2024

fix(sdk): Mark tracked users as dirty when the SS connection is reset. matrix-org/matrix-rust-sdk#3965

Merged

client-api: Implement MSC4186 ruma/ruma#1907

Merged

kfiven mentioned this pull request Sep 14, 2024

Sliding sync cinnyapp/cinny#1436

Open

tcpipuk mentioned this pull request Sep 19, 2024

accuracy of the nginx routing rules with the recent changes in auth media and sliding sync? tcpipuk/tcpipuk.github.io#20

Open

deepbluev7 reviewed Sep 19, 2024

View reviewed changes

MadLittleMods reviewed Sep 19, 2024

View reviewed changes

Hywan reviewed Sep 24, 2024

View reviewed changes

ara4n reviewed Sep 24, 2024

View reviewed changes

Hywan reviewed Sep 25, 2024

View reviewed changes

Hywan mentioned this pull request Sep 25, 2024

Sliding sync returns initial: false for a first response element-hq/synapse#17754

Closed

Hywan mentioned this pull request Sep 25, 2024

test(integration): Enable test_notification matrix-org/matrix-rust-sdk#4029

Merged

Hywan reviewed Sep 25, 2024

View reviewed changes

gnunicorn mentioned this pull request Oct 6, 2024

Implement Simplified Sliding Sync acterglobal/a3#2225

Open

MadLittleMods mentioned this pull request Oct 16, 2024

Native sliding sync: add membership field to rooms element-hq/synapse#17838

Open

jmartinesp reviewed Oct 17, 2024

View reviewed changes

Gnuxie mentioned this pull request Oct 29, 2024

MSC4222: Adding state_after to sync v2 #4222

Open

HarHarLinks reviewed Nov 2, 2024

View reviewed changes

MadLittleMods reviewed Nov 19, 2024

View reviewed changes


		The core differences between sync v2 and simplified sliding sync are:

		- The server initially only sends the most recent N rooms to the client (where N is specified by the lcient), which then

		- Time from launch to confident usability should be as low as possible.
		- Time from login on existing accounts to usability should be as low as possible.

	limit, etc). The order of rooms is always done based on when the server received the most recent event for the room.
	limit, etc). The order of rooms is always done based on the arrival time of the most recent event for the room on the homeserver.

	// The room name, if one exists. Only sent initially and when it changes.
	// The room name (`m.room.name` -> `content.name`), if one exists. Only sent initially and when it changes.

		// Same as in sync v2. Indicates whether there are more events to fetch than those in the timeline.
		"limited:" true\|null,

		Care must be taken, as with sync v2, to ensure that only the data that the user is authorized to see is returned in the
		response.

	// An opaque integer that can be used to sort the rooms by "Bump Stamp"
	// An opaque unsigned 64-bits integer that can be used to sort the rooms by "Bump Stamp"


		# Unstable prefix

		The unstable URL for simplified sliding sync is `/org.matrix.simplified_msc3575/sync`. The flag in `/versions` is

MSC4186: Simplified Sliding Sync #4186

Are you sure you want to change the base?

MSC4186: Simplified Sliding Sync #4186

Conversation

erikjohnston commented Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Gnuxie Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gnuxie Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmartinesp commented Oct 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

erikjohnston commented Aug 30, 2024 •

edited

Loading

Gnuxie Aug 30, 2024 •

edited

Loading

Gnuxie Aug 30, 2024 •

edited

Loading

MadLittleMods Sep 5, 2024 •

edited

Loading

MadLittleMods Sep 19, 2024 •

edited

Loading

MadLittleMods Sep 19, 2024 •

edited

Loading

MadLittleMods Nov 19, 2024 •

edited

Loading