Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Speed up updating state in large rooms #15971

Merged
merged 3 commits into from
Jul 20, 2023
Merged

Speed up updating state in large rooms #15971

merged 3 commits into from
Jul 20, 2023

Conversation

erikjohnston
Copy link
Member

@erikjohnston erikjohnston commented Jul 20, 2023 •
edited
Loading

We carefully do a lot of clever tricks to avoid having to calculate the joined hosts by pulling out the full state... but we always pull out the full state anyway.

The first commit is just a move of some functions from the data store to the storage controller, the second commit is where we actually change the logic.

@erikjohnston erikjohnston marked this pull request as ready for review July 20, 2023 13:29
@erikjohnston erikjohnston requested a review from a team as a code owner July 20, 2023 13:29
@reivilibre
Copy link
Contributor

TIL about the StateCacheEntry stuff :)

@erikjohnston erikjohnston merged commit fc1e534 into develop Jul 20, 2023
@erikjohnston erikjohnston deleted the erikj/dont_pull_out_state branch July 20, 2023 14:51
yingziwu added a commit to yingziwu/synapse that referenced this pull request Aug 2, 2023
@yingziwu
Merge tag 'v1.89.0' into develop
ce16c05 
No significant changes since 1.89.0rc1.

- Add Unix Socket support for HTTP Replication Listeners. [Document and provide usage instructions](https://matrix-org.github.io/synapse/v1.89/usage/configuration/config_documentation.html#listeners) for utilizing Unix sockets in Synapse. Contributed by Jason Little. ([\matrix-org#15708](matrix-org#15708), [\matrix-org#15924](matrix-org#15924))
- Allow `+` in Matrix IDs, per [MSC4009](matrix-org/matrix-spec-proposals#4009). ([\matrix-org#15911](matrix-org#15911))
- Support room version 11 from [MSC3820](matrix-org/matrix-spec-proposals#3820). ([\matrix-org#15912](matrix-org#15912))
- Allow configuring the set of workers to proxy outbound federation traffic through via `outbound_federation_restricted_to`. ([\matrix-org#15913](matrix-org#15913), [\matrix-org#15969](matrix-org#15969))
- Implement [MSC3814](matrix-org/matrix-spec-proposals#3814), dehydrated devices v2/shrivelled sessions and move [MSC2697](matrix-org/matrix-spec-proposals#2697) behind a config flag. Contributed by Nico from Famedly, H-Shay and poljar. ([\matrix-org#15929](matrix-org#15929))

- Fix a long-standing bug where remote invites weren't correctly pushed. ([\matrix-org#15820](matrix-org#15820))
- Fix background schema updates failing over a large upgrade gap. ([\matrix-org#15887](matrix-org#15887))
- Fix a bug introduced in 1.86.0 where Synapse starting with an empty `experimental_features` configuration setting. ([\matrix-org#15925](matrix-org#15925))
- Fixed deploy annotations in the provided Grafana dashboard config, so that it shows for any homeserver and not just matrix.org. Contributed by @wrjlewis. ([\matrix-org#15957](matrix-org#15957))
- Ensure a long state res does not starve CPU by occasionally yielding to the reactor. ([\matrix-org#15960](matrix-org#15960))
- Properly handle redactions of creation events. ([\matrix-org#15973](matrix-org#15973))
- Fix a bug where resyncing stale device lists could block responding to federation transactions, and thus delay receiving new data from the remote server. ([\matrix-org#15975](matrix-org#15975))

- Better clarify how to run a worker instance (pass both configs). ([\matrix-org#15921](matrix-org#15921))
- Improve [the documentation](https://matrix-org.github.io/synapse/v1.89/admin_api/user_admin_api.html#login-as-a-user) for the login as a user admin API. ([\matrix-org#15938](matrix-org#15938))
- Fix broken Arch Linux package link. Contributed by @SnipeXandrej. ([\matrix-org#15981](matrix-org#15981))

- Remove support for calling the `/register` endpoint with an unspecced `user` property for application services. ([\matrix-org#15928](matrix-org#15928))

- Mark `get_user_in_directory` private since it is only used in tests. Also remove the cache from it. ([\matrix-org#15884](matrix-org#15884))
- Document which Python version runs on a given Linux distribution so we can more easily clean up later. ([\matrix-org#15909](matrix-org#15909))
- Add details to warning in log when we fail to fetch an alias. ([\matrix-org#15922](matrix-org#15922))
- Remove unneeded `__init__`. ([\matrix-org#15926](matrix-org#15926))
- Fix bug with read/write lock implementation. This is currently unused so has no observable effects. ([\matrix-org#15933](matrix-org#15933), [\matrix-org#15958](matrix-org#15958))
- Unbreak the nix development environment by pinning the Rust version to 1.70.0. ([\matrix-org#15940](matrix-org#15940))
- Update presence metrics to differentiate remote vs local users. ([\matrix-org#15952](matrix-org#15952))
- Stop reading from column `user_id` of table `profiles`. ([\matrix-org#15955](matrix-org#15955))
- Build packages for Debian Trixie. ([\matrix-org#15961](matrix-org#15961))
- Reduce the amount of state we pull out. ([\matrix-org#15968](matrix-org#15968))
- Speed up updating state in large rooms. ([\matrix-org#15971](matrix-org#15971))

* Bump anyhow from 1.0.71 to 1.0.72. ([\matrix-org#15949](matrix-org#15949))
* Bump click from 8.1.3 to 8.1.6. ([\matrix-org#15984](matrix-org#15984))
* Bump cryptography from 41.0.1 to 41.0.2. ([\matrix-org#15943](matrix-org#15943))
* Bump jsonschema from 4.17.3 to 4.18.3. ([\matrix-org#15948](matrix-org#15948))
* Bump pillow from 9.4.0 to 10.0.0. ([\matrix-org#15986](matrix-org#15986))
* Bump prometheus-client from 0.17.0 to 0.17.1. ([\matrix-org#15945](matrix-org#15945))
* Bump pydantic from 1.10.10 to 1.10.11. ([\matrix-org#15946](matrix-org#15946))
* Bump pygithub from 1.58.2 to 1.59.0. ([\matrix-org#15834](matrix-org#15834))
* Bump pyo3-log from 0.8.2 to 0.8.3. ([\matrix-org#15951](matrix-org#15951))
* Bump sentry-sdk from 1.26.0 to 1.28.1. ([\matrix-org#15985](matrix-org#15985))
* Bump serde_json from 1.0.100 to 1.0.103. ([\matrix-org#15950](matrix-org#15950))
* Bump types-pillow from 9.5.0.4 to 10.0.0.1. ([\matrix-org#15932](matrix-org#15932))
* Bump types-requests from 2.31.0.1 to 2.31.0.2. ([\matrix-org#15983](matrix-org#15983))
* Bump typing-extensions from 4.5.0 to 4.7.1. ([\matrix-org#15947](matrix-org#15947))
Fizzadar added a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 29, 2023
@Fizzadar
Merge tag 'v1.89.0' into merge-1.89
13e9b31 
No significant changes since 1.89.0rc1.

- Add Unix Socket support for HTTP Replication Listeners. [Document and provide usage instructions](https://matrix-org.github.io/synapse/v1.89/usage/configuration/config_documentation.html#listeners) for utilizing Unix sockets in Synapse. Contributed by Jason Little. ([\matrix-org#15708](matrix-org#15708), [\matrix-org#15924](matrix-org#15924))
- Allow `+` in Matrix IDs, per [MSC4009](matrix-org/matrix-spec-proposals#4009). ([\matrix-org#15911](matrix-org#15911))
- Support room version 11 from [MSC3820](matrix-org/matrix-spec-proposals#3820). ([\matrix-org#15912](matrix-org#15912))
- Allow configuring the set of workers to proxy outbound federation traffic through via `outbound_federation_restricted_to`. ([\matrix-org#15913](matrix-org#15913), [\matrix-org#15969](matrix-org#15969))
- Implement [MSC3814](matrix-org/matrix-spec-proposals#3814), dehydrated devices v2/shrivelled sessions and move [MSC2697](matrix-org/matrix-spec-proposals#2697) behind a config flag. Contributed by Nico from Famedly, H-Shay and poljar. ([\matrix-org#15929](matrix-org#15929))

- Fix a long-standing bug where remote invites weren't correctly pushed. ([\matrix-org#15820](matrix-org#15820))
- Fix background schema updates failing over a large upgrade gap. ([\matrix-org#15887](matrix-org#15887))
- Fix a bug introduced in 1.86.0 where Synapse starting with an empty `experimental_features` configuration setting. ([\matrix-org#15925](matrix-org#15925))
- Fixed deploy annotations in the provided Grafana dashboard config, so that it shows for any homeserver and not just matrix.org. Contributed by @wrjlewis. ([\matrix-org#15957](matrix-org#15957))
- Ensure a long state res does not starve CPU by occasionally yielding to the reactor. ([\matrix-org#15960](matrix-org#15960))
- Properly handle redactions of creation events. ([\matrix-org#15973](matrix-org#15973))
- Fix a bug where resyncing stale device lists could block responding to federation transactions, and thus delay receiving new data from the remote server. ([\matrix-org#15975](matrix-org#15975))

- Better clarify how to run a worker instance (pass both configs). ([\matrix-org#15921](matrix-org#15921))
- Improve [the documentation](https://matrix-org.github.io/synapse/v1.89/admin_api/user_admin_api.html#login-as-a-user) for the login as a user admin API. ([\matrix-org#15938](matrix-org#15938))
- Fix broken Arch Linux package link. Contributed by @SnipeXandrej. ([\matrix-org#15981](matrix-org#15981))

- Remove support for calling the `/register` endpoint with an unspecced `user` property for application services. ([\matrix-org#15928](matrix-org#15928))

- Mark `get_user_in_directory` private since it is only used in tests. Also remove the cache from it. ([\matrix-org#15884](matrix-org#15884))
- Document which Python version runs on a given Linux distribution so we can more easily clean up later. ([\matrix-org#15909](matrix-org#15909))
- Add details to warning in log when we fail to fetch an alias. ([\matrix-org#15922](matrix-org#15922))
- Remove unneeded `__init__`. ([\matrix-org#15926](matrix-org#15926))
- Fix bug with read/write lock implementation. This is currently unused so has no observable effects. ([\matrix-org#15933](matrix-org#15933), [\matrix-org#15958](matrix-org#15958))
- Unbreak the nix development environment by pinning the Rust version to 1.70.0. ([\matrix-org#15940](matrix-org#15940))
- Update presence metrics to differentiate remote vs local users. ([\matrix-org#15952](matrix-org#15952))
- Stop reading from column `user_id` of table `profiles`. ([\matrix-org#15955](matrix-org#15955))
- Build packages for Debian Trixie. ([\matrix-org#15961](matrix-org#15961))
- Reduce the amount of state we pull out. ([\matrix-org#15968](matrix-org#15968))
- Speed up updating state in large rooms. ([\matrix-org#15971](matrix-org#15971))

* Bump anyhow from 1.0.71 to 1.0.72. ([\matrix-org#15949](matrix-org#15949))
* Bump click from 8.1.3 to 8.1.6. ([\matrix-org#15984](matrix-org#15984))
* Bump cryptography from 41.0.1 to 41.0.2. ([\matrix-org#15943](matrix-org#15943))
* Bump jsonschema from 4.17.3 to 4.18.3. ([\matrix-org#15948](matrix-org#15948))
* Bump pillow from 9.4.0 to 10.0.0. ([\matrix-org#15986](matrix-org#15986))
* Bump prometheus-client from 0.17.0 to 0.17.1. ([\matrix-org#15945](matrix-org#15945))
* Bump pydantic from 1.10.10 to 1.10.11. ([\matrix-org#15946](matrix-org#15946))
* Bump pygithub from 1.58.2 to 1.59.0. ([\matrix-org#15834](matrix-org#15834))
* Bump pyo3-log from 0.8.2 to 0.8.3. ([\matrix-org#15951](matrix-org#15951))
* Bump sentry-sdk from 1.26.0 to 1.28.1. ([\matrix-org#15985](matrix-org#15985))
* Bump serde_json from 1.0.100 to 1.0.103. ([\matrix-org#15950](matrix-org#15950))
* Bump types-pillow from 9.5.0.4 to 10.0.0.1. ([\matrix-org#15932](matrix-org#15932))
* Bump types-requests from 2.31.0.1 to 2.31.0.2. ([\matrix-org#15983](matrix-org#15983))
* Bump typing-extensions from 4.5.0 to 4.7.1. ([\matrix-org#15947](matrix-org#15947))

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmTI2e4ACgkQkD7OEIo5
# 3t2x1RAAohu1Rmjv0mOqFR4P1YZpA5RFbYajcyq77n/ciDKSM1dqBelONqKOq2A9
# uGbVNm6rC+EFwIl5MF5TrFdsDQHvGcRgW6NpQDZ+uIUOYizjZH1g37BoNPLlGYQx
# fmKG7/XqdWhSc5tHN9HsRHyHKmsndebjXoUCPKmieGZa1GLXvGwrNkWQlEpwd9Qu
# mj3uewJxLFGgIIAOiplJ4UO8FaCbMD+By27hSiWtVsLT6pyav4HC2P8RQD1iv0jW
# OXNHvEWyqfBPlsPOkCD4nQZrmZqa5GWLYfBm8zFgIBxNy+e33C07L4bO+QdCE86v
# /SUKug/0nsp66jSZst1fM/M2ssXvjU+LNO9fqonOCZ4TiJ4i/yoa8AvmcAg5hy7C
# HR9IBp9cMrQ2u1y2/knxF657AGHxgXEltgw0PDvZHowqsqoSb+5HWl0zv1wnVjMa
# 2QYLKWPBk/AdlHkmC3S4/+gfVZVsT2RSBP3JUCbFyOqug9vXFvSGTfH07Lk4PDI3
# o5idBzumvyonsuC2ypkzlj49FAj21l/8DInxEpY9JcHdVncLWvu9gmLd+H7GY7H7
# ODa2gOynrsSGVH7IpOl6dpw/GH6R8ZlfHl87bFslOqVObBxquL/ODIoFOgld+MpT
# YYXp+0tW564mg+AYw3+eo44JTq0lKh7eyENP3SqKN/Z8ssQL97c=
# =Ar/g
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue Aug  1 11:09:50 2023 BST
# gpg:                using RSA key D79D3CA0B61429A8A760525A903ECE108A39DEDD
# gpg: key 903ECE108A39DEDD: new key but contains no user ID - skipped
# gpg: Total number processed: 1
# gpg:           w/o user IDs: 1
# gpg: Can't check signature: No public key

# Conflicts:
#	poetry.lock
#	synapse/http/site.py
#	synapse/storage/databases/main/roommember.py
Fizzadar pushed a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 29, 2023
@erikjohnston @Fizzadar
Speed up updating state in large rooms (matrix-org#15971)
b037e3f 
This should speed up updating state in rooms with lots of state.
Fizzadar pushed a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 31, 2023
@erikjohnston @Fizzadar
Speed up updating state in large rooms (matrix-org#15971)
5367fde 
This should speed up updating state in rooms with lots of state.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@reivilibre reivilibre reivilibre approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@erikjohnston @reivilibre