From 2278562d04b7751ed9e58a92876c602f33141a23 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 11:16:53 +0100 Subject: [PATCH 01/26] Update worker documentation with latest additions --- docs/workers.md | 338 +++++++++++++++++++++++------------------------- 1 file changed, 165 insertions(+), 173 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index f4cbbc040042..e55e653ad7cc 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -18,14 +18,22 @@ running PostgreSQL. ## Master/worker communication -The workers communicate with the master process via a Synapse-specific protocol -called 'replication' (analogous to MySQL- or Postgres-style database -replication) which feeds a stream of relevant data from the master to the -workers so they can be kept in sync with the master process and database state. +The processes communicate with each other via a Synapse-specific protocol called +'replication' (analogous to MySQL- or Postgres-style database replication) which +feeds streams of newly written data between processes so they can be kept in +sync with the database state. + +Additionally, processes may make HTTP requests to each other. Typically this is +used for operations which need to wait for a reply - such as sending an event. + +As of Synapse v1.13.0, it is possible to configure Synapse to send replication +via a [Redis pub/sub channel](https://redis.io/topics/pubsub), and is now the +recommended way of configuring replication. This is an alternative to the old +direct TCP connections to the master: rather than all the workers connecting to +the master, all the workers and the master connect to Redis, which relays +replication commands between processes. This can give a significant cpu saving +on the master and will be a prerequisite for upcoming performance improvements. -Additionally, workers may make HTTP requests to the master, to send information -in the other direction. Typically this is used for operations which need to -wait for a reply - such as sending an event. ## Configuration @@ -35,26 +43,26 @@ the correct worker, or to the main synapse instance. Note that this includes requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse proxy. -To enable workers, you need to add *two* replication listeners to the -main Synapse configuration file (`homeserver.yaml`). For example: +To enable workers, you need to add both a HTTP replication listener and redis +config to the main Synapse configuration file (`homeserver.yaml`). For example: ```yaml listeners: - # The TCP replication port - - port: 9092 - bind_address: '127.0.0.1' - type: replication - # The HTTP replication port - port: 9093 bind_address: '127.0.0.1' type: http resources: - names: [replication] + +redis: + enabled: true ``` -Under **no circumstances** should these replication API listeners be exposed to -the public internet; they have no authentication and are unencrypted. +See the sample config for the full documentation of each option. + +Under **no circumstances** should the replication API listener be exposed to the +public internet; it has no authentication and is unencrypted. You should then create a set of configs for the various worker processes. Each worker configuration file inherits the configuration of the main homeserver @@ -64,21 +72,21 @@ configuration; etc. You should minimise the number of overrides though to maintain a usable config. In the config file for each worker, you must specify the type of worker -application (`worker_app`). The currently available worker applications are -listed below. You must also specify the replication endpoints that it should -talk to on the main synapse process. `worker_replication_host` should specify -the host of the main synapse, `worker_replication_port` should point to the TCP -replication listener port and `worker_replication_http_port` should point to -the HTTP replication port. +application (`worker_app`), and you should specify a unqiue name for the worker +(`worker_name`). The currently available worker applications are listed below. +You must also specify the HTTP replication endpoint that it should talk to on +the main synapse process. `worker_replication_host` should specify the host of +the main synapse and `worker_replication_http_port` should point to the HTTP +replication port. For example: ```yaml -worker_app: synapse.app.synchrotron +worker_app: synapse.app.generic_worker +worker_name: worker1 # The replication listener on the synapse to talk to. worker_replication_host: 127.0.0.1 -worker_replication_port: 9092 worker_replication_http_port: 9093 worker_listeners: @@ -88,12 +96,12 @@ worker_listeners: - names: - client -worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml +worker_log_config: /home/matrix/synapse/config/worker1_log_config.yaml ``` -...is a full configuration for a synchrotron worker instance, which will expose a -plain HTTP `/sync` endpoint on port 8083 separately from the `/sync` endpoint provided -by the main synapse. +...is a full configuration for a generic worker instance, which will expose a +plain HTTP endpoint on port 8083 separately serving various endpoints, e.g. +`/sync`, which are listed below. Obviously you should configure your reverse-proxy to route the relevant endpoints to the worker (`localhost:8083` in the above example). @@ -104,49 +112,6 @@ recommend the use of `systemd` where available: for information on setting up `systemd` to start synapse workers, see [systemd-with-workers](systemd-with-workers). To use `synctl`, see below. -### **Experimental** support for replication over redis - -As of Synapse v1.13.0, it is possible to configure Synapse to send replication -via a [Redis pub/sub channel](https://redis.io/topics/pubsub). This is an -alternative to direct TCP connections to the master: rather than all the -workers connecting to the master, all the workers and the master connect to -Redis, which relays replication commands between processes. This can give a -significant cpu saving on the master and will be a prerequisite for upcoming -performance improvements. - -Note that this support is currently experimental; you may experience lost -messages and similar problems! It is strongly recommended that admins setting -up workers for the first time use direct TCP replication as above. - -To configure Synapse to use Redis: - -1. Install Redis following the normal procedure for your distribution - for - example, on Debian, `apt install redis-server`. (It is safe to use an - existing Redis deployment if you have one: we use a pub/sub stream named - according to the `server_name` of your synapse server.) -2. Check Redis is running and accessible: you should be able to `echo PING | nc -q1 - localhost 6379` and get a response of `+PONG`. -3. Install the python prerequisites. If you installed synapse into a - virtualenv, this can be done with: - ```sh - pip install matrix-synapse[redis] - ``` - The debian packages from matrix.org already include the required - dependencies. -4. Add config to the shared configuration (`homeserver.yaml`): - ```yaml - redis: - enabled: true - ``` - Optional parameters which can go alongside `enabled` are `host`, `port`, - `password`. Normally none of these are required. -5. Restart master and all workers. - -Once redis replication is in use, `worker_replication_port` is redundant and -can be removed from the worker configuration files. Similarly, the -configuration for the `listener` for the TCP replication port can be removed -from the main configuration file. Note that the HTTP replication port is -still required. ### Using synctl @@ -163,7 +128,7 @@ to which it will write its process ID when it starts. For example, for a synchrotron, you might write: ```yaml -worker_pid_file: /home/matrix/synapse/synchrotron.pid +worker_pid_file: /home/matrix/synapse/worker1.pid ``` Finally, to actually run your worker-based synapse, you must pass synctl the `-a` @@ -179,50 +144,29 @@ notifications. To manipulate a specific worker, you pass the -w option to synctl: - synctl -w $CONFIG/workers/synchrotron.yaml restart + synctl -w $CONFIG/workers/worker1.yaml restart ## Available worker applications -### `synapse.app.pusher` +*Note:* Historically there used to be more apps, however they have been +amalgamated into a single `synapse.app.generic_worker` app. The remaining apps +are ones that do specific processing unrelated to requests, e.g. the `pusher` +that handles sending out push notifications for new events. The intention is for +all these to be folded into the `generic_worker` app and to use config to define +which processes handle the various proccessing such as push notifications. -Handles sending push notifications to sygnal and email. Doesn't handle any -REST endpoints itself, but you should set `start_pushers: False` in the -shared configuration file to stop the main synapse sending these notifications. +### `synapse.app.generic_worker` -Note this worker cannot be load-balanced: only one instance should be active. - -### `synapse.app.synchrotron` - -The synchrotron handles `sync` requests from clients. In particular, it can -handle REST endpoints matching the following regular expressions: +Handles the following API requests listed below matching the following regular +expressions: + # Sync requests ^/_matrix/client/(v2_alpha|r0)/sync$ ^/_matrix/client/(api/v1|v2_alpha|r0)/events$ ^/_matrix/client/(api/v1|r0)/initialSync$ ^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$ -The above endpoints should all be routed to the synchrotron worker by the -reverse-proxy configuration. - -It is possible to run multiple instances of the synchrotron to scale -horizontally. In this case the reverse-proxy should be configured to -load-balance across the instances, though it will be more efficient if all -requests from a particular user are routed to a single instance. Extracting -a userid from the access token is currently left as an exercise for the reader. - -### `synapse.app.appservice` - -Handles sending output traffic to Application Services. Doesn't handle any -REST endpoints itself, but you should set `notify_appservices: False` in the -shared configuration file to stop the main synapse sending these notifications. - -Note this worker cannot be load-balanced: only one instance should be active. - -### `synapse.app.federation_reader` - -Handles a subset of federation endpoints. In particular, it can handle REST -endpoints matching the following regular expressions: - + # Federation requests ^/_matrix/federation/v1/event/ ^/_matrix/federation/v1/state/ ^/_matrix/federation/v1/state_ids/ @@ -242,40 +186,141 @@ endpoints matching the following regular expressions: ^/_matrix/federation/v1/event_auth/ ^/_matrix/federation/v1/exchange_third_party_invite/ ^/_matrix/federation/v1/user/devices/ - ^/_matrix/federation/v1/send/ ^/_matrix/federation/v1/get_groups_publicised$ ^/_matrix/key/v2/query + # Inbound federation transaction request + ^/_matrix/federation/v1/send/ + + # Client API requests + ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$ + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$ + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$ + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$ + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$ + ^/_matrix/client/(api/v1|r0|unstable)/account/3pid$ + ^/_matrix/client/(api/v1|r0|unstable)/keys/query$ + ^/_matrix/client/(api/v1|r0|unstable)/keys/changes$ + ^/_matrix/client/versions$ + ^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$ + ^/_matrix/client/(api/v1|r0|unstable)/joined_groups$ + ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$ + ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/ + + # Registration/login requests + ^/_matrix/client/(api/v1|r0|unstable)/login$ + ^/_matrix/client/(r0|unstable)/register$ + ^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$ + + # Event sending requests + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/ + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ + ^/_matrix/client/(api/v1|r0|unstable)/join/ + ^/_matrix/client/(api/v1|r0|unstable)/profile/ + + Additionally, the following REST endpoints can be handled for GET requests: ^/_matrix/federation/v1/groups/ -The above endpoints should all be routed to the federation_reader worker by the -reverse-proxy configuration. +Pagination requests can also be handled, but all requests with the same path +room must be routed to the same instance. Additionally, care must be taken to +ensure that the purge history admin API is not used while pagination requests +for the room are in flight: + + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$ + +Note that the `client` and `federation` must be added to the listener resources +in the worker config. + + +#### Load balancing + +Multiple instances of this app can be run and requests load balanced between +them. However, different endpoints have different characteristics and so admins +may wish to run multiple groups of workers handling different endpoints so that +load balancing can be done in different ways. + +For `/sync` and `/initialSync` requests it will be more efficient if all +requests from a particular user are routed to a single instance. Extracting a +user ID from the access token or `Authorization` header is currently left as an +exercise for the reader. Admins may additionally wish to separate out `/sync` +requests that have a `since` query parameter from those that don't (and +`/initialSync`), as requests that don't are known as "initial sync" that happens +when a user logs in on a new device and can be *very* resource intensive, so +isolating these requests will stop them from intefering with other users ongoing +syncs. + +Federation and client requests can be balanced via simple round robin. + +The inbound federation transaction request `^/_matrix/federation/v1/send/` +should be balanced by source IP so that transactions from the same remote server +go to the same process. -The `^/_matrix/federation/v1/send/` endpoint must only be handled by a single -instance. +Registration/login requests can be handled separately purely to help ensure that +unexpected load doesn't effect new logins and sign ups. -Note that `federation` must be added to the listener resources in the worker config: +Finally, event sending requests can be balanced by the embedded room ID (or +URI, or even just round robin). If there is a large bridge connected that is +sending or may send lots of events, then a dedicated set of workers can be +provisioned to ensure that bursts or increases of event sending is isolated from +effecting events sent by real users. + +#### Stream writers + +Additionally, there is *experimental* support for moving writing of specific +streams (such as events and typing) off of master to a particular worker. This +requires use of Redis. + +To enable this then the worker must have a HTTP replication listener configured, +have a `worker_name` and be listed in the `instance_map` config. For example to +move event persistence off to a dedicated worker, the main shared config would +include: ```yaml -worker_app: synapse.app.federation_reader -... -worker_listeners: - - type: http - port: - resources: - - names: - - federation +instance_map: + event_persister1: + host: localhost + port: 8034 + +streams_writers: + events: event_persister1 ``` + +### `synapse.app.pusher` + +Handles sending push notifications to sygnal and email. Doesn't handle any +REST endpoints itself, but you should set `start_pushers: False` in the +shared configuration file to stop the main synapse sending these notifications. + +Note this worker cannot be load-balanced: only one instance should be active. + +### `synapse.app.appservice` + +Handles sending output traffic to Application Services. Doesn't handle any +REST endpoints itself, but you should set `notify_appservices: False` in the +shared configuration file to stop the main synapse sending these notifications. + +Note this worker cannot be load-balanced: only one instance should be active. + + ### `synapse.app.federation_sender` Handles sending federation traffic to other servers. Doesn't handle any REST endpoints itself, but you should set `send_federation: False` in the shared configuration file to stop the main synapse sending this traffic. -Note this worker cannot be load-balanced: only one instance should be active. +Note that if running multiple federation senders then you must list each +instance in the `federation_sender_instances` option by their `worker_name`. If +you add or remove instances they must all be stopped and started together. For example: + +```yaml +federation_sender_instances: + - federation_sender1 + - federation_sender2 +``` ### `synapse.app.media_repository` @@ -314,46 +359,6 @@ and you must configure a single instance to run the background tasks, e.g.: media_instance_running_background_jobs: "media-repository-1" ``` -### `synapse.app.client_reader` - -Handles client API endpoints. It can handle REST endpoints matching the -following regular expressions: - - ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$ - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$ - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$ - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$ - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$ - ^/_matrix/client/(api/v1|r0|unstable)/login$ - ^/_matrix/client/(api/v1|r0|unstable)/account/3pid$ - ^/_matrix/client/(api/v1|r0|unstable)/keys/query$ - ^/_matrix/client/(api/v1|r0|unstable)/keys/changes$ - ^/_matrix/client/versions$ - ^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$ - ^/_matrix/client/(api/v1|r0|unstable)/joined_groups$ - ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$ - ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/ - -Additionally, the following REST endpoints can be handled for GET requests: - - ^/_matrix/client/(api/v1|r0|unstable)/pushrules/.*$ - ^/_matrix/client/(api/v1|r0|unstable)/groups/.*$ - ^/_matrix/client/(api/v1|r0|unstable)/user/[^/]*/account_data/ - ^/_matrix/client/(api/v1|r0|unstable)/user/[^/]*/rooms/[^/]*/account_data/ - -Additionally, the following REST endpoints can be handled, but all requests must -be routed to the same instance: - - ^/_matrix/client/(r0|unstable)/register$ - ^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$ - -Pagination requests can also be handled, but all requests with the same path -room must be routed to the same instance. Additionally, care must be taken to -ensure that the purge history admin API is not used while pagination requests -for the room are in flight: - - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$ - ### `synapse.app.user_dir` Handles searches in the user directory. It can handle REST endpoints matching @@ -387,16 +392,3 @@ the `worker_main_http_uri` setting in the `frontend_proxy` worker configuration file. For example: worker_main_http_uri: http://127.0.0.1:8008 - -### `synapse.app.event_creator` - -Handles some event creation. It can handle REST endpoints matching: - - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/ - ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ - ^/_matrix/client/(api/v1|r0|unstable)/join/ - ^/_matrix/client/(api/v1|r0|unstable)/profile/ - -It will create events locally and then send them on to the main synapse -instance to be persisted and handled. From a880415089165fcab208323b0c3be62e1f2b12bb Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 11:17:09 +0100 Subject: [PATCH 02/26] Add some of the new worker options to sample config --- docs/sample_config.yaml | 44 ++++++++++++++++++++++++++++++++++++ synapse/config/federation.py | 9 ++++++++ synapse/config/homeserver.py | 2 +- synapse/config/redis.py | 20 +++++++++++++++- synapse/config/workers.py | 21 +++++++++++++++++ 5 files changed, 94 insertions(+), 2 deletions(-) diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml index 3227294e0b27..b02e6dc8893c 100644 --- a/docs/sample_config.yaml +++ b/docs/sample_config.yaml @@ -610,6 +610,15 @@ federation_ip_range_blacklist: - 'fe80::/64' - 'fc00::/7' +# If running with federation sender worker instances then they should +# be listed by their `worker_name` here, as well as setting +# `send_federation` to false. +# +#send_federation: false +# +#federation_sender_instances: +# - federation_sender1 + ## Caching ## @@ -2398,3 +2407,38 @@ opentracing: # # logging: # false + + +## Workers ## + +# When using workers this should be a map from `worker_name` to the +# HTTP replication listener of the worker, if configured. +# +#instance_map: +# worker1: +# host: localhost +# port: 8034 + +# Experimental: When using workers you can define if workers should +# handle event persistence or typing notifications here. Any worker +# specified here must also be in the `instance_map`. +# +#stream_writers: +# events: worker1 +# typing: worker1 + + +# Configuration for Redis when using workers. +# +redis: + #enabled: true + + # Optional host and port to use to connect to redis. Defaults to + # localhost and 6379 + # + #host: localhost + #port: 6379 + + # Optional password if configured on the Redis instance + # + #password: diff --git a/synapse/config/federation.py b/synapse/config/federation.py index 82ff9664de54..19987a317b60 100644 --- a/synapse/config/federation.py +++ b/synapse/config/federation.py @@ -95,4 +95,13 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs): - '::1/128' - 'fe80::/64' - 'fc00::/7' + + # If running with federation sender worker instances then they should + # be listed by their `worker_name` here, as well as setting + # `send_federation` to false. + # + #send_federation: false + # + #federation_sender_instances: + # - federation_sender1 """ diff --git a/synapse/config/homeserver.py b/synapse/config/homeserver.py index 8e93d3139433..556e291495f5 100644 --- a/synapse/config/homeserver.py +++ b/synapse/config/homeserver.py @@ -78,7 +78,6 @@ class HomeServerConfig(RootConfig): JWTConfig, PasswordConfig, EmailConfig, - WorkerConfig, PasswordAuthProviderConfig, PushConfig, SpamCheckerConfig, @@ -91,6 +90,7 @@ class HomeServerConfig(RootConfig): RoomDirectoryConfig, ThirdPartyRulesConfig, TracerConfig, + WorkerConfig, RedisConfig, FederationConfig, ] diff --git a/synapse/config/redis.py b/synapse/config/redis.py index d5d3ca1c9e1e..b9299c5345ea 100644 --- a/synapse/config/redis.py +++ b/synapse/config/redis.py @@ -21,7 +21,7 @@ class RedisConfig(Config): section = "redis" def read_config(self, config, **kwargs): - redis_config = config.get("redis", {}) + redis_config = config.get("redis", {}) or {} self.redis_enabled = redis_config.get("enabled", False) if not self.redis_enabled: @@ -32,3 +32,21 @@ def read_config(self, config, **kwargs): self.redis_host = redis_config.get("host", "localhost") self.redis_port = redis_config.get("port", 6379) self.redis_password = redis_config.get("password") + + def generate_config_section(self, config_dir_path, server_name, **kwargs): + return """\ + # Configuration for Redis when using workers. + # + redis: + #enabled: true + + # Optional host and port to use to connect to redis. Defaults to + # localhost and 6379 + # + #host: localhost + #port: 6379 + + # Optional password if configured on the Redis instance + # + #password: + """ diff --git a/synapse/config/workers.py b/synapse/config/workers.py index 2574cd3aa170..abc07cf25dfe 100644 --- a/synapse/config/workers.py +++ b/synapse/config/workers.py @@ -105,6 +105,27 @@ def read_config(self, config, **kwargs): % (instance, stream) ) + def generate_config_section(self, config_dir_path, server_name, **kwargs): + return """\ + ## Workers ## + + # When using workers this should be a map from `worker_name` to the + # HTTP replication listener of the worker, if configured. + # + #instance_map: + # worker1: + # host: localhost + # port: 8034 + + # Experimental: When using workers you can define if workers should + # handle event persistence or typing notifications here. Any worker + # specified here must also be in the `instance_map`. + # + #stream_writers: + # events: worker1 + # typing: worker1 + """ + def read_arguments(self, args): # We support a bunch of command line arguments that override options in # the config. A lot of these options have a worker_* prefix when running From 43df3fd3fe0be2dc6a8b060e28a3ae78b798c056 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 11:28:09 +0100 Subject: [PATCH 03/26] Newsfile --- changelog.d/7969.doc | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelog.d/7969.doc diff --git a/changelog.d/7969.doc b/changelog.d/7969.doc new file mode 100644 index 000000000000..68d2ed5fadc9 --- /dev/null +++ b/changelog.d/7969.doc @@ -0,0 +1 @@ +Update worker docs with latest enhancements. From 48ba5648b4c49206376fb0b7012718e04d723842 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:12:36 +0100 Subject: [PATCH 04/26] Update docs/workers.md Co-authored-by: Brendan Abolivier --- docs/workers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workers.md b/docs/workers.md index e55e653ad7cc..a8dcc967464a 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -249,7 +249,7 @@ exercise for the reader. Admins may additionally wish to separate out `/sync` requests that have a `since` query parameter from those that don't (and `/initialSync`), as requests that don't are known as "initial sync" that happens when a user logs in on a new device and can be *very* resource intensive, so -isolating these requests will stop them from intefering with other users ongoing +isolating these requests will stop them from interfering with other users ongoing syncs. Federation and client requests can be balanced via simple round robin. From 5494d0c81bd6af9cb1cd843d02c6dcec93c2d1c0 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:16:55 +0100 Subject: [PATCH 05/26] Update synapse/config/redis.py Co-authored-by: Patrick Cloke --- synapse/config/redis.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/synapse/config/redis.py b/synapse/config/redis.py index b9299c5345ea..ff19054014f1 100644 --- a/synapse/config/redis.py +++ b/synapse/config/redis.py @@ -21,7 +21,7 @@ class RedisConfig(Config): section = "redis" def read_config(self, config, **kwargs): - redis_config = config.get("redis", {}) or {} + redis_config = config.get("redis") or {} self.redis_enabled = redis_config.get("enabled", False) if not self.redis_enabled: From 3e6af96ed4cebb1c75db27e712ac8c5204527dab Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:17:09 +0100 Subject: [PATCH 06/26] Update synapse/config/redis.py Co-authored-by: Patrick Cloke --- synapse/config/redis.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/synapse/config/redis.py b/synapse/config/redis.py index ff19054014f1..1ba7d4c78dd1 100644 --- a/synapse/config/redis.py +++ b/synapse/config/redis.py @@ -38,6 +38,8 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs): # Configuration for Redis when using workers. # redis: + # Uncomment the below to enable using Redis to replicate data between workers. + # #enabled: true # Optional host and port to use to connect to redis. Defaults to From 8bd0b083d935ce24636943ce371b8aeaa78d7d83 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:19:14 +0100 Subject: [PATCH 07/26] Update synapse/config/workers.py Co-authored-by: Patrick Cloke --- synapse/config/workers.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/synapse/config/workers.py b/synapse/config/workers.py index abc07cf25dfe..9a4980bb86ee 100644 --- a/synapse/config/workers.py +++ b/synapse/config/workers.py @@ -117,8 +117,8 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs): # host: localhost # port: 8034 - # Experimental: When using workers you can define if workers should - # handle event persistence or typing notifications here. Any worker + # Experimental: When using workers you can define which worker should + # handle event persistence or typing notifications. Any worker # specified here must also be in the `instance_map`. # #stream_writers: From b91e4fb1fb558ca714074cba7c31774887aca478 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:19:40 +0100 Subject: [PATCH 08/26] Update docs/workers.md Co-authored-by: Patrick Cloke --- docs/workers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workers.md b/docs/workers.md index a8dcc967464a..2bb2d9924f6b 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -85,7 +85,7 @@ For example: worker_app: synapse.app.generic_worker worker_name: worker1 -# The replication listener on the synapse to talk to. +# The replication listener on the master synapse process. worker_replication_host: 127.0.0.1 worker_replication_http_port: 9093 From 3dd4b73356e7e4538c6facb94ba95da4731b5484 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:21:09 +0100 Subject: [PATCH 09/26] Apply suggestions from code review Co-authored-by: Patrick Cloke --- docs/workers.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index 2bb2d9924f6b..ef9beac27185 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -261,7 +261,7 @@ go to the same process. Registration/login requests can be handled separately purely to help ensure that unexpected load doesn't effect new logins and sign ups. -Finally, event sending requests can be balanced by the embedded room ID (or +Finally, event sending requests can be balanced by the embedded room ID (or URI, or even just round robin). If there is a large bridge connected that is sending or may send lots of events, then a dedicated set of workers can be provisioned to ensure that bursts or increases of event sending is isolated from @@ -293,7 +293,7 @@ streams_writers: Handles sending push notifications to sygnal and email. Doesn't handle any REST endpoints itself, but you should set `start_pushers: False` in the -shared configuration file to stop the main synapse sending these notifications. +shared configuration file to stop the main synapse sending push notifications. Note this worker cannot be load-balanced: only one instance should be active. @@ -301,7 +301,7 @@ Note this worker cannot be load-balanced: only one instance should be active. Handles sending output traffic to Application Services. Doesn't handle any REST endpoints itself, but you should set `notify_appservices: False` in the -shared configuration file to stop the main synapse sending these notifications. +shared configuration file to stop the main synapse sending appservice notifications. Note this worker cannot be load-balanced: only one instance should be active. @@ -312,9 +312,10 @@ Handles sending federation traffic to other servers. Doesn't handle any REST endpoints itself, but you should set `send_federation: False` in the shared configuration file to stop the main synapse sending this traffic. -Note that if running multiple federation senders then you must list each -instance in the `federation_sender_instances` option by their `worker_name`. If -you add or remove instances they must all be stopped and started together. For example: +If running multiple federation senders then you must list each +instance in the `federation_sender_instances` option by their `worker_name`. +All instances must be stopped and started when adding or removing instances. +For example: ```yaml federation_sender_instances: From 1fe0335653898f1aa18df3e4220d0d81b3e670bd Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:15:38 +0100 Subject: [PATCH 10/26] s/master/main/ --- docs/workers.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index ef9beac27185..c4a0df8baa45 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -16,7 +16,7 @@ workers only work with PostgreSQL-based Synapse deployments. SQLite should only be used for demo purposes and any admin considering workers should already be running PostgreSQL. -## Master/worker communication +## Main process/worker communication The processes communicate with each other via a Synapse-specific protocol called 'replication' (analogous to MySQL- or Postgres-style database replication) which @@ -29,10 +29,11 @@ used for operations which need to wait for a reply - such as sending an event. As of Synapse v1.13.0, it is possible to configure Synapse to send replication via a [Redis pub/sub channel](https://redis.io/topics/pubsub), and is now the recommended way of configuring replication. This is an alternative to the old -direct TCP connections to the master: rather than all the workers connecting to -the master, all the workers and the master connect to Redis, which relays -replication commands between processes. This can give a significant cpu saving -on the master and will be a prerequisite for upcoming performance improvements. +direct TCP connections to the main process: rather than all the workers +connecting to the main process, all the workers and the main process connect to +Redis, which relays replication commands between processes. This can give a +significant cpu saving on the main process and will be a prerequisite for +upcoming performance improvements. ## Configuration @@ -85,7 +86,7 @@ For example: worker_app: synapse.app.generic_worker worker_name: worker1 -# The replication listener on the master synapse process. +# The replication listener on the main synapse process. worker_replication_host: 127.0.0.1 worker_replication_http_port: 9093 @@ -116,7 +117,7 @@ recommend the use of `systemd` where available: for information on setting up ### Using synctl If you want to use `synctl` to manage your synapse processes, you will need to -create an an additional configuration file for the master synapse process. That +create an an additional configuration file for the main synapse process. That configuration should look like this: ```yaml @@ -270,8 +271,8 @@ effecting events sent by real users. #### Stream writers Additionally, there is *experimental* support for moving writing of specific -streams (such as events and typing) off of master to a particular worker. This -requires use of Redis. +streams (such as events and typing) off of the main process to a particular +worker. This requires use of Redis. To enable this then the worker must have a HTTP replication listener configured, have a `worker_name` and be listed in the `instance_map` config. For example to From dad0be0491a578aa4f4afe060717a08b9155088a Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:25:14 +0100 Subject: [PATCH 11/26] Add back some notes about redis dependencies --- docs/workers.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/workers.md b/docs/workers.md index c4a0df8baa45..eea2b3a8711e 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -35,6 +35,17 @@ Redis, which relays replication commands between processes. This can give a significant cpu saving on the main process and will be a prerequisite for upcoming performance improvements. +If using Redis the appropriate dependencies must be installed. If using a +virtual env these can be installed by: + +```sh +pip install matrix-synapse[redis] +``` + +Note that these dependencies are included if synapse was installed with `[all]`, +or if using the debian packages from matrix.org. + + ## Configuration From 31191fa56aee78d94b63ac6bdba5298836d4e96a Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:31:13 +0100 Subject: [PATCH 12/26] Clarify the 'federation_sender_instances' config --- synapse/config/federation.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/synapse/config/federation.py b/synapse/config/federation.py index 19987a317b60..cb0010187415 100644 --- a/synapse/config/federation.py +++ b/synapse/config/federation.py @@ -96,11 +96,15 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs): - 'fe80::/64' - 'fc00::/7' - # If running with federation sender worker instances then they should - # be listed by their `worker_name` here, as well as setting - # `send_federation` to false. + # Uncomment if using a federation sender worker. # #send_federation: false + + # Multiple federation sender workers can be run, in which case the work is sharded + # between them. Note that this config must be shared between all instances, and if + # changed all federation sender workers must be stopped at the same time and then + # started, to ensure that all instances are running with the same config (otherwise + # events may be dropped). # #federation_sender_instances: # - federation_sender1 From 5d8a970a3a32f912dfdacbc52a99cd5d02859561 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Tue, 28 Jul 2020 17:34:50 +0100 Subject: [PATCH 13/26] sample config --- docs/sample_config.yaml | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml index b02e6dc8893c..e5e001a28770 100644 --- a/docs/sample_config.yaml +++ b/docs/sample_config.yaml @@ -610,11 +610,15 @@ federation_ip_range_blacklist: - 'fe80::/64' - 'fc00::/7' -# If running with federation sender worker instances then they should -# be listed by their `worker_name` here, as well as setting -# `send_federation` to false. +# Uncomment if using a federation sender worker. # #send_federation: false + +# Multiple federation sender workers can be run, in which case the work is sharded +# between them. Note that this config must be shared between all instances, and if +# changed all federation sender workers must be stopped at the same time and then +# started, to ensure that all instances are running with the same config (otherwise +# events may be dropped). # #federation_sender_instances: # - federation_sender1 @@ -2419,8 +2423,8 @@ opentracing: # host: localhost # port: 8034 -# Experimental: When using workers you can define if workers should -# handle event persistence or typing notifications here. Any worker +# Experimental: When using workers you can define which worker should +# handle event persistence or typing notifications. Any worker # specified here must also be in the `instance_map`. # #stream_writers: @@ -2431,6 +2435,8 @@ opentracing: # Configuration for Redis when using workers. # redis: + # Uncomment the below to enable using Redis to replicate data between workers. + # #enabled: true # Optional host and port to use to connect to redis. Defaults to From 196e585605ecf81fb62fb02a758703fa511a5317 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 10:07:52 +0100 Subject: [PATCH 14/26] Review suggestions --- docs/sample_config.yaml | 11 +++++++---- docs/workers.md | 9 +++++---- synapse/config/federation.py | 7 +++++-- synapse/config/workers.py | 4 ++-- 4 files changed, 19 insertions(+), 12 deletions(-) diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml index e5e001a28770..deb2e96b1368 100644 --- a/docs/sample_config.yaml +++ b/docs/sample_config.yaml @@ -610,12 +610,15 @@ federation_ip_range_blacklist: - 'fe80::/64' - 'fc00::/7' +# Disables sending of outbound federation transactions on the main process. # Uncomment if using a federation sender worker. # #send_federation: false -# Multiple federation sender workers can be run, in which case the work is sharded -# between them. Note that this config must be shared between all instances, and if +# It is possible to run multiple federation sender workers, in which case the +# work is balanced across them. +# +# This configuration must be shared between all federation sender workers, and if # changed all federation sender workers must be stopped at the same time and then # started, to ensure that all instances are running with the same config (otherwise # events may be dropped). @@ -2423,8 +2426,8 @@ opentracing: # host: localhost # port: 8034 -# Experimental: When using workers you can define which worker should -# handle event persistence or typing notifications. Any worker +# Experimental: When using workers you can define which workers should +# handle event persistence and typing notifications. Any worker # specified here must also be in the `instance_map`. # #stream_writers: diff --git a/docs/workers.md b/docs/workers.md index eea2b3a8711e..9ba6bc1a8540 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -56,7 +56,8 @@ requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse proxy. To enable workers, you need to add both a HTTP replication listener and redis -config to the main Synapse configuration file (`homeserver.yaml`). For example: +config to the shared Synapse configuration file (`homeserver.yaml`). For +example: ```yaml listeners: @@ -76,8 +77,8 @@ See the sample config for the full documentation of each option. Under **no circumstances** should the replication API listener be exposed to the public internet; it has no authentication and is unencrypted. -You should then create a set of configs for the various worker processes. Each -worker configuration file inherits the configuration of the main homeserver +You should then create a configuration file for each worker process. Each +worker configuration file inherits the configuration of the shared homeserver configuration file. You can then override configuration specific to that worker, e.g. the HTTP listener that it provides (if any); logging configuration; etc. You should minimise the number of overrides though to @@ -287,7 +288,7 @@ worker. This requires use of Redis. To enable this then the worker must have a HTTP replication listener configured, have a `worker_name` and be listed in the `instance_map` config. For example to -move event persistence off to a dedicated worker, the main shared config would +move event persistence off to a dedicated worker, the shared configuration would include: ```yaml diff --git a/synapse/config/federation.py b/synapse/config/federation.py index cb0010187415..b638b6e7e348 100644 --- a/synapse/config/federation.py +++ b/synapse/config/federation.py @@ -96,12 +96,15 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs): - 'fe80::/64' - 'fc00::/7' + # Disables sending of outbound federation transactions on the main process. # Uncomment if using a federation sender worker. # #send_federation: false - # Multiple federation sender workers can be run, in which case the work is sharded - # between them. Note that this config must be shared between all instances, and if + # It is possible to run multiple federation sender workers, in which case the + # work is balanced across them. + # + # This configuration must be shared between all federation sender workers, and if # changed all federation sender workers must be stopped at the same time and then # started, to ensure that all instances are running with the same config (otherwise # events may be dropped). diff --git a/synapse/config/workers.py b/synapse/config/workers.py index 9a4980bb86ee..c6e8a2ba3d5a 100644 --- a/synapse/config/workers.py +++ b/synapse/config/workers.py @@ -117,8 +117,8 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs): # host: localhost # port: 8034 - # Experimental: When using workers you can define which worker should - # handle event persistence or typing notifications. Any worker + # Experimental: When using workers you can define which workers should + # handle event persistence and typing notifications. Any worker # specified here must also be in the `instance_map`. # #stream_writers: From 21ef5cd47e9ad83de88cd8895e2fba6fb7e18db5 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 10:22:02 +0100 Subject: [PATCH 15/26] Add notes about installing Redis --- docs/workers.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index 9ba6bc1a8540..ed3df8fc5445 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -35,7 +35,16 @@ Redis, which relays replication commands between processes. This can give a significant cpu saving on the main process and will be a prerequisite for upcoming performance improvements. -If using Redis the appropriate dependencies must be installed. If using a + +## Setting up workers + +Redis is recommended over the old direct TCP connection configuration, so this +should be installed following the normal procedure for your distribution (e.g. +`apt install redis-server` on Debian). Once installed check thta Redis is +running and accessible from the host running Synapse, e.g. by on Debian by +executing `echo PING | nc -q1 localhost 6379` and seeing a response of `+PONG`. + +The appropriate dependencies must also be installed for Synapse. If using a virtual env these can be installed by: ```sh @@ -45,10 +54,6 @@ pip install matrix-synapse[redis] Note that these dependencies are included if synapse was installed with `[all]`, or if using the debian packages from matrix.org. - - -## Configuration - To make effective use of the workers, you will need to configure an HTTP reverse-proxy such as nginx or haproxy, which will direct incoming requests to the correct worker, or to the main synapse instance. Note that this includes From 8741ac78492f6cd60ce174c0b35b352bf3bb0ed2 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 10:38:21 +0100 Subject: [PATCH 16/26] Move fedearation shard config to worker config --- docs/sample_config.yaml | 32 +++++++++---------- synapse/app/generic_worker.py | 6 ++-- synapse/config/federation.py | 28 +--------------- synapse/config/logger.py | 2 +- synapse/config/workers.py | 28 +++++++++++++++- synapse/federation/send_queue.py | 2 +- synapse/federation/sender/__init__.py | 2 +- .../sender/per_destination_queue.py | 2 +- synapse/storage/data_stores/main/stream.py | 2 +- 9 files changed, 52 insertions(+), 52 deletions(-) diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml index deb2e96b1368..30d3722de875 100644 --- a/docs/sample_config.yaml +++ b/docs/sample_config.yaml @@ -610,22 +610,6 @@ federation_ip_range_blacklist: - 'fe80::/64' - 'fc00::/7' -# Disables sending of outbound federation transactions on the main process. -# Uncomment if using a federation sender worker. -# -#send_federation: false - -# It is possible to run multiple federation sender workers, in which case the -# work is balanced across them. -# -# This configuration must be shared between all federation sender workers, and if -# changed all federation sender workers must be stopped at the same time and then -# started, to ensure that all instances are running with the same config (otherwise -# events may be dropped). -# -#federation_sender_instances: -# - federation_sender1 - ## Caching ## @@ -2418,6 +2402,22 @@ opentracing: ## Workers ## +# Disables sending of outbound federation transactions on the main process. +# Uncomment if using a federation sender worker. +# +#send_federation: false + +# It is possible to run multiple federation sender workers, in which case the +# work is balanced across them. +# +# This configuration must be shared between all federation sender workers, and if +# changed all federation sender workers must be stopped at the same time and then +# started, to ensure that all instances are running with the same config (otherwise +# events may be dropped). +# +#federation_sender_instances: +# - federation_sender1 + # When using workers this should be a map from `worker_name` to the # HTTP replication listener of the worker, if configured. # diff --git a/synapse/app/generic_worker.py b/synapse/app/generic_worker.py index ec0dbddb8ce3..5841454c9a11 100644 --- a/synapse/app/generic_worker.py +++ b/synapse/app/generic_worker.py @@ -940,7 +940,7 @@ def start(config_options): config.server.update_user_directory = False if config.worker_app == "synapse.app.federation_sender": - if config.federation.send_federation: + if config.worker.send_federation: sys.stderr.write( "\nThe send_federation must be disabled in the main synapse process" "\nbefore they can be run in a separate worker." @@ -950,10 +950,10 @@ def start(config_options): sys.exit(1) # Force the pushers to start since they will be disabled in the main config - config.federation.send_federation = True + config.worker.send_federation = True else: # For other worker types we force this to off. - config.federation.send_federation = False + config.worker.send_federation = False synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts diff --git a/synapse/config/federation.py b/synapse/config/federation.py index b638b6e7e348..2c77d8f85bb7 100644 --- a/synapse/config/federation.py +++ b/synapse/config/federation.py @@ -17,23 +17,13 @@ from netaddr import IPSet -from ._base import Config, ConfigError, ShardedWorkerHandlingConfig +from ._base import Config, ConfigError class FederationConfig(Config): section = "federation" def read_config(self, config, **kwargs): - # Whether to send federation traffic out in this process. This only - # applies to some federation traffic, and so shouldn't be used to - # "disable" federation - self.send_federation = config.get("send_federation", True) - - federation_sender_instances = config.get("federation_sender_instances") or [] - self.federation_shard_config = ShardedWorkerHandlingConfig( - federation_sender_instances - ) - # FIXME: federation_domain_whitelist needs sytests self.federation_domain_whitelist = None # type: Optional[dict] federation_domain_whitelist = config.get("federation_domain_whitelist", None) @@ -95,20 +85,4 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs): - '::1/128' - 'fe80::/64' - 'fc00::/7' - - # Disables sending of outbound federation transactions on the main process. - # Uncomment if using a federation sender worker. - # - #send_federation: false - - # It is possible to run multiple federation sender workers, in which case the - # work is balanced across them. - # - # This configuration must be shared between all federation sender workers, and if - # changed all federation sender workers must be stopped at the same time and then - # started, to ensure that all instances are running with the same config (otherwise - # events may be dropped). - # - #federation_sender_instances: - # - federation_sender1 """ diff --git a/synapse/config/logger.py b/synapse/config/logger.py index 49f6c32beb7f..dd775a97e884 100644 --- a/synapse/config/logger.py +++ b/synapse/config/logger.py @@ -214,7 +214,7 @@ def setup_logging( Set up the logging subsystem. Args: - config (LoggingConfig | synapse.config.workers.WorkerConfig): + config (LoggingConfig | synapse.config.worker.WorkerConfig): configuration data use_worker_options (bool): True to use the 'worker_log_config' option diff --git a/synapse/config/workers.py b/synapse/config/workers.py index c6e8a2ba3d5a..c784a7150897 100644 --- a/synapse/config/workers.py +++ b/synapse/config/workers.py @@ -15,7 +15,7 @@ import attr -from ._base import Config, ConfigError +from ._base import Config, ConfigError, ShardedWorkerHandlingConfig from .server import ListenerConfig, parse_listener_def @@ -85,6 +85,16 @@ def read_config(self, config, **kwargs): ) ) + # Whether to send federation traffic out in this process. This only + # applies to some federation traffic, and so shouldn't be used to + # "disable" federation + self.send_federation = config.get("send_federation", True) + + federation_sender_instances = config.get("federation_sender_instances") or [] + self.federation_shard_config = ShardedWorkerHandlingConfig( + federation_sender_instances + ) + # A map from instance name to host/port of their HTTP replication endpoint. instance_map = config.get("instance_map") or {} self.instance_map = { @@ -109,6 +119,22 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs): return """\ ## Workers ## + # Disables sending of outbound federation transactions on the main process. + # Uncomment if using a federation sender worker. + # + #send_federation: false + + # It is possible to run multiple federation sender workers, in which case the + # work is balanced across them. + # + # This configuration must be shared between all federation sender workers, and if + # changed all federation sender workers must be stopped at the same time and then + # started, to ensure that all instances are running with the same config (otherwise + # events may be dropped). + # + #federation_sender_instances: + # - federation_sender1 + # When using workers this should be a map from `worker_name` to the # HTTP replication listener of the worker, if configured. # diff --git a/synapse/federation/send_queue.py b/synapse/federation/send_queue.py index 4fc9ff92e599..2b0ab2dcbfc8 100644 --- a/synapse/federation/send_queue.py +++ b/synapse/federation/send_queue.py @@ -57,7 +57,7 @@ def __init__(self, hs): # We may have multiple federation sender instances, so we need to track # their positions separately. - self._sender_instances = hs.config.federation.federation_shard_config.instances + self._sender_instances = hs.config.worker.federation_shard_config.instances self._sender_positions = {} # Pending presence map user_id -> UserPresenceState diff --git a/synapse/federation/sender/__init__.py b/synapse/federation/sender/__init__.py index ba4ddd2370a2..6ae6522f87c3 100644 --- a/synapse/federation/sender/__init__.py +++ b/synapse/federation/sender/__init__.py @@ -70,7 +70,7 @@ def __init__(self, hs: "synapse.server.HomeServer"): self._transaction_manager = TransactionManager(hs) self._instance_name = hs.get_instance_name() - self._federation_shard_config = hs.config.federation.federation_shard_config + self._federation_shard_config = hs.config.worker.federation_shard_config # map from destination to PerDestinationQueue self._per_destination_queues = {} # type: Dict[str, PerDestinationQueue] diff --git a/synapse/federation/sender/per_destination_queue.py b/synapse/federation/sender/per_destination_queue.py index 343674178327..dd150f89a6f0 100644 --- a/synapse/federation/sender/per_destination_queue.py +++ b/synapse/federation/sender/per_destination_queue.py @@ -75,7 +75,7 @@ def __init__( self._store = hs.get_datastore() self._transaction_manager = transaction_manager self._instance_name = hs.get_instance_name() - self._federation_shard_config = hs.config.federation.federation_shard_config + self._federation_shard_config = hs.config.worker.federation_shard_config self._should_send_on_this_instance = True if not self._federation_shard_config.should_handle( diff --git a/synapse/storage/data_stores/main/stream.py b/synapse/storage/data_stores/main/stream.py index 5e32c7aa1e5f..10d39b369943 100644 --- a/synapse/storage/data_stores/main/stream.py +++ b/synapse/storage/data_stores/main/stream.py @@ -255,7 +255,7 @@ def __init__(self, database: Database, db_conn, hs): self._instance_name = hs.get_instance_name() self._send_federation = hs.should_send_federation() - self._federation_shard_config = hs.config.federation.federation_shard_config + self._federation_shard_config = hs.config.worker.federation_shard_config # If we're a process that sends federation we may need to reset the # `federation_stream_position` table to match the current sharding From c375b159a5e38fd5389fecaadb8f092bd923368b Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 10:48:02 +0100 Subject: [PATCH 17/26] Note that redis must be enabled when using workers --- synapse/config/redis.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/synapse/config/redis.py b/synapse/config/redis.py index 1ba7d4c78dd1..1373302335b3 100644 --- a/synapse/config/redis.py +++ b/synapse/config/redis.py @@ -35,10 +35,11 @@ def read_config(self, config, **kwargs): def generate_config_section(self, config_dir_path, server_name, **kwargs): return """\ - # Configuration for Redis when using workers. + # Configuration for Redis when using workers. This *must* be enabled when + # using workers (unless using old style direct TCP configuration). # redis: - # Uncomment the below to enable using Redis to replicate data between workers. + # Uncomment the below to enable Redis support. # #enabled: true From 63263efdb80392064fa0b9443508c7ab4d37e386 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 11:13:56 +0100 Subject: [PATCH 18/26] Fix sample config --- docs/sample_config.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml index 30d3722de875..b21e36bb6d63 100644 --- a/docs/sample_config.yaml +++ b/docs/sample_config.yaml @@ -2435,10 +2435,11 @@ opentracing: # typing: worker1 -# Configuration for Redis when using workers. +# Configuration for Redis when using workers. This *must* be enabled when +# using workers (unless using old style direct TCP configuration). # redis: - # Uncomment the below to enable using Redis to replicate data between workers. + # Uncomment the below to enable Redis support. # #enabled: true From 6c78895f17d3712483c8a9c7e54722e3aabe361f Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 13:23:46 +0100 Subject: [PATCH 19/26] Apply suggestions from code review Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- docs/workers.md | 43 +++++++++++++++++++++++++------------------ 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index ed3df8fc5445..93c4b4689adf 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -38,26 +38,30 @@ upcoming performance improvements. ## Setting up workers -Redis is recommended over the old direct TCP connection configuration, so this +A Redis server is required to manage the communication between the processes. +(The older direct TCP connections are now deprecated.) The Redis server should be installed following the normal procedure for your distribution (e.g. -`apt install redis-server` on Debian). Once installed check thta Redis is -running and accessible from the host running Synapse, e.g. by on Debian by +`apt install redis-server` on Debian). It is safe to use an existing Redis +deployment if you have one. + +Once installed, check that Redis is +running and accessible from the host running Synapse, for example by executing `echo PING | nc -q1 localhost 6379` and seeing a response of `+PONG`. The appropriate dependencies must also be installed for Synapse. If using a -virtual env these can be installed by: +virtualenv, these can be installed with: ```sh pip install matrix-synapse[redis] ``` -Note that these dependencies are included if synapse was installed with `[all]`, -or if using the debian packages from matrix.org. +Note that these dependencies are included when synapse is installed with `pip install matrix-synapse[all]`. +They are also included in the debian packages from `matrix.org` and in the docker +images at https://hub.docker.com/r/matrixdotorg/synapse/. To make effective use of the workers, you will need to configure an HTTP reverse-proxy such as nginx or haproxy, which will direct incoming requests to -the correct worker, or to the main synapse instance. Note that this includes -requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) +the correct worker, or to the main synapse instance. See [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse proxy. To enable workers, you need to add both a HTTP replication listener and redis @@ -65,6 +69,8 @@ config to the shared Synapse configuration file (`homeserver.yaml`). For example: ```yaml +# extend the existing `listeners` section. This defines the ports that the +# main process will listen on. listeners: # The HTTP replication port - port: 9093 @@ -79,7 +85,7 @@ redis: See the sample config for the full documentation of each option. -Under **no circumstances** should the replication API listener be exposed to the +Under **no circumstances** should the replication listener be exposed to the public internet; it has no authentication and is unencrypted. You should then create a configuration file for each worker process. Each @@ -175,7 +181,7 @@ which processes handle the various proccessing such as push notifications. ### `synapse.app.generic_worker` -Handles the following API requests listed below matching the following regular +This worker can handle API requests matching the following regular expressions: # Sync requests @@ -242,7 +248,7 @@ Additionally, the following REST endpoints can be handled for GET requests: ^/_matrix/federation/v1/groups/ -Pagination requests can also be handled, but all requests with the same path +Pagination requests can also be handled, but all requests for a given room must be routed to the same instance. Additionally, care must be taken to ensure that the purge history admin API is not used while pagination requests for the room are in flight: @@ -255,8 +261,9 @@ in the worker config. #### Load balancing -Multiple instances of this app can be run and requests load balanced between -them. However, different endpoints have different characteristics and so admins +It is possible to run multiple instances of this worker app, with incoming requests +being load-balanced between them by the reverse-proxy. However, different endpoints +have different characteristics and so admins may wish to run multiple groups of workers handling different endpoints so that load balancing can be done in different ways. @@ -277,21 +284,21 @@ should be balanced by source IP so that transactions from the same remote server go to the same process. Registration/login requests can be handled separately purely to help ensure that -unexpected load doesn't effect new logins and sign ups. +unexpected load doesn't affect new logins and sign ups. Finally, event sending requests can be balanced by the embedded room ID (or URI, or even just round robin). If there is a large bridge connected that is sending or may send lots of events, then a dedicated set of workers can be -provisioned to ensure that bursts or increases of event sending is isolated from -effecting events sent by real users. +provisioned to limit the effects of bursts of events from that bridge on events sent by +normal users. #### Stream writers Additionally, there is *experimental* support for moving writing of specific streams (such as events and typing) off of the main process to a particular -worker. This requires use of Redis. +worker. (This is only supported with Redis-based replication.) -To enable this then the worker must have a HTTP replication listener configured, +To enable this, the worker must have a HTTP replication listener configured, have a `worker_name` and be listed in the `instance_map` config. For example to move event persistence off to a dedicated worker, the shared configuration would include: From 9d891f80388587be61bb63922df1b0e1c068f121 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 13:31:36 +0100 Subject: [PATCH 20/26] Move config file explanation up --- docs/workers.md | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index 93c4b4689adf..24a8925295b6 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -43,10 +43,10 @@ A Redis server is required to manage the communication between the processes. should be installed following the normal procedure for your distribution (e.g. `apt install redis-server` on Debian). It is safe to use an existing Redis deployment if you have one. - -Once installed, check that Redis is -running and accessible from the host running Synapse, for example by -executing `echo PING | nc -q1 localhost 6379` and seeing a response of `+PONG`. + +Once installed, check that Redis is running and accessible from the host running +Synapse, for example by executing `echo PING | nc -q1 localhost 6379` and seeing +a response of `+PONG`. The appropriate dependencies must also be installed for Synapse. If using a virtualenv, these can be installed with: @@ -55,18 +55,25 @@ virtualenv, these can be installed with: pip install matrix-synapse[redis] ``` -Note that these dependencies are included when synapse is installed with `pip install matrix-synapse[all]`. -They are also included in the debian packages from `matrix.org` and in the docker -images at https://hub.docker.com/r/matrixdotorg/synapse/. +Note that these dependencies are included when synapse is installed with `pip +install matrix-synapse[all]`. They are also included in the debian packages from +`matrix.org` and in the docker images at +https://hub.docker.com/r/matrixdotorg/synapse/. To make effective use of the workers, you will need to configure an HTTP reverse-proxy such as nginx or haproxy, which will direct incoming requests to the correct worker, or to the main synapse instance. See [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse proxy. -To enable workers, you need to add both a HTTP replication listener and redis -config to the shared Synapse configuration file (`homeserver.yaml`). For -example: +To enable workers you should create a configuration file for each worker +process. Each worker configuration file inherits the configuration of the shared +homeserver configuration file. You can then override configuration specific to +that worker, e.g. the HTTP listener that it provides (if any); logging +configuration; etc. You should minimise the number of overrides though to +maintain a usable config. + +Next you need to add both a HTTP replication listener and redis config to the +shared Synapse configuration file (`homeserver.yaml`). For example: ```yaml # extend the existing `listeners` section. This defines the ports that the @@ -88,13 +95,6 @@ See the sample config for the full documentation of each option. Under **no circumstances** should the replication listener be exposed to the public internet; it has no authentication and is unencrypted. -You should then create a configuration file for each worker process. Each -worker configuration file inherits the configuration of the shared homeserver -configuration file. You can then override configuration specific to that -worker, e.g. the HTTP listener that it provides (if any); logging -configuration; etc. You should minimise the number of overrides though to -maintain a usable config. - In the config file for each worker, you must specify the type of worker application (`worker_app`), and you should specify a unqiue name for the worker (`worker_name`). The currently available worker applications are listed below. @@ -286,10 +286,10 @@ go to the same process. Registration/login requests can be handled separately purely to help ensure that unexpected load doesn't affect new logins and sign ups. -Finally, event sending requests can be balanced by the embedded room ID (or -URI, or even just round robin). If there is a large bridge connected that is -sending or may send lots of events, then a dedicated set of workers can be -provisioned to limit the effects of bursts of events from that bridge on events sent by +Finally, event sending requests can be balanced by the embedded room ID (or URI, +or even just round robin). If there is a large bridge connected that is sending +or may send lots of events, then a dedicated set of workers can be provisioned +to limit the effects of bursts of events from that bridge on events sent by normal users. #### Stream writers From af26701093cc05a9218b7a21de683fc1dbf19cc7 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 13:35:45 +0100 Subject: [PATCH 21/26] Move note about historical apps down --- docs/workers.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index 24a8925295b6..ccf269cf00ee 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -172,13 +172,6 @@ To manipulate a specific worker, you pass the -w option to synctl: ## Available worker applications -*Note:* Historically there used to be more apps, however they have been -amalgamated into a single `synapse.app.generic_worker` app. The remaining apps -are ones that do specific processing unrelated to requests, e.g. the `pusher` -that handles sending out push notifications for new events. The intention is for -all these to be folded into the `generic_worker` app and to use config to define -which processes handle the various proccessing such as push notifications. - ### `synapse.app.generic_worker` This worker can handle API requests matching the following regular @@ -418,3 +411,12 @@ the `worker_main_http_uri` setting in the `frontend_proxy` worker configuration file. For example: worker_main_http_uri: http://127.0.0.1:8008 + +### Historical apps + +*Note:* Historically there used to be more apps, however they have been +amalgamated into a single `synapse.app.generic_worker` app. The remaining apps +are ones that do specific processing unrelated to requests, e.g. the `pusher` +that handles sending out push notifications for new events. The intention is for +all these to be folded into the `generic_worker` app and to use config to define +which processes handle the various proccessing such as push notifications. From a236824fb41b917d83265d5d1c12c11d0c61e262 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 14:10:20 +0100 Subject: [PATCH 22/26] Doc worker_listeners --- docs/workers.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index ccf269cf00ee..a67a7bcbdddf 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -101,7 +101,9 @@ application (`worker_app`), and you should specify a unqiue name for the worker You must also specify the HTTP replication endpoint that it should talk to on the main synapse process. `worker_replication_host` should specify the host of the main synapse and `worker_replication_http_port` should point to the HTTP -replication port. +replication port. If the worker will handle HTTP requests then the +`worker_listeners` option should be set with a `http` listener, in the same way +as the `listeners` option in the shared config. For example: @@ -119,6 +121,7 @@ worker_listeners: resources: - names: - client + - federation worker_log_config: /home/matrix/synapse/config/worker1_log_config.yaml ``` @@ -248,8 +251,8 @@ for the room are in flight: ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$ -Note that the `client` and `federation` must be added to the listener resources -in the worker config. +Note that a HTTP listener with `client` and `federation` resources must be +configured in the `worker_listeners` option in the worker config. #### Load balancing From 753c4f32423d989db688e56415fa480851b1b61a Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 14:18:02 +0100 Subject: [PATCH 23/26] Specify what the room ID is in the path --- docs/workers.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index a67a7bcbdddf..b2812291d461 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -282,11 +282,11 @@ go to the same process. Registration/login requests can be handled separately purely to help ensure that unexpected load doesn't affect new logins and sign ups. -Finally, event sending requests can be balanced by the embedded room ID (or URI, -or even just round robin). If there is a large bridge connected that is sending -or may send lots of events, then a dedicated set of workers can be provisioned -to limit the effects of bursts of events from that bridge on events sent by -normal users. +Finally, event sending requests can be balanced by the room ID in the URI (or +the full URI, or even just round robin), the room ID is the path component after +`/rooms/`. If there is a large bridge connected that is sending or may send lots +of events, then a dedicated set of workers can be provisioned to limit the +effects of bursts of events from that bridge on events sent by normal users. #### Stream writers From 308b1814e9117ead614b5d6a94f8265bdf79423c Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 14:18:47 +0100 Subject: [PATCH 24/26] Mention explicitly which streams can be moved off master --- docs/workers.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index b2812291d461..b07f7e0922d2 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -291,8 +291,10 @@ effects of bursts of events from that bridge on events sent by normal users. #### Stream writers Additionally, there is *experimental* support for moving writing of specific -streams (such as events and typing) off of the main process to a particular -worker. (This is only supported with Redis-based replication.) +streams (such as events) off of the main process to a particular worker. (This +is only supported with Redis-based replication.) + +Currently support streams are `events` and `typing`. To enable this, the worker must have a HTTP replication listener configured, have a `worker_name` and be listed in the `instance_map` config. For example to From 09aebbcb1d97ab517f32bab782d3fa982bacd7a6 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 14:36:29 +0100 Subject: [PATCH 25/26] Move synctl worker config to seperate file --- docs/synctl_workers.md | 32 ++++++++++++++++++++++++++++++++ docs/workers.md | 36 ++---------------------------------- 2 files changed, 34 insertions(+), 34 deletions(-) create mode 100644 docs/synctl_workers.md diff --git a/docs/synctl_workers.md b/docs/synctl_workers.md new file mode 100644 index 000000000000..8da4a31852ec --- /dev/null +++ b/docs/synctl_workers.md @@ -0,0 +1,32 @@ +### Using synctl with workers + +If you want to use `synctl` to manage your synapse processes, you will need to +create an an additional configuration file for the main synapse process. That +configuration should look like this: + +```yaml +worker_app: synapse.app.homeserver +``` + +Additionally, each worker app must be configured with the name of a "pid file", +to which it will write its process ID when it starts. For example, for a +synchrotron, you might write: + +```yaml +worker_pid_file: /home/matrix/synapse/worker1.pid +``` + +Finally, to actually run your worker-based synapse, you must pass synctl the `-a` +commandline option to tell it to operate on all the worker configurations found +in the given directory, e.g.: + + synctl -a $CONFIG/workers start + +Currently one should always restart all workers when restarting or upgrading +synapse, unless you explicitly know it's safe not to. For instance, restarting +synapse without restarting all the synchrotrons may result in broken typing +notifications. + +To manipulate a specific worker, you pass the -w option to synctl: + + synctl -w $CONFIG/workers/worker1.yaml restart diff --git a/docs/workers.md b/docs/workers.md index b07f7e0922d2..2aed44c76501 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -137,42 +137,10 @@ Finally, you need to start your worker processes. This can be done with either `synctl` or your distribution's preferred service manager such as `systemd`. We recommend the use of `systemd` where available: for information on setting up `systemd` to start synapse workers, see -[systemd-with-workers](systemd-with-workers). To use `synctl`, see below. +[systemd-with-workers](systemd-with-workers). To use `synctl`, see +[synctl_workers.md](synctl_workers.md). -### Using synctl - -If you want to use `synctl` to manage your synapse processes, you will need to -create an an additional configuration file for the main synapse process. That -configuration should look like this: - -```yaml -worker_app: synapse.app.homeserver -``` - -Additionally, each worker app must be configured with the name of a "pid file", -to which it will write its process ID when it starts. For example, for a -synchrotron, you might write: - -```yaml -worker_pid_file: /home/matrix/synapse/worker1.pid -``` - -Finally, to actually run your worker-based synapse, you must pass synctl the `-a` -commandline option to tell it to operate on all the worker configurations found -in the given directory, e.g.: - - synctl -a $CONFIG/workers start - -Currently one should always restart all workers when restarting or upgrading -synapse, unless you explicitly know it's safe not to. For instance, restarting -synapse without restarting all the synchrotrons may result in broken typing -notifications. - -To manipulate a specific worker, you pass the -w option to synctl: - - synctl -w $CONFIG/workers/worker1.yaml restart - ## Available worker applications ### `synapse.app.generic_worker` From 49e804b9e9d658087c3991d29ac10af981f67356 Mon Sep 17 00:00:00 2001 From: Erik Johnston Date: Wed, 29 Jul 2020 16:57:23 +0100 Subject: [PATCH 26/26] Add architectural diagram --- docs/workers.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/docs/workers.md b/docs/workers.md index 2aed44c76501..38bd758e57fb 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -35,6 +35,9 @@ Redis, which relays replication commands between processes. This can give a significant cpu saving on the main process and will be a prerequisite for upcoming performance improvements. +(See the [Architectural diagram](#architectural-diagram) section at the end for +a visualisation of what this looks like) + ## Setting up workers @@ -393,3 +396,40 @@ are ones that do specific processing unrelated to requests, e.g. the `pusher` that handles sending out push notifications for new events. The intention is for all these to be folded into the `generic_worker` app and to use config to define which processes handle the various proccessing such as push notifications. + + +## Architectural diagram + +The following shows an example setup using Redis and a reverse proxy: + +``` + Clients & Federation + | + v + +-----------+ + | | + | Reverse | + | Proxy | + | | + +-----------+ + | | | + | | | HTTP requests + +-------------------+ | +-----------+ + | +---+ | + | | | + v v v ++--------------+ +--------------+ +--------------+ +--------------+ +| Main | | Generic | | Generic | | Event | +| Process | | Worker 1 | | Worker 2 | | Persister | ++--------------+ +--------------+ +--------------+ +--------------+ + ^ ^ | ^ | | ^ | ^ ^ + | | | | | | | | | | + | | | | | HTTP | | | | | + | +----------+<--|---|---------+ | | | | + | | +-------------|-->+----------+ | + | | | | + | | | | + v v v v +==================================================================== + Redis pub/sub channel +```