diff --git a/CHANGES.md b/CHANGES.md index adb2b3e163d8..225fced285a3 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,15 +1,204 @@ -Next version -============ - -* New templates (`sso_auth_confirm.html`, `sso_auth_success.html`, and - `sso_account_deactivated.html`) were added to Synapse. If your Synapse is - configured to use SSO and a custom `sso_redirect_confirm_template_dir` - configuration then these templates will need to be duplicated into that - directory. - -* Plugins using the `complete_sso_login` method of `synapse.module_api.ModuleApi` - should update to using the async/await version `complete_sso_login_async` which - includes additional checks. The non-async version is considered deprecated. +Synapse 1.13.0 (2020-05-19) +=========================== + +This release brings some potential changes necessary for certain +configurations of Synapse: + +* If your Synapse is configured to use SSO and have a custom + `sso_redirect_confirm_template_dir` configuration option set, you will need + to duplicate the new `sso_auth_confirm.html`, `sso_auth_success.html` and + `sso_account_deactivated.html` templates into that directory. +* Synapse plugins using the `complete_sso_login` method of + `synapse.module_api.ModuleApi` should instead switch to the async/await + version, `complete_sso_login_async`, which includes additional checks. The + former version is now deprecated. +* A bug was introduced in Synapse 1.4.0 which could cause the room directory + to be incomplete or empty if Synapse was upgraded directly from v1.2.1 or + earlier, to versions between v1.4.0 and v1.12.x. + +Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes +and for general upgrade guidance. + + +Notice of change to the default `git` branch for Synapse +-------------------------------------------------------- + +With the release of Synapse 1.13.0, the default `git` branch for Synapse has +changed to `develop`, which is the development tip. This is more consistent with +common practice and modern `git` usage. + +The `master` branch, which tracks the latest release, is still available. It is +recommended that developers and distributors who have scripts which run builds +using the default branch of Synapse should therefore consider pinning their +scripts to `master`. + + +Internal Changes +---------------- + +- Update the version of dh-virtualenv we use to build debs, and add focal to the list of target distributions. ([\#7526](https://github.com/matrix-org/synapse/issues/7526)) + + +Synapse 1.13.0rc3 (2020-05-18) +============================== + +Bugfixes +-------- + +- Hash passwords as early as possible during registration. ([\#7523](https://github.com/matrix-org/synapse/issues/7523)) + + +Synapse 1.13.0rc2 (2020-05-14) +============================== + +Bugfixes +-------- + +- Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376)) +- Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483)) + +Internal Changes +---------------- + +- Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470)) + + +Synapse 1.13.0rc1 (2020-05-11) +============================== + +Features +-------- + +- Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata. ([\#7006](https://github.com/matrix-org/synapse/issues/7006)) +- Set `Referrer-Policy` header to `no-referrer` on media downloads. ([\#7009](https://github.com/matrix-org/synapse/issues/7009)) +- Add support for running replication over Redis when using workers. ([\#7040](https://github.com/matrix-org/synapse/issues/7040), [\#7325](https://github.com/matrix-org/synapse/issues/7325), [\#7352](https://github.com/matrix-org/synapse/issues/7352), [\#7401](https://github.com/matrix-org/synapse/issues/7401), [\#7427](https://github.com/matrix-org/synapse/issues/7427), [\#7439](https://github.com/matrix-org/synapse/issues/7439), [\#7446](https://github.com/matrix-org/synapse/issues/7446), [\#7450](https://github.com/matrix-org/synapse/issues/7450), [\#7454](https://github.com/matrix-org/synapse/issues/7454)) +- Admin API `POST /_synapse/admin/v1/join/` to join users to a room like `auto_join_rooms` for creation of users. ([\#7051](https://github.com/matrix-org/synapse/issues/7051)) +- Add options to prevent users from changing their profile or associated 3PIDs. ([\#7096](https://github.com/matrix-org/synapse/issues/7096)) +- Support SSO in the user interactive authentication workflow. ([\#7102](https://github.com/matrix-org/synapse/issues/7102), [\#7186](https://github.com/matrix-org/synapse/issues/7186), [\#7279](https://github.com/matrix-org/synapse/issues/7279), [\#7343](https://github.com/matrix-org/synapse/issues/7343)) +- Allow server admins to define and enforce a password policy ([MSC2000](https://github.com/matrix-org/matrix-doc/issues/2000)). ([\#7118](https://github.com/matrix-org/synapse/issues/7118)) +- Improve the support for SSO authentication on the login fallback page. ([\#7152](https://github.com/matrix-org/synapse/issues/7152), [\#7235](https://github.com/matrix-org/synapse/issues/7235)) +- Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set. ([\#7153](https://github.com/matrix-org/synapse/issues/7153)) +- Admin users are no longer required to be in a room to create an alias for it. ([\#7191](https://github.com/matrix-org/synapse/issues/7191)) +- Require admin privileges to enable room encryption by default. This does not affect existing rooms. ([\#7230](https://github.com/matrix-org/synapse/issues/7230)) +- Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews. ([\#7265](https://github.com/matrix-org/synapse/issues/7265)) +- Allow `/requestToken` endpoints to hide the existence (or lack thereof) of 3PID associations on the homeserver. ([\#7315](https://github.com/matrix-org/synapse/issues/7315)) +- Add a configuration setting to tweak the threshold for dummy events. ([\#7422](https://github.com/matrix-org/synapse/issues/7422)) + + +Bugfixes +-------- + +- Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak. ([\#6573](https://github.com/matrix-org/synapse/issues/6573)) +- Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm. ([\#6634](https://github.com/matrix-org/synapse/issues/6634)) +- Fix missing field `default` when fetching user-defined push rules. ([\#6639](https://github.com/matrix-org/synapse/issues/6639)) +- Improve error responses when accessing remote public room lists. ([\#6899](https://github.com/matrix-org/synapse/issues/6899), [\#7368](https://github.com/matrix-org/synapse/issues/7368)) +- Transfer alias mappings on room upgrade. ([\#6946](https://github.com/matrix-org/synapse/issues/6946)) +- Ensure that a user interactive authentication session is tied to a single request. ([\#7068](https://github.com/matrix-org/synapse/issues/7068), [\#7455](https://github.com/matrix-org/synapse/issues/7455)) +- Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors. ([\#7089](https://github.com/matrix-org/synapse/issues/7089)) +- Return the proper error (`M_BAD_ALIAS`) when a non-existant canonical alias is provided. ([\#7109](https://github.com/matrix-org/synapse/issues/7109)) +- Fix a bug which meant that groups updates were not correctly replicated between workers. ([\#7117](https://github.com/matrix-org/synapse/issues/7117)) +- Fix starting workers when federation sending not split out. ([\#7133](https://github.com/matrix-org/synapse/issues/7133)) +- Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param. ([\#7150](https://github.com/matrix-org/synapse/issues/7150)) +- Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response. ([\#7151](https://github.com/matrix-org/synapse/issues/7151)) +- Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. ([\#7155](https://github.com/matrix-org/synapse/issues/7155)) +- Fix excessive CPU usage by `prune_old_outbound_device_pokes` job. ([\#7159](https://github.com/matrix-org/synapse/issues/7159)) +- Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. ([\#7177](https://github.com/matrix-org/synapse/issues/7177)) +- Fix a bug which could cause incorrect 'cyclic dependency' error. ([\#7178](https://github.com/matrix-org/synapse/issues/7178)) +- Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent. ([\#7199](https://github.com/matrix-org/synapse/issues/7199)) +- Fix some worker-mode replication handling not being correctly recorded in CPU usage stats. ([\#7203](https://github.com/matrix-org/synapse/issues/7203)) +- Do not allow a deactivated user to login via SSO. ([\#7240](https://github.com/matrix-org/synapse/issues/7240), [\#7259](https://github.com/matrix-org/synapse/issues/7259)) +- Fix --help command-line argument. ([\#7249](https://github.com/matrix-org/synapse/issues/7249)) +- Fix room publish permissions not being checked on room creation. ([\#7260](https://github.com/matrix-org/synapse/issues/7260)) +- Reject unknown session IDs during user interactive authentication instead of silently creating a new session. ([\#7268](https://github.com/matrix-org/synapse/issues/7268)) +- Fix a SQL query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log. ([\#7274](https://github.com/matrix-org/synapse/issues/7274)) +- Persist user interactive authentication sessions across workers and Synapse restarts. ([\#7302](https://github.com/matrix-org/synapse/issues/7302)) +- Fixed backwards compatibility logic of the first value of `trusted_third_party_id_servers` being used for `account_threepid_delegates.email`, which occurs when the former, deprecated option is set and the latter is not. ([\#7316](https://github.com/matrix-org/synapse/issues/7316)) +- Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. ([\#7337](https://github.com/matrix-org/synapse/issues/7337), [\#7358](https://github.com/matrix-org/synapse/issues/7358)) +- Fix bad error handling that would cause Synapse to crash if it's provided with a YAML configuration file that's either empty or doesn't parse into a key-value map. ([\#7341](https://github.com/matrix-org/synapse/issues/7341)) +- Fix incorrect metrics reporting for `renew_attestations` background task. ([\#7344](https://github.com/matrix-org/synapse/issues/7344)) +- Prevent non-federating rooms from appearing in responses to federated `POST /publicRoom` requests when a filter was included. ([\#7367](https://github.com/matrix-org/synapse/issues/7367)) +- Fix a bug which would cause the room durectory to be incorrectly populated if Synapse was upgraded directly from v1.2.1 or earlier to v1.4.0 or later. Note that this fix does not apply retrospectively; see the [upgrade notes](UPGRADE.rst#upgrading-to-v1130) for more information. ([\#7387](https://github.com/matrix-org/synapse/issues/7387)) +- Fix bug in `EventContext.deserialize`. ([\#7393](https://github.com/matrix-org/synapse/issues/7393)) + + +Improved Documentation +---------------------- + +- Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`. ([\#6892](https://github.com/matrix-org/synapse/issues/6892)) +- Improve the documentation for database configuration. ([\#6988](https://github.com/matrix-org/synapse/issues/6988)) +- Improve the documentation of application service configuration files. ([\#7091](https://github.com/matrix-org/synapse/issues/7091)) +- Update pre-built package name for FreeBSD. ([\#7107](https://github.com/matrix-org/synapse/issues/7107)) +- Update postgres docs with login troubleshooting information. ([\#7119](https://github.com/matrix-org/synapse/issues/7119)) +- Clean up INSTALL.md a bit. ([\#7141](https://github.com/matrix-org/synapse/issues/7141)) +- Add documentation for running a local CAS server for testing. ([\#7147](https://github.com/matrix-org/synapse/issues/7147)) +- Improve README.md by being explicit about public IP recommendation for TURN relaying. ([\#7167](https://github.com/matrix-org/synapse/issues/7167)) +- Fix a small typo in the `metrics_flags` config option. ([\#7171](https://github.com/matrix-org/synapse/issues/7171)) +- Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution. ([\#7234](https://github.com/matrix-org/synapse/issues/7234)) +- Add documentation to the `password_providers` config option. Add known password provider implementations to docs. ([\#7238](https://github.com/matrix-org/synapse/issues/7238), [\#7248](https://github.com/matrix-org/synapse/issues/7248)) +- Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev. ([\#7251](https://github.com/matrix-org/synapse/issues/7251)) +- Documentation of media_storage_providers options updated to avoid misunderstandings. Contributed by Tristan Lins. ([\#7272](https://github.com/matrix-org/synapse/issues/7272)) +- Add documentation on monitoring workers with Prometheus. ([\#7357](https://github.com/matrix-org/synapse/issues/7357)) +- Clarify endpoint usage in the users admin api documentation. ([\#7361](https://github.com/matrix-org/synapse/issues/7361)) + + +Deprecations and Removals +------------------------- + +- Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`. ([\#7137](https://github.com/matrix-org/synapse/issues/7137)) + + +Internal Changes +---------------- + +- Add benchmarks for LruCache. ([\#6446](https://github.com/matrix-org/synapse/issues/6446)) +- Return total number of users and profile attributes in admin users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#6881](https://github.com/matrix-org/synapse/issues/6881)) +- Change device list streams to have one row per ID. ([\#7010](https://github.com/matrix-org/synapse/issues/7010)) +- Remove concept of a non-limited stream. ([\#7011](https://github.com/matrix-org/synapse/issues/7011)) +- Move catchup of replication streams logic to worker. ([\#7024](https://github.com/matrix-org/synapse/issues/7024), [\#7195](https://github.com/matrix-org/synapse/issues/7195), [\#7226](https://github.com/matrix-org/synapse/issues/7226), [\#7239](https://github.com/matrix-org/synapse/issues/7239), [\#7286](https://github.com/matrix-org/synapse/issues/7286), [\#7290](https://github.com/matrix-org/synapse/issues/7290), [\#7318](https://github.com/matrix-org/synapse/issues/7318), [\#7326](https://github.com/matrix-org/synapse/issues/7326), [\#7378](https://github.com/matrix-org/synapse/issues/7378), [\#7421](https://github.com/matrix-org/synapse/issues/7421)) +- Convert some of synapse.rest.media to async/await. ([\#7110](https://github.com/matrix-org/synapse/issues/7110), [\#7184](https://github.com/matrix-org/synapse/issues/7184), [\#7241](https://github.com/matrix-org/synapse/issues/7241)) +- De-duplicate / remove unused REST code for login and auth. ([\#7115](https://github.com/matrix-org/synapse/issues/7115)) +- Convert `*StreamRow` classes to inner classes. ([\#7116](https://github.com/matrix-org/synapse/issues/7116)) +- Clean up some LoggingContext code. ([\#7120](https://github.com/matrix-org/synapse/issues/7120), [\#7181](https://github.com/matrix-org/synapse/issues/7181), [\#7183](https://github.com/matrix-org/synapse/issues/7183), [\#7408](https://github.com/matrix-org/synapse/issues/7408), [\#7426](https://github.com/matrix-org/synapse/issues/7426)) +- Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage. ([\#7128](https://github.com/matrix-org/synapse/issues/7128)) +- Refactored the CAS authentication logic to a separate class. ([\#7136](https://github.com/matrix-org/synapse/issues/7136)) +- Run replication streamers on workers. ([\#7146](https://github.com/matrix-org/synapse/issues/7146)) +- Add tests for outbound device pokes. ([\#7157](https://github.com/matrix-org/synapse/issues/7157)) +- Fix device list update stream ids going backward. ([\#7158](https://github.com/matrix-org/synapse/issues/7158)) +- Use `stream.current_token()` and remove `stream_positions()`. ([\#7172](https://github.com/matrix-org/synapse/issues/7172)) +- Move client command handling out of TCP protocol. ([\#7185](https://github.com/matrix-org/synapse/issues/7185)) +- Move server command handling out of TCP protocol. ([\#7187](https://github.com/matrix-org/synapse/issues/7187)) +- Fix consistency of HTTP status codes reported in log lines. ([\#7188](https://github.com/matrix-org/synapse/issues/7188)) +- Only run one background database update at a time. ([\#7190](https://github.com/matrix-org/synapse/issues/7190)) +- Remove sent outbound device list pokes from the database. ([\#7192](https://github.com/matrix-org/synapse/issues/7192)) +- Add a background database update job to clear out duplicate `device_lists_outbound_pokes`. ([\#7193](https://github.com/matrix-org/synapse/issues/7193)) +- Remove some extraneous debugging log lines. ([\#7207](https://github.com/matrix-org/synapse/issues/7207)) +- Add explicit Python build tooling as dependencies for the snapcraft build. ([\#7213](https://github.com/matrix-org/synapse/issues/7213)) +- Add typing information to federation server code. ([\#7219](https://github.com/matrix-org/synapse/issues/7219)) +- Extend room admin api (`GET /_synapse/admin/v1/rooms`) with additional attributes. ([\#7225](https://github.com/matrix-org/synapse/issues/7225)) +- Unblacklist '/upgrade creates a new room' sytest for workers. ([\#7228](https://github.com/matrix-org/synapse/issues/7228)) +- Remove redundant checks on `daemonize` from synctl. ([\#7233](https://github.com/matrix-org/synapse/issues/7233)) +- Upgrade jQuery to v3.4.1 on fallback login/registration pages. ([\#7236](https://github.com/matrix-org/synapse/issues/7236)) +- Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible. ([\#7237](https://github.com/matrix-org/synapse/issues/7237)) +- Correct the parameters of a test fixture. Contributed by Isaiah Singletary. ([\#7243](https://github.com/matrix-org/synapse/issues/7243)) +- Convert auth handler to async/await. ([\#7261](https://github.com/matrix-org/synapse/issues/7261)) +- Add some unit tests for replication. ([\#7278](https://github.com/matrix-org/synapse/issues/7278)) +- Improve typing annotations in `synapse.replication.tcp.streams.Stream`. ([\#7291](https://github.com/matrix-org/synapse/issues/7291)) +- Reduce log verbosity of url cache cleanup tasks. ([\#7295](https://github.com/matrix-org/synapse/issues/7295)) +- Fix sample SAML Service Provider configuration. Contributed by @frcl. ([\#7300](https://github.com/matrix-org/synapse/issues/7300)) +- Fix StreamChangeCache to work with multiple entities changing on the same stream id. ([\#7303](https://github.com/matrix-org/synapse/issues/7303)) +- Fix an incorrect import in IdentityHandler. ([\#7319](https://github.com/matrix-org/synapse/issues/7319)) +- Reduce logging verbosity for successful federation requests. ([\#7321](https://github.com/matrix-org/synapse/issues/7321)) +- Convert some federation handler code to async/await. ([\#7338](https://github.com/matrix-org/synapse/issues/7338)) +- Fix collation for postgres for unit tests. ([\#7359](https://github.com/matrix-org/synapse/issues/7359)) +- Convert RegistrationWorkerStore.is_server_admin and dependent code to async/await. ([\#7363](https://github.com/matrix-org/synapse/issues/7363)) +- Add an `instance_name` to `RDATA` and `POSITION` replication commands. ([\#7364](https://github.com/matrix-org/synapse/issues/7364)) +- Thread through instance name to replication client. ([\#7369](https://github.com/matrix-org/synapse/issues/7369)) +- Convert synapse.server_notices to async/await. ([\#7394](https://github.com/matrix-org/synapse/issues/7394)) +- Convert synapse.notifier to async/await. ([\#7395](https://github.com/matrix-org/synapse/issues/7395)) +- Fix issues with the Python package manifest. ([\#7404](https://github.com/matrix-org/synapse/issues/7404)) +- Prevent methods in `synapse.handlers.auth` from polling the homeserver config every request. ([\#7420](https://github.com/matrix-org/synapse/issues/7420)) +- Speed up fetching device lists changes when handling `/sync` requests. ([\#7423](https://github.com/matrix-org/synapse/issues/7423)) +- Run group attestation renewal in series rather than parallel for performance. ([\#7442](https://github.com/matrix-org/synapse/issues/7442)) Synapse 1.12.4 (2020-04-23) diff --git a/UPGRADE.rst b/UPGRADE.rst index d1408be2af47..41c47e964d57 100644 --- a/UPGRADE.rst +++ b/UPGRADE.rst @@ -78,12 +78,13 @@ for example: Upgrading to v1.13.0 ==================== + Incorrect database migration in old synapse versions ---------------------------------------------------- A bug was introduced in Synapse 1.4.0 which could cause the room directory to -be incomplete or empty if Synapse was upgraded directly from v1.2.1 or earlier, -to versions between v1.4.0 and v1.12.x. +be incomplete or empty if Synapse was upgraded directly from v1.2.1 or +earlier, to versions between v1.4.0 and v1.12.x. This will *not* be a problem for Synapse installations which were: * created at v1.4.0 or later, @@ -105,6 +106,42 @@ affected can be repaired as follows: 2. Restart synapse. +New Single Sign-on HTML Templates +--------------------------------- + +New templates (``sso_auth_confirm.html``, ``sso_auth_success.html``, and +``sso_account_deactivated.html``) were added to Synapse. If your Synapse is +configured to use SSO and a custom ``sso_redirect_confirm_template_dir`` +configuration then these templates will need to be copied from +`synapse/res/templates `_ into that directory. + +Synapse SSO Plugins Method Deprecation +-------------------------------------- + +Plugins using the ``complete_sso_login`` method of +``synapse.module_api.ModuleApi`` should update to using the async/await +version ``complete_sso_login_async`` which includes additional checks. The +non-async version is considered deprecated. + +Rolling back to v1.12.4 after a failed upgrade +---------------------------------------------- + +v1.13.0 includes a lot of large changes. If something problematic occurs, you +may want to roll-back to a previous version of Synapse. Because v1.13.0 also +includes a new database schema version, reverting that version is also required +alongside the generic rollback instructions mentioned above. In short, to roll +back to v1.12.4 you need to: + +1. Stop the server +2. Decrease the schema version in the database: + + .. code:: sql + + UPDATE schema_version SET version = 57; + +3. Downgrade Synapse by following the instructions for your installation method + in the "Rolling back to older versions" section above. + Upgrading to v1.12.0 ==================== diff --git a/changelog.d/6446.misc b/changelog.d/6446.misc deleted file mode 100644 index c42df16f1aa3..000000000000 --- a/changelog.d/6446.misc +++ /dev/null @@ -1 +0,0 @@ -Add benchmarks for LruCache. diff --git a/changelog.d/6573.bugfix b/changelog.d/6573.bugfix deleted file mode 100644 index 1bb8014db795..000000000000 --- a/changelog.d/6573.bugfix +++ /dev/null @@ -1 +0,0 @@ -Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak. diff --git a/changelog.d/6634.bugfix b/changelog.d/6634.bugfix deleted file mode 100644 index ec48fdc0a085..000000000000 --- a/changelog.d/6634.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm. diff --git a/changelog.d/6639.bugfix b/changelog.d/6639.bugfix deleted file mode 100644 index c7593a6e8443..000000000000 --- a/changelog.d/6639.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix missing field `default` when fetching user-defined push rules. diff --git a/changelog.d/6881.misc b/changelog.d/6881.misc deleted file mode 100644 index 03b89ccd3d87..000000000000 --- a/changelog.d/6881.misc +++ /dev/null @@ -1 +0,0 @@ -Return total number of users and profile attributes in admin users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. diff --git a/changelog.d/6892.doc b/changelog.d/6892.doc deleted file mode 100644 index 0d04cf0bdb5d..000000000000 --- a/changelog.d/6892.doc +++ /dev/null @@ -1 +0,0 @@ -Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`. \ No newline at end of file diff --git a/changelog.d/6899.bugfix b/changelog.d/6899.bugfix deleted file mode 100644 index efa8a40b1f0b..000000000000 --- a/changelog.d/6899.bugfix +++ /dev/null @@ -1 +0,0 @@ -Improve error responses when accessing remote public room lists. \ No newline at end of file diff --git a/changelog.d/6946.bugfix b/changelog.d/6946.bugfix deleted file mode 100644 index a238c83a18f7..000000000000 --- a/changelog.d/6946.bugfix +++ /dev/null @@ -1 +0,0 @@ -Transfer alias mappings on room upgrade. \ No newline at end of file diff --git a/changelog.d/6988.doc b/changelog.d/6988.doc deleted file mode 100644 index b6f71bb96687..000000000000 --- a/changelog.d/6988.doc +++ /dev/null @@ -1 +0,0 @@ -Improve the documentation for database configuration. diff --git a/changelog.d/7006.feature b/changelog.d/7006.feature deleted file mode 100644 index d2ce9dbaca89..000000000000 --- a/changelog.d/7006.feature +++ /dev/null @@ -1 +0,0 @@ -Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata. diff --git a/changelog.d/7009.feature b/changelog.d/7009.feature deleted file mode 100644 index cd2705d5baa8..000000000000 --- a/changelog.d/7009.feature +++ /dev/null @@ -1 +0,0 @@ -Set `Referrer-Policy` header to `no-referrer` on media downloads. diff --git a/changelog.d/7010.misc b/changelog.d/7010.misc deleted file mode 100644 index 4ba1f6cdf8fc..000000000000 --- a/changelog.d/7010.misc +++ /dev/null @@ -1 +0,0 @@ -Change device list streams to have one row per ID. diff --git a/changelog.d/7011.misc b/changelog.d/7011.misc deleted file mode 100644 index 41c3b37574fa..000000000000 --- a/changelog.d/7011.misc +++ /dev/null @@ -1 +0,0 @@ -Remove concept of a non-limited stream. diff --git a/changelog.d/7024.misc b/changelog.d/7024.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7024.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7040.feature b/changelog.d/7040.feature deleted file mode 100644 index ce6140fdd111..000000000000 --- a/changelog.d/7040.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7051.feature b/changelog.d/7051.feature deleted file mode 100644 index 3e36a3f65e40..000000000000 --- a/changelog.d/7051.feature +++ /dev/null @@ -1 +0,0 @@ -Admin API `POST /_synapse/admin/v1/join/` to join users to a room like `auto_join_rooms` for creation of users. \ No newline at end of file diff --git a/changelog.d/7068.bugfix b/changelog.d/7068.bugfix deleted file mode 100644 index d1693a7f2248..000000000000 --- a/changelog.d/7068.bugfix +++ /dev/null @@ -1 +0,0 @@ -Ensure that a user inteactive authentication session is tied to a single request. diff --git a/changelog.d/7089.bugfix b/changelog.d/7089.bugfix deleted file mode 100644 index f1f440f23ae6..000000000000 --- a/changelog.d/7089.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors. diff --git a/changelog.d/7091.doc b/changelog.d/7091.doc deleted file mode 100644 index 463536c8128d..000000000000 --- a/changelog.d/7091.doc +++ /dev/null @@ -1 +0,0 @@ -Improve the documentation of application service configuration files. diff --git a/changelog.d/7096.feature b/changelog.d/7096.feature deleted file mode 100644 index 00f47b2a14a5..000000000000 --- a/changelog.d/7096.feature +++ /dev/null @@ -1 +0,0 @@ -Add options to prevent users from changing their profile or associated 3PIDs. \ No newline at end of file diff --git a/changelog.d/7102.feature b/changelog.d/7102.feature deleted file mode 100644 index 01057aa396ba..000000000000 --- a/changelog.d/7102.feature +++ /dev/null @@ -1 +0,0 @@ -Support SSO in the user interactive authentication workflow. diff --git a/changelog.d/7107.doc b/changelog.d/7107.doc deleted file mode 100644 index f6da32d406b4..000000000000 --- a/changelog.d/7107.doc +++ /dev/null @@ -1 +0,0 @@ -Update pre-built package name for FreeBSD. diff --git a/changelog.d/7109.bugfix b/changelog.d/7109.bugfix deleted file mode 100644 index 268de9978eed..000000000000 --- a/changelog.d/7109.bugfix +++ /dev/null @@ -1 +0,0 @@ -Return the proper error (M_BAD_ALIAS) when a non-existant canonical alias is provided. diff --git a/changelog.d/7110.misc b/changelog.d/7110.misc deleted file mode 100644 index fac5bc04032c..000000000000 --- a/changelog.d/7110.misc +++ /dev/null @@ -1 +0,0 @@ -Convert some of synapse.rest.media to async/await. diff --git a/changelog.d/7115.misc b/changelog.d/7115.misc deleted file mode 100644 index 7d4a011e3e4f..000000000000 --- a/changelog.d/7115.misc +++ /dev/null @@ -1 +0,0 @@ -De-duplicate / remove unused REST code for login and auth. diff --git a/changelog.d/7116.misc b/changelog.d/7116.misc deleted file mode 100644 index 89d90bd49ee5..000000000000 --- a/changelog.d/7116.misc +++ /dev/null @@ -1 +0,0 @@ -Convert `*StreamRow` classes to inner classes. diff --git a/changelog.d/7117.bugfix b/changelog.d/7117.bugfix deleted file mode 100644 index 1896d7ad4986..000000000000 --- a/changelog.d/7117.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug which meant that groups updates were not correctly replicated between workers. diff --git a/changelog.d/7118.feature b/changelog.d/7118.feature deleted file mode 100644 index 5cbfd981607b..000000000000 --- a/changelog.d/7118.feature +++ /dev/null @@ -1 +0,0 @@ -Allow server admins to define and enforce a password policy (MSC2000). \ No newline at end of file diff --git a/changelog.d/7119.doc b/changelog.d/7119.doc deleted file mode 100644 index 05192966c350..000000000000 --- a/changelog.d/7119.doc +++ /dev/null @@ -1 +0,0 @@ -Update postgres docs with login troubleshooting information. \ No newline at end of file diff --git a/changelog.d/7120.misc b/changelog.d/7120.misc deleted file mode 100644 index 731f4dcb52e4..000000000000 --- a/changelog.d/7120.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up some LoggingContext code. diff --git a/changelog.d/7128.misc b/changelog.d/7128.misc deleted file mode 100644 index 5703f6d2ecde..000000000000 --- a/changelog.d/7128.misc +++ /dev/null @@ -1 +0,0 @@ -Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage. diff --git a/changelog.d/7133.bugfix b/changelog.d/7133.bugfix deleted file mode 100644 index 61a86fd34e6e..000000000000 --- a/changelog.d/7133.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix starting workers when federation sending not split out. diff --git a/changelog.d/7136.misc b/changelog.d/7136.misc deleted file mode 100644 index 3f666d25fdea..000000000000 --- a/changelog.d/7136.misc +++ /dev/null @@ -1 +0,0 @@ -Refactored the CAS authentication logic to a separate class. diff --git a/changelog.d/7137.removal b/changelog.d/7137.removal deleted file mode 100644 index 75266a06bb3d..000000000000 --- a/changelog.d/7137.removal +++ /dev/null @@ -1 +0,0 @@ -Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`. \ No newline at end of file diff --git a/changelog.d/7141.doc b/changelog.d/7141.doc deleted file mode 100644 index 2fcbd666c29f..000000000000 --- a/changelog.d/7141.doc +++ /dev/null @@ -1 +0,0 @@ -Clean up INSTALL.md a bit. \ No newline at end of file diff --git a/changelog.d/7146.misc b/changelog.d/7146.misc deleted file mode 100644 index facde0695951..000000000000 --- a/changelog.d/7146.misc +++ /dev/null @@ -1 +0,0 @@ -Run replication streamers on workers. diff --git a/changelog.d/7147.doc b/changelog.d/7147.doc deleted file mode 100644 index 2c855ff5f7b3..000000000000 --- a/changelog.d/7147.doc +++ /dev/null @@ -1 +0,0 @@ -Add documentation for running a local CAS server for testing. diff --git a/changelog.d/7150.bugfix b/changelog.d/7150.bugfix deleted file mode 100644 index 1feb294799a9..000000000000 --- a/changelog.d/7150.bugfix +++ /dev/null @@ -1 +0,0 @@ -Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param. \ No newline at end of file diff --git a/changelog.d/7151.bugfix b/changelog.d/7151.bugfix deleted file mode 100644 index 8aaa2dc65971..000000000000 --- a/changelog.d/7151.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response. diff --git a/changelog.d/7152.feature b/changelog.d/7152.feature deleted file mode 100644 index fafa79c7e7f5..000000000000 --- a/changelog.d/7152.feature +++ /dev/null @@ -1 +0,0 @@ -Improve the support for SSO authentication on the login fallback page. diff --git a/changelog.d/7153.feature b/changelog.d/7153.feature deleted file mode 100644 index 414ebe1f6978..000000000000 --- a/changelog.d/7153.feature +++ /dev/null @@ -1 +0,0 @@ -Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set. diff --git a/changelog.d/7155.bugfix b/changelog.d/7155.bugfix deleted file mode 100644 index 0bf51e7aba34..000000000000 --- a/changelog.d/7155.bugfix +++ /dev/null @@ -1 +0,0 @@ -Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. diff --git a/changelog.d/7157.misc b/changelog.d/7157.misc deleted file mode 100644 index 0eb1128c7a42..000000000000 --- a/changelog.d/7157.misc +++ /dev/null @@ -1 +0,0 @@ -Add tests for outbound device pokes. diff --git a/changelog.d/7158.misc b/changelog.d/7158.misc deleted file mode 100644 index 269b8daeb086..000000000000 --- a/changelog.d/7158.misc +++ /dev/null @@ -1 +0,0 @@ -Fix device list update stream ids going backward. diff --git a/changelog.d/7159.bugfix b/changelog.d/7159.bugfix deleted file mode 100644 index 1b341b127b0c..000000000000 --- a/changelog.d/7159.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix excessive CPU usage by `prune_old_outbound_device_pokes` job. diff --git a/changelog.d/7167.doc b/changelog.d/7167.doc deleted file mode 100644 index a7e7ba9b51f5..000000000000 --- a/changelog.d/7167.doc +++ /dev/null @@ -1 +0,0 @@ -Improve README.md by being explicit about public IP recommendation for TURN relaying. diff --git a/changelog.d/7171.doc b/changelog.d/7171.doc deleted file mode 100644 index 25a3bd8ac6dc..000000000000 --- a/changelog.d/7171.doc +++ /dev/null @@ -1 +0,0 @@ -Fix a small typo in the `metrics_flags` config option. \ No newline at end of file diff --git a/changelog.d/7172.misc b/changelog.d/7172.misc deleted file mode 100644 index ffecdf97fe36..000000000000 --- a/changelog.d/7172.misc +++ /dev/null @@ -1 +0,0 @@ -Use `stream.current_token()` and remove `stream_positions()`. diff --git a/changelog.d/7177.bugfix b/changelog.d/7177.bugfix deleted file mode 100644 index 329a96cb0b65..000000000000 --- a/changelog.d/7177.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. \ No newline at end of file diff --git a/changelog.d/7178.bugfix b/changelog.d/7178.bugfix deleted file mode 100644 index 35ea645d7596..000000000000 --- a/changelog.d/7178.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug which could cause incorrect 'cyclic dependency' error. diff --git a/changelog.d/7181.misc b/changelog.d/7181.misc deleted file mode 100644 index 731f4dcb52e4..000000000000 --- a/changelog.d/7181.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up some LoggingContext code. diff --git a/changelog.d/7183.misc b/changelog.d/7183.misc deleted file mode 100644 index 731f4dcb52e4..000000000000 --- a/changelog.d/7183.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up some LoggingContext code. diff --git a/changelog.d/7184.misc b/changelog.d/7184.misc deleted file mode 100644 index fac5bc04032c..000000000000 --- a/changelog.d/7184.misc +++ /dev/null @@ -1 +0,0 @@ -Convert some of synapse.rest.media to async/await. diff --git a/changelog.d/7185.misc b/changelog.d/7185.misc deleted file mode 100644 index deb9ca702122..000000000000 --- a/changelog.d/7185.misc +++ /dev/null @@ -1 +0,0 @@ -Move client command handling out of TCP protocol. diff --git a/changelog.d/7186.feature b/changelog.d/7186.feature deleted file mode 100644 index 01057aa396ba..000000000000 --- a/changelog.d/7186.feature +++ /dev/null @@ -1 +0,0 @@ -Support SSO in the user interactive authentication workflow. diff --git a/changelog.d/7187.misc b/changelog.d/7187.misc deleted file mode 100644 index 60d68ae87704..000000000000 --- a/changelog.d/7187.misc +++ /dev/null @@ -1 +0,0 @@ -Move server command handling out of TCP protocol. diff --git a/changelog.d/7188.misc b/changelog.d/7188.misc deleted file mode 100644 index f72955b95b3b..000000000000 --- a/changelog.d/7188.misc +++ /dev/null @@ -1 +0,0 @@ -Fix consistency of HTTP status codes reported in log lines. diff --git a/changelog.d/7190.misc b/changelog.d/7190.misc deleted file mode 100644 index 34348873f171..000000000000 --- a/changelog.d/7190.misc +++ /dev/null @@ -1 +0,0 @@ -Only run one background database update at a time. diff --git a/changelog.d/7191.feature b/changelog.d/7191.feature deleted file mode 100644 index 83d5685bb2c4..000000000000 --- a/changelog.d/7191.feature +++ /dev/null @@ -1 +0,0 @@ -Admin users are no longer required to be in a room to create an alias for it. diff --git a/changelog.d/7192.misc b/changelog.d/7192.misc deleted file mode 100644 index e401e363997e..000000000000 --- a/changelog.d/7192.misc +++ /dev/null @@ -1 +0,0 @@ -Remove sent outbound device list pokes from the database. diff --git a/changelog.d/7193.misc b/changelog.d/7193.misc deleted file mode 100644 index 383a738e6497..000000000000 --- a/changelog.d/7193.misc +++ /dev/null @@ -1 +0,0 @@ -Add a background database update job to clear out duplicate `device_lists_outbound_pokes`. diff --git a/changelog.d/7195.misc b/changelog.d/7195.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7195.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7199.bugfix b/changelog.d/7199.bugfix deleted file mode 100644 index b234163ea823..000000000000 --- a/changelog.d/7199.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent. diff --git a/changelog.d/7203.bugfix b/changelog.d/7203.bugfix deleted file mode 100644 index 8b383952e53c..000000000000 --- a/changelog.d/7203.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix some worker-mode replication handling not being correctly recorded in CPU usage stats. diff --git a/changelog.d/7207.misc b/changelog.d/7207.misc deleted file mode 100644 index 4f9b6a1089a0..000000000000 --- a/changelog.d/7207.misc +++ /dev/null @@ -1 +0,0 @@ -Remove some extraneous debugging log lines. \ No newline at end of file diff --git a/changelog.d/7213.misc b/changelog.d/7213.misc deleted file mode 100644 index 03cbfb5f6269..000000000000 --- a/changelog.d/7213.misc +++ /dev/null @@ -1 +0,0 @@ -Add explicit Python build tooling as dependencies for the snapcraft build. diff --git a/changelog.d/7219.misc b/changelog.d/7219.misc deleted file mode 100644 index dbf7a530bea2..000000000000 --- a/changelog.d/7219.misc +++ /dev/null @@ -1 +0,0 @@ -Add typing annotations in `synapse.federation`. diff --git a/changelog.d/7225.misc b/changelog.d/7225.misc deleted file mode 100644 index 375e2a475fd4..000000000000 --- a/changelog.d/7225.misc +++ /dev/null @@ -1 +0,0 @@ -Extend room admin api (`GET /_synapse/admin/v1/rooms`) with additional attributes. \ No newline at end of file diff --git a/changelog.d/7226.misc b/changelog.d/7226.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7226.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7228.misc b/changelog.d/7228.misc deleted file mode 100644 index 50e206377fab..000000000000 --- a/changelog.d/7228.misc +++ /dev/null @@ -1 +0,0 @@ -Unblacklist '/upgrade creates a new room' sytest for workers. \ No newline at end of file diff --git a/changelog.d/7230.feature b/changelog.d/7230.feature deleted file mode 100644 index aab777648f21..000000000000 --- a/changelog.d/7230.feature +++ /dev/null @@ -1 +0,0 @@ -Require admin privileges to enable room encryption by default. This does not affect existing rooms. diff --git a/changelog.d/7233.misc b/changelog.d/7233.misc deleted file mode 100644 index d9ad582726ee..000000000000 --- a/changelog.d/7233.misc +++ /dev/null @@ -1 +0,0 @@ -Remove redundant checks on `daemonize` from synctl. diff --git a/changelog.d/7234.doc b/changelog.d/7234.doc deleted file mode 100644 index d284f1422b3f..000000000000 --- a/changelog.d/7234.doc +++ /dev/null @@ -1 +0,0 @@ -Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution. diff --git a/changelog.d/7235.feature b/changelog.d/7235.feature deleted file mode 100644 index fafa79c7e7f5..000000000000 --- a/changelog.d/7235.feature +++ /dev/null @@ -1 +0,0 @@ -Improve the support for SSO authentication on the login fallback page. diff --git a/changelog.d/7236.misc b/changelog.d/7236.misc deleted file mode 100644 index e4a2702b5483..000000000000 --- a/changelog.d/7236.misc +++ /dev/null @@ -1 +0,0 @@ -Upgrade jQuery to v3.4.1 on fallback login/registration pages. \ No newline at end of file diff --git a/changelog.d/7237.misc b/changelog.d/7237.misc deleted file mode 100644 index 92e67ea31fb9..000000000000 --- a/changelog.d/7237.misc +++ /dev/null @@ -1 +0,0 @@ -Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible. \ No newline at end of file diff --git a/changelog.d/7238.doc b/changelog.d/7238.doc deleted file mode 100644 index 0e3b4be428f6..000000000000 --- a/changelog.d/7238.doc +++ /dev/null @@ -1 +0,0 @@ -Add documentation to the `password_providers` config option. Add known password provider implementations to docs. \ No newline at end of file diff --git a/changelog.d/7239.misc b/changelog.d/7239.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7239.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7240.bugfix b/changelog.d/7240.bugfix deleted file mode 100644 index 83b18d3e1143..000000000000 --- a/changelog.d/7240.bugfix +++ /dev/null @@ -1 +0,0 @@ -Do not allow a deactivated user to login via SSO. diff --git a/changelog.d/7241.misc b/changelog.d/7241.misc deleted file mode 100644 index fac5bc04032c..000000000000 --- a/changelog.d/7241.misc +++ /dev/null @@ -1 +0,0 @@ -Convert some of synapse.rest.media to async/await. diff --git a/changelog.d/7243.misc b/changelog.d/7243.misc deleted file mode 100644 index a39c257a5431..000000000000 --- a/changelog.d/7243.misc +++ /dev/null @@ -1 +0,0 @@ -Correct the parameters of a test fixture. Contributed by Isaiah Singletary. diff --git a/changelog.d/7248.doc b/changelog.d/7248.doc deleted file mode 100644 index 0e3b4be428f6..000000000000 --- a/changelog.d/7248.doc +++ /dev/null @@ -1 +0,0 @@ -Add documentation to the `password_providers` config option. Add known password provider implementations to docs. \ No newline at end of file diff --git a/changelog.d/7249.bugfix b/changelog.d/7249.bugfix deleted file mode 100644 index 6ae700d36518..000000000000 --- a/changelog.d/7249.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix --help command-line argument. diff --git a/changelog.d/7251.doc b/changelog.d/7251.doc deleted file mode 100644 index 596a28e65dad..000000000000 --- a/changelog.d/7251.doc +++ /dev/null @@ -1 +0,0 @@ -Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev. diff --git a/changelog.d/7259.bugfix b/changelog.d/7259.bugfix deleted file mode 100644 index 55bb06be8c89..000000000000 --- a/changelog.d/7259.bugfix +++ /dev/null @@ -1 +0,0 @@ - Do not allow a deactivated user to login via SSO. diff --git a/changelog.d/7260.bugfix b/changelog.d/7260.bugfix deleted file mode 100644 index 9e50b56f2336..000000000000 --- a/changelog.d/7260.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix room publish permissions not being checked on room creation. diff --git a/changelog.d/7261.misc b/changelog.d/7261.misc deleted file mode 100644 index 88165f0105d5..000000000000 --- a/changelog.d/7261.misc +++ /dev/null @@ -1 +0,0 @@ -Convert auth handler to async/await. diff --git a/changelog.d/7265.feature b/changelog.d/7265.feature deleted file mode 100644 index 345b63e0b78b..000000000000 --- a/changelog.d/7265.feature +++ /dev/null @@ -1 +0,0 @@ -Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews. \ No newline at end of file diff --git a/changelog.d/7268.bugfix b/changelog.d/7268.bugfix deleted file mode 100644 index ab280da18edd..000000000000 --- a/changelog.d/7268.bugfix +++ /dev/null @@ -1 +0,0 @@ -Reject unknown session IDs during user interactive authentication instead of silently creating a new session. diff --git a/changelog.d/7272.doc b/changelog.d/7272.doc deleted file mode 100644 index 13a1ee340d46..000000000000 --- a/changelog.d/7272.doc +++ /dev/null @@ -1 +0,0 @@ -Documentation of media_storage_providers options updated to avoid misunderstandings. Contributed by Tristan Lins. \ No newline at end of file diff --git a/changelog.d/7274.bugfix b/changelog.d/7274.bugfix deleted file mode 100644 index 211a38befc01..000000000000 --- a/changelog.d/7274.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a sql query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log. diff --git a/changelog.d/7278.misc b/changelog.d/7278.misc deleted file mode 100644 index 8a4c4328f4a6..000000000000 --- a/changelog.d/7278.misc +++ /dev/null @@ -1 +0,0 @@ -Add some unit tests for replication. diff --git a/changelog.d/7279.feature b/changelog.d/7279.feature deleted file mode 100644 index 9aed0754744b..000000000000 --- a/changelog.d/7279.feature +++ /dev/null @@ -1 +0,0 @@ - Support SSO in the user interactive authentication workflow. diff --git a/changelog.d/7286.misc b/changelog.d/7286.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7286.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7290.misc b/changelog.d/7290.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7290.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7291.misc b/changelog.d/7291.misc deleted file mode 100644 index 02e7ae3fa2a9..000000000000 --- a/changelog.d/7291.misc +++ /dev/null @@ -1 +0,0 @@ -Improve typing annotations in `synapse.replication.tcp.streams.Stream`. diff --git a/changelog.d/7295.misc b/changelog.d/7295.misc deleted file mode 100644 index 239195e75c22..000000000000 --- a/changelog.d/7295.misc +++ /dev/null @@ -1 +0,0 @@ -Reduce log verbosity of url cache cleanup tasks. diff --git a/changelog.d/7300.misc b/changelog.d/7300.misc deleted file mode 100644 index 7b3bc362b822..000000000000 --- a/changelog.d/7300.misc +++ /dev/null @@ -1 +0,0 @@ -Fix sample SAML Service Provider configuration. Contributed by @frcl. diff --git a/changelog.d/7302.bugfix b/changelog.d/7302.bugfix deleted file mode 100644 index 820646d1f90d..000000000000 --- a/changelog.d/7302.bugfix +++ /dev/null @@ -1 +0,0 @@ -Persist user interactive authentication sessions across workers and Synapse restarts. diff --git a/changelog.d/7303.misc b/changelog.d/7303.misc deleted file mode 100644 index aa89c2b25444..000000000000 --- a/changelog.d/7303.misc +++ /dev/null @@ -1 +0,0 @@ -Fix StreamChangeCache to work with multiple entities changing on the same stream id. diff --git a/changelog.d/7315.feature b/changelog.d/7315.feature deleted file mode 100644 index ebcb4741b76f..000000000000 --- a/changelog.d/7315.feature +++ /dev/null @@ -1 +0,0 @@ -Allow `/requestToken` endpoints to hide the existence (or lack thereof) of 3PID associations on the homeserver. diff --git a/changelog.d/7316.bugfix b/changelog.d/7316.bugfix deleted file mode 100644 index 0692696c7bf8..000000000000 --- a/changelog.d/7316.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fixed backwards compatibility logic of the first value of `trusted_third_party_id_servers` being used for `account_threepid_delegates.email`, which occurs when the former, deprecated option is set and the latter is not. \ No newline at end of file diff --git a/changelog.d/7318.misc b/changelog.d/7318.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7318.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7319.misc b/changelog.d/7319.misc deleted file mode 100644 index 62ea6b6df966..000000000000 --- a/changelog.d/7319.misc +++ /dev/null @@ -1 +0,0 @@ -Fix an incorrect import in IdentityHandler. \ No newline at end of file diff --git a/changelog.d/7321.misc b/changelog.d/7321.misc deleted file mode 100644 index a4b3e67af9a4..000000000000 --- a/changelog.d/7321.misc +++ /dev/null @@ -1 +0,0 @@ -Reduce logging verbosity for successful federation requests. diff --git a/changelog.d/7325.feature b/changelog.d/7325.feature deleted file mode 100644 index ce6140fdd111..000000000000 --- a/changelog.d/7325.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7326.misc b/changelog.d/7326.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7326.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7337.bugfix b/changelog.d/7337.bugfix deleted file mode 100644 index f49c600173b9..000000000000 --- a/changelog.d/7337.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. diff --git a/changelog.d/7338.misc b/changelog.d/7338.misc deleted file mode 100644 index 7cafd074cad1..000000000000 --- a/changelog.d/7338.misc +++ /dev/null @@ -1 +0,0 @@ -Convert some federation handler code to async/await. diff --git a/changelog.d/7341.bugfix b/changelog.d/7341.bugfix deleted file mode 100644 index 8f0958bcb4b9..000000000000 --- a/changelog.d/7341.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix bad error handling that would cause Synapse to crash if it's provided with a YAML configuration file that's either empty or doesn't parse into a key-value map. diff --git a/changelog.d/7343.feature b/changelog.d/7343.feature deleted file mode 100644 index 01057aa396ba..000000000000 --- a/changelog.d/7343.feature +++ /dev/null @@ -1 +0,0 @@ -Support SSO in the user interactive authentication workflow. diff --git a/changelog.d/7344.bugfix b/changelog.d/7344.bugfix deleted file mode 100644 index 8c38f9ef8003..000000000000 --- a/changelog.d/7344.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix incorrect metrics reporting for `renew_attestations` background task. diff --git a/changelog.d/7352.feature b/changelog.d/7352.feature deleted file mode 100644 index ce6140fdd111..000000000000 --- a/changelog.d/7352.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7357.doc b/changelog.d/7357.doc deleted file mode 100644 index a3d5616ad263..000000000000 --- a/changelog.d/7357.doc +++ /dev/null @@ -1 +0,0 @@ -Add documentation on monitoring workers with Prometheus. diff --git a/changelog.d/7358.bugfix b/changelog.d/7358.bugfix deleted file mode 100644 index f49c600173b9..000000000000 --- a/changelog.d/7358.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. diff --git a/changelog.d/7359.misc b/changelog.d/7359.misc deleted file mode 100644 index b99f257d9ae5..000000000000 --- a/changelog.d/7359.misc +++ /dev/null @@ -1 +0,0 @@ -Fix collation for postgres for unit tests. diff --git a/changelog.d/7361.doc b/changelog.d/7361.doc deleted file mode 100644 index b35dbc36eeb2..000000000000 --- a/changelog.d/7361.doc +++ /dev/null @@ -1 +0,0 @@ -Clarify endpoint usage in the users admin api documentation. \ No newline at end of file diff --git a/changelog.d/7363.misc b/changelog.d/7363.misc deleted file mode 100644 index 1e3cddde79fd..000000000000 --- a/changelog.d/7363.misc +++ /dev/null @@ -1 +0,0 @@ -Convert RegistrationWorkerStore.is_server_admin and dependent code to async/await. \ No newline at end of file diff --git a/changelog.d/7364.misc b/changelog.d/7364.misc deleted file mode 100644 index bb5d727cf4fd..000000000000 --- a/changelog.d/7364.misc +++ /dev/null @@ -1 +0,0 @@ -Add an `instance_name` to `RDATA` and `POSITION` replication commands. diff --git a/changelog.d/7367.bugfix b/changelog.d/7367.bugfix deleted file mode 100644 index 12171d4e1cb6..000000000000 --- a/changelog.d/7367.bugfix +++ /dev/null @@ -1 +0,0 @@ -Prevent non-federating rooms from appearing in responses to federated `POST /publicRoom` requests when a filter was included. diff --git a/changelog.d/7368.bugfix b/changelog.d/7368.bugfix deleted file mode 100644 index efa8a40b1f0b..000000000000 --- a/changelog.d/7368.bugfix +++ /dev/null @@ -1 +0,0 @@ -Improve error responses when accessing remote public room lists. \ No newline at end of file diff --git a/changelog.d/7369.misc b/changelog.d/7369.misc deleted file mode 100644 index 060b09c888fd..000000000000 --- a/changelog.d/7369.misc +++ /dev/null @@ -1 +0,0 @@ -Thread through instance name to replication client. diff --git a/changelog.d/7378.misc b/changelog.d/7378.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7378.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7387.bugfix b/changelog.d/7387.bugfix deleted file mode 100644 index a250517b4958..000000000000 --- a/changelog.d/7387.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug which would cause the room durectory to be incorrectly populated if Synapse was upgraded directly from v1.2.1 or earlier to v1.4.0 or later. Note that this fix does not apply retrospectively; see the [upgrade notes](UPGRADE.rst#upgrading-to-v1130) for more information. diff --git a/changelog.d/7393.bugfix b/changelog.d/7393.bugfix deleted file mode 100644 index 74419af85812..000000000000 --- a/changelog.d/7393.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix bug in `EventContext.deserialize`. diff --git a/changelog.d/7394.misc b/changelog.d/7394.misc deleted file mode 100644 index f1390308b3d2..000000000000 --- a/changelog.d/7394.misc +++ /dev/null @@ -1 +0,0 @@ -Convert synapse.server_notices to async/await. diff --git a/changelog.d/7395.misc b/changelog.d/7395.misc deleted file mode 100644 index bc0ad59e2c9f..000000000000 --- a/changelog.d/7395.misc +++ /dev/null @@ -1 +0,0 @@ -Convert synapse.notifier to async/await. diff --git a/changelog.d/7401.feature b/changelog.d/7401.feature deleted file mode 100644 index ce6140fdd111..000000000000 --- a/changelog.d/7401.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7404.misc b/changelog.d/7404.misc deleted file mode 100644 index 9ac17958cc8e..000000000000 --- a/changelog.d/7404.misc +++ /dev/null @@ -1 +0,0 @@ -Fix issues with the Python package manifest. diff --git a/changelog.d/7408.misc b/changelog.d/7408.misc deleted file mode 100644 index 731f4dcb52e4..000000000000 --- a/changelog.d/7408.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up some LoggingContext code. diff --git a/changelog.d/7420.misc b/changelog.d/7420.misc deleted file mode 100644 index e834a9163e35..000000000000 --- a/changelog.d/7420.misc +++ /dev/null @@ -1 +0,0 @@ -Prevent methods in `synapse.handlers.auth` from polling the homeserver config every request. \ No newline at end of file diff --git a/changelog.d/7421.misc b/changelog.d/7421.misc deleted file mode 100644 index 676f285377f5..000000000000 --- a/changelog.d/7421.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7423.misc b/changelog.d/7423.misc deleted file mode 100644 index eb1767ac13b0..000000000000 --- a/changelog.d/7423.misc +++ /dev/null @@ -1 +0,0 @@ -Speed up fetching device lists changes when handling `/sync` requests. diff --git a/changelog.d/7427.feature b/changelog.d/7427.feature deleted file mode 100644 index ce6140fdd111..000000000000 --- a/changelog.d/7427.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/debian/build_virtualenv b/debian/build_virtualenv index d892fd5c9d9a..4c9aabcac386 100755 --- a/debian/build_virtualenv +++ b/debian/build_virtualenv @@ -36,7 +36,6 @@ esac dh_virtualenv \ --install-suffix "matrix-synapse" \ --builtin-venv \ - --setuptools \ --python "$SNAKE" \ --upgrade-pip \ --preinstall="lxml" \ diff --git a/debian/changelog b/debian/changelog index 8641571986e9..e7842d417416 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,16 +1,18 @@ -<<<<<<< HEAD -matrix-synapse-py3 (1.12.3ubuntu1) UNRELEASED; urgency=medium +matrix-synapse-py3 (1.13.0) stable; urgency=medium + [ Patrick Cloke ] * Add information about .well-known files to Debian installation scripts. - -- Patrick Cloke Mon, 06 Apr 2020 10:10:38 -0400 -======= + [ Synapse Packaging team ] + * New synapse release 1.13.0. + + -- Synapse Packaging team Tue, 19 May 2020 09:16:56 -0400 + matrix-synapse-py3 (1.12.4) stable; urgency=medium * New synapse release 1.12.4. -- Synapse Packaging team Thu, 23 Apr 2020 10:58:14 -0400 ->>>>>>> master matrix-synapse-py3 (1.12.3) stable; urgency=medium diff --git a/docker/Dockerfile-dhvirtualenv b/docker/Dockerfile-dhvirtualenv index ac9ebcfd88c2..579724685c2b 100644 --- a/docker/Dockerfile-dhvirtualenv +++ b/docker/Dockerfile-dhvirtualenv @@ -27,15 +27,16 @@ RUN env DEBIAN_FRONTEND=noninteractive apt-get install \ wget # fetch and unpack the package -RUN wget -q -O /dh-virtuenv-1.1.tar.gz https://github.com/spotify/dh-virtualenv/archive/1.1.tar.gz -RUN tar xvf /dh-virtuenv-1.1.tar.gz +RUN mkdir /dh-virtualenv +RUN wget -q -O /dh-virtualenv.tar.gz https://github.com/matrix-org/dh-virtualenv/archive/matrixorg-20200519.tar.gz +RUN tar -xv --strip-components=1 -C /dh-virtualenv -f /dh-virtualenv.tar.gz # install its build deps -RUN cd dh-virtualenv-1.1/ \ - && env DEBIAN_FRONTEND=noninteractive mk-build-deps -ri -t "apt-get -yqq --no-install-recommends" +RUN cd /dh-virtualenv \ + && env DEBIAN_FRONTEND=noninteractive mk-build-deps -ri -t "apt-get -y --no-install-recommends" # build it -RUN cd dh-virtualenv-1.1 && dpkg-buildpackage -us -uc -b +RUN cd /dh-virtualenv && dpkg-buildpackage -us -uc -b ### ### Stage 1 @@ -68,12 +69,12 @@ RUN apt-get update -qq -o Acquire::Languages=none \ sqlite3 \ libpq-dev -COPY --from=builder /dh-virtualenv_1.1-1_all.deb / +COPY --from=builder /dh-virtualenv_1.2~dev-1_all.deb / # install dhvirtualenv. Update the apt cache again first, in case we got a # cached cache from docker the first time. RUN apt-get update -qq -o Acquire::Languages=none \ - && apt-get install -yq /dh-virtualenv_1.1-1_all.deb + && apt-get install -yq /dh-virtualenv_1.2~dev-1_all.deb WORKDIR /synapse/source ENTRYPOINT ["bash","/synapse/source/docker/build_debian.sh"] diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml index fc970986c63f..98ead7dc0e6e 100644 --- a/docs/sample_config.yaml +++ b/docs/sample_config.yaml @@ -253,6 +253,18 @@ listeners: # bind_addresses: ['::1', '127.0.0.1'] # type: manhole +# Forward extremities can build up in a room due to networking delays between +# homeservers. Once this happens in a large room, calculation of the state of +# that room can become quite expensive. To mitigate this, once the number of +# forward extremities reaches a given threshold, Synapse will send an +# org.matrix.dummy_event event, which will reduce the forward extremities +# in the room. +# +# This setting defines the threshold (i.e. number of forward extremities in the +# room) at which dummy events are sent. The default value is 10. +# +#dummy_events_threshold: 5 + ## Homeserver blocking ## diff --git a/docs/workers.md b/docs/workers.md index cc0b23197ffd..7512eff43a77 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -1,23 +1,31 @@ # Scaling synapse via workers -Synapse has experimental support for splitting out functionality into -multiple separate python processes, helping greatly with scalability. These +For small instances it recommended to run Synapse in monolith mode (the +default). For larger instances where performance is a concern it can be helpful +to split out functionality into multiple separate python processes. These processes are called 'workers', and are (eventually) intended to scale horizontally independently. -All of the below is highly experimental and subject to change as Synapse evolves, -but documenting it here to help folks needing highly scalable Synapses similar -to the one running matrix.org! +Synapse's worker support is under active development and subject to change as +we attempt to rapidly scale ever larger Synapse instances. However we are +documenting it here to help admins needing a highly scalable Synapse instance +similar to the one running `matrix.org`. -All processes continue to share the same database instance, and as such, workers -only work with postgres based synapse deployments (sharing a single sqlite -across multiple processes is a recipe for disaster, plus you should be using -postgres anyway if you care about scalability). +All processes continue to share the same database instance, and as such, +workers only work with PostgreSQL-based Synapse deployments. SQLite should only +be used for demo purposes and any admin considering workers should already be +running PostgreSQL. -The workers communicate with the master synapse process via a synapse-specific -TCP protocol called 'replication' - analogous to MySQL or Postgres style -database replication; feeding a stream of relevant data to the workers so they -can be kept in sync with the main synapse process and database state. +## Master/worker communication + +The workers communicate with the master process via a Synapse-specific protocol +called 'replication' (analogous to MySQL- or Postgres-style database +replication) which feeds a stream of relevant data from the master to the +workers so they can be kept in sync with the master process and database state. + +Additionally, workers may make HTTP requests to the master, to send information +in the other direction. Typically this is used for operations which need to +wait for a reply - such as sending an event. ## Configuration @@ -27,66 +35,61 @@ the correct worker, or to the main synapse instance. Note that this includes requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse proxy. -To enable workers, you need to add two replication listeners to the master -synapse, e.g.: - - listeners: - # The TCP replication port - - port: 9092 - bind_address: '127.0.0.1' - type: replication - # The HTTP replication port - - port: 9093 - bind_address: '127.0.0.1' - type: http - resources: - - names: [replication] +To enable workers, you need to add *two* replication listeners to the +main Synapse configuration file (`homeserver.yaml`). For example: -Under **no circumstances** should these replication API listeners be exposed to -the public internet; it currently implements no authentication whatsoever and is -unencrypted. - -(Roughly, the TCP port is used for streaming data from the master to the -workers, and the HTTP port for the workers to send data to the main -synapse process.) +```yaml +listeners: + # The TCP replication port + - port: 9092 + bind_address: '127.0.0.1' + type: replication + + # The HTTP replication port + - port: 9093 + bind_address: '127.0.0.1' + type: http + resources: + - names: [replication] +``` -You then create a set of configs for the various worker processes. These -should be worker configuration files, and should be stored in a dedicated -subdirectory, to allow synctl to manipulate them. +Under **no circumstances** should these replication API listeners be exposed to +the public internet; they have no authentication and are unencrypted. -Each worker configuration file inherits the configuration of the main homeserver -configuration file. You can then override configuration specific to that worker, -e.g. the HTTP listener that it provides (if any); logging configuration; etc. -You should minimise the number of overrides though to maintain a usable config. +You should then create a set of configs for the various worker processes. Each +worker configuration file inherits the configuration of the main homeserver +configuration file. You can then override configuration specific to that +worker, e.g. the HTTP listener that it provides (if any); logging +configuration; etc. You should minimise the number of overrides though to +maintain a usable config. In the config file for each worker, you must specify the type of worker application (`worker_app`). The currently available worker applications are -listed below. You must also specify the replication endpoints that it's talking -to on the main synapse process. `worker_replication_host` should specify the -host of the main synapse, `worker_replication_port` should point to the TCP +listed below. You must also specify the replication endpoints that it should +talk to on the main synapse process. `worker_replication_host` should specify +the host of the main synapse, `worker_replication_port` should point to the TCP replication listener port and `worker_replication_http_port` should point to the HTTP replication port. -Currently, the `event_creator` and `federation_reader` workers require specifying -`worker_replication_http_port`. - -For instance: +For example: - worker_app: synapse.app.synchrotron +```yaml +worker_app: synapse.app.synchrotron - # The replication listener on the synapse to talk to. - worker_replication_host: 127.0.0.1 - worker_replication_port: 9092 - worker_replication_http_port: 9093 +# The replication listener on the synapse to talk to. +worker_replication_host: 127.0.0.1 +worker_replication_port: 9092 +worker_replication_http_port: 9093 - worker_listeners: - - type: http - port: 8083 - resources: - - names: - - client +worker_listeners: + - type: http + port: 8083 + resources: + - names: + - client - worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml +worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml +``` ...is a full configuration for a synchrotron worker instance, which will expose a plain HTTP `/sync` endpoint on port 8083 separately from the `/sync` endpoint provided @@ -101,6 +104,50 @@ recommend the use of `systemd` where available: for information on setting up `systemd` to start synapse workers, see [systemd-with-workers](systemd-with-workers). To use `synctl`, see below. +### **Experimental** support for replication over redis + +As of Synapse v1.13.0, it is possible to configure Synapse to send replication +via a [Redis pub/sub channel](https://redis.io/topics/pubsub). This is an +alternative to direct TCP connections to the master: rather than all the +workers connecting to the master, all the workers and the master connect to +Redis, which relays replication commands between processes. This can give a +significant cpu saving on the master and will be a prerequisite for upcoming +performance improvements. + +Note that this support is currently experimental; you may experience lost +messages and similar problems! It is strongly recommended that admins setting +up workers for the first time use direct TCP replication as above. + +To configure Synapse to use Redis: + +1. Install Redis following the normal procedure for your distribution - for + example, on Debian, `apt install redis-server`. (It is safe to use an + existing Redis deployment if you have one: we use a pub/sub stream named + according to the `server_name` of your synapse server.) +2. Check Redis is running and accessible: you should be able to `echo PING | nc -q1 + localhost 6379` and get a response of `+PONG`. +3. Install the python prerequisites. If you installed synapse into a + virtualenv, this can be done with: + ```sh + pip install matrix-synapse[redis] + ``` + The debian packages from matrix.org already include the required + dependencies. +4. Add config to the shared configuration (`homeserver.yaml`): + ```yaml + redis: + enabled: true + ``` + Optional parameters which can go alongside `enabled` are `host`, `port`, + `password`. Normally none of these are required. +5. Restart master and all workers. + +Once redis replication is in use, `worker_replication_port` is redundant and +can be removed from the worker configuration files. Similarly, the +configuration for the `listener` for the TCP replication port can be removed +from the main configuration file. Note that the HTTP replication port is +still required. + ### Using synctl If you want to use `synctl` to manage your synapse processes, you will need to diff --git a/scripts-dev/build_debian_packages b/scripts-dev/build_debian_packages index 84eaec6a9512..ae2145d71723 100755 --- a/scripts-dev/build_debian_packages +++ b/scripts-dev/build_debian_packages @@ -27,6 +27,7 @@ DISTS = ( "ubuntu:cosmic", "ubuntu:disco", "ubuntu:eoan", + "ubuntu:focal", ) DESC = '''\ diff --git a/synapse/__init__.py b/synapse/__init__.py index d8d340f4268a..0abf4911729d 100644 --- a/synapse/__init__.py +++ b/synapse/__init__.py @@ -36,7 +36,7 @@ except ImportError: pass -__version__ = "1.12.4" +__version__ = "1.13.0" if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)): # We import here so that we don't have to install a bunch of deps when diff --git a/synapse/app/_base.py b/synapse/app/_base.py index 628292b890b1..dedff81af3bd 100644 --- a/synapse/app/_base.py +++ b/synapse/app/_base.py @@ -22,6 +22,7 @@ import traceback from daemonize import Daemonize +from typing_extensions import NoReturn from twisted.internet import defer, error, reactor from twisted.protocols.tls import TLSMemoryBIOFactory @@ -139,9 +140,9 @@ def run(): run() -def quit_with_error(error_string): +def quit_with_error(error_string: str) -> NoReturn: message_lines = error_string.split("\n") - line_length = max(len(l) for l in message_lines if len(l) < 80) + 2 + line_length = max(len(line) for line in message_lines if len(line) < 80) + 2 sys.stderr.write("*" * line_length + "\n") for line in message_lines: sys.stderr.write(" %s\n" % (line.rstrip(),)) diff --git a/synapse/config/redis.py b/synapse/config/redis.py index 81a27619ec7a..d5d3ca1c9e1e 100644 --- a/synapse/config/redis.py +++ b/synapse/config/redis.py @@ -31,5 +31,4 @@ def read_config(self, config, **kwargs): self.redis_host = redis_config.get("host", "localhost") self.redis_port = redis_config.get("port", 6379) - self.redis_dbid = redis_config.get("dbid") self.redis_password = redis_config.get("password") diff --git a/synapse/config/server.py b/synapse/config/server.py index c6d58effd419..ed28da3deb9b 100644 --- a/synapse/config/server.py +++ b/synapse/config/server.py @@ -505,6 +505,9 @@ class LimitRemoteRoomsConfig(object): "cleanup_extremities_with_dummy_events", True ) + # The number of forward extremities in a room needed to send a dummy event. + self.dummy_events_threshold = config.get("dummy_events_threshold", 10) + self.enable_ephemeral_messages = config.get("enable_ephemeral_messages", False) # Inhibits the /requestToken endpoints from returning an error that might leak @@ -519,7 +522,7 @@ class LimitRemoteRoomsConfig(object): ) def has_tls_listener(self) -> bool: - return any(l["tls"] for l in self.listeners) + return any(listener["tls"] for listener in self.listeners) def generate_config_section( self, server_name, data_dir_path, open_private_ports, listeners, **kwargs @@ -823,6 +826,18 @@ def generate_config_section( # bind_addresses: ['::1', '127.0.0.1'] # type: manhole + # Forward extremities can build up in a room due to networking delays between + # homeservers. Once this happens in a large room, calculation of the state of + # that room can become quite expensive. To mitigate this, once the number of + # forward extremities reaches a given threshold, Synapse will send an + # org.matrix.dummy_event event, which will reduce the forward extremities + # in the room. + # + # This setting defines the threshold (i.e. number of forward extremities in the + # room) at which dummy events are sent. The default value is 10. + # + #dummy_events_threshold: 5 + ## Homeserver blocking ## diff --git a/synapse/groups/attestations.py b/synapse/groups/attestations.py index 1eec3874b600..27b0c026556c 100644 --- a/synapse/groups/attestations.py +++ b/synapse/groups/attestations.py @@ -46,7 +46,6 @@ from synapse.api.errors import HttpResponseException, RequestSendFailed, SynapseError from synapse.metrics.background_process_metrics import run_as_background_process from synapse.types import get_domain_from_id -from synapse.util.async_helpers import yieldable_gather_results logger = logging.getLogger(__name__) @@ -208,6 +207,5 @@ def _renew_attestation(group_user: Tuple[str, str]): "Error renewing attestation of %r in %r", user_id, group_id ) - await yieldable_gather_results( - _renew_attestation, ((row["group_id"], row["user_id"]) for row in rows) - ) + for row in rows: + await _renew_attestation((row["group_id"], row["user_id"])) diff --git a/synapse/handlers/auth.py b/synapse/handlers/auth.py index 7613e5b6ab3b..5c20e2917114 100644 --- a/synapse/handlers/auth.py +++ b/synapse/handlers/auth.py @@ -317,30 +317,55 @@ async def check_auth( except StoreError: raise SynapseError(400, "Unknown session ID: %s" % (sid,)) + # If the client provides parameters, update what is persisted, + # otherwise use whatever was last provided. + # + # This was designed to allow the client to omit the parameters + # and just supply the session in subsequent calls so it split + # auth between devices by just sharing the session, (eg. so you + # could continue registration from your phone having clicked the + # email auth link on there). It's probably too open to abuse + # because it lets unauthenticated clients store arbitrary objects + # on a homeserver. + # + # Revisit: Assuming the REST APIs do sensible validation, the data + # isn't arbitrary. + # + # Note that the registration endpoint explicitly removes the + # "initial_device_display_name" parameter if it is provided + # without a "password" parameter. See the changes to + # synapse.rest.client.v2_alpha.register.RegisterRestServlet.on_POST + # in commit 544722bad23fc31056b9240189c3cbbbf0ffd3f9. if not clientdict: - # This was designed to allow the client to omit the parameters - # and just supply the session in subsequent calls so it split - # auth between devices by just sharing the session, (eg. so you - # could continue registration from your phone having clicked the - # email auth link on there). It's probably too open to abuse - # because it lets unauthenticated clients store arbitrary objects - # on a homeserver. - # Revisit: Assuming the REST APIs do sensible validation, the data - # isn't arbitrary. clientdict = session.clientdict # Ensure that the queried operation does not vary between stages of # the UI authentication session. This is done by generating a stable - # comparator based on the URI, method, and body (minus the auth dict) - # and storing it during the initial query. Subsequent queries ensure - # that this comparator has not changed. - comparator = (uri, method, clientdict) - if (session.uri, session.method, session.clientdict) != comparator: + # comparator and storing it during the initial query. Subsequent + # queries ensure that this comparator has not changed. + # + # The comparator is based on the requested URI and HTTP method. The + # client dict (minus the auth dict) should also be checked, but some + # clients are not spec compliant, just warn for now if the client + # dict changes. + if (session.uri, session.method) != (uri, method): raise SynapseError( 403, "Requested operation has changed during the UI authentication session.", ) + if session.clientdict != clientdict: + logger.warning( + "Requested operation has changed during the UI " + "authentication session. A future version of Synapse " + "will remove this capability." + ) + + # For backwards compatibility, changes to the client dict are + # persisted as clients modify them throughout their user interactive + # authentication flow. + await self.store.set_ui_auth_clientdict(sid, clientdict) + if not authdict: raise InteractiveAuthIncompleteError( self._auth_dict_for_flows(flows, session.session_id) diff --git a/synapse/handlers/message.py b/synapse/handlers/message.py index a324f09340ab..a622a600b480 100644 --- a/synapse/handlers/message.py +++ b/synapse/handlers/message.py @@ -419,6 +419,8 @@ def __init__(self, hs): self._ephemeral_events_enabled = hs.config.enable_ephemeral_messages + self._dummy_events_threshold = hs.config.dummy_events_threshold + @defer.inlineCallbacks def create_event( self, @@ -1085,7 +1087,7 @@ async def _send_dummy_events_to_fill_extremities(self): """ self._expire_rooms_to_exclude_from_dummy_event_insertion() room_ids = await self.store.get_rooms_with_many_extremities( - min_count=10, + min_count=self._dummy_events_threshold, limit=5, room_id_filter=self._rooms_to_exclude_from_dummy_event_insertion.keys(), ) diff --git a/synapse/handlers/register.py b/synapse/handlers/register.py index 1e6bdac0add3..a6178e74a19b 100644 --- a/synapse/handlers/register.py +++ b/synapse/handlers/register.py @@ -132,7 +132,7 @@ def check_username(self, localpart, guest_access_token=None, assigned_user_id=No def register_user( self, localpart=None, - password=None, + password_hash=None, guest_access_token=None, make_guest=False, admin=False, @@ -147,7 +147,7 @@ def register_user( Args: localpart: The local part of the user ID to register. If None, one will be generated. - password (unicode): The password to assign to this user so they can + password_hash (str|None): The hashed password to assign to this user so they can login again. This can be None which means they cannot login again via a password (e.g. the user is an application service user). user_type (str|None): type of user. One of the values from @@ -164,11 +164,6 @@ def register_user( yield self.check_registration_ratelimit(address) yield self.auth.check_auth_blocking(threepid=threepid) - password_hash = None - if password: - password_hash = yield defer.ensureDeferred( - self._auth_handler.hash(password) - ) if localpart is not None: yield self.check_username(localpart, guest_access_token=guest_access_token) diff --git a/synapse/logging/context.py b/synapse/logging/context.py index 856534e91a75..8b9c4e38bd9f 100644 --- a/synapse/logging/context.py +++ b/synapse/logging/context.py @@ -431,15 +431,7 @@ def stop(self, rusage: "Optional[resource._RUsage]") -> None: return utime_delta, stime_delta = self._get_cputime(rusage) - self._resource_usage.ru_utime += utime_delta - self._resource_usage.ru_stime += stime_delta - - # if we have a parent, pass our CPU usage stats on - if self.parent_context: - self.parent_context._resource_usage += self._resource_usage - - # reset them in case we get entered again - self._resource_usage.reset() + self.add_cputime(utime_delta, stime_delta) finally: self.usage_start = None @@ -497,30 +489,52 @@ def _get_cputime(self, current: "resource._RUsage") -> Tuple[float, float]: return utime_delta, stime_delta + def add_cputime(self, utime_delta: float, stime_delta: float) -> None: + """Update the CPU time usage of this context (and any parents, recursively). + + Args: + utime_delta: additional user time, in seconds, spent in this context. + stime_delta: additional system time, in seconds, spent in this context. + """ + self._resource_usage.ru_utime += utime_delta + self._resource_usage.ru_stime += stime_delta + if self.parent_context: + self.parent_context.add_cputime(utime_delta, stime_delta) + def add_database_transaction(self, duration_sec: float) -> None: + """Record the use of a database transaction and the length of time it took. + + Args: + duration_sec: The number of seconds the database transaction took. + """ if duration_sec < 0: raise ValueError("DB txn time can only be non-negative") self._resource_usage.db_txn_count += 1 self._resource_usage.db_txn_duration_sec += duration_sec + if self.parent_context: + self.parent_context.add_database_transaction(duration_sec) def add_database_scheduled(self, sched_sec: float) -> None: """Record a use of the database pool Args: - sched_sec (float): number of seconds it took us to get a - connection + sched_sec: number of seconds it took us to get a connection """ if sched_sec < 0: raise ValueError("DB scheduling time can only be non-negative") self._resource_usage.db_sched_duration_sec += sched_sec + if self.parent_context: + self.parent_context.add_database_scheduled(sched_sec) def record_event_fetch(self, event_count: int) -> None: """Record a number of events being fetched from the db Args: - event_count (int): number of events being fetched + event_count: number of events being fetched """ self._resource_usage.evt_db_fetch_count += event_count + if self.parent_context: + self.parent_context.record_event_fetch(event_count) class LoggingContextFilter(logging.Filter): diff --git a/synapse/notifier.py b/synapse/notifier.py index 71d9ed62b014..87c120a59ce1 100644 --- a/synapse/notifier.py +++ b/synapse/notifier.py @@ -15,7 +15,7 @@ import logging from collections import namedtuple -from typing import Callable, List +from typing import Callable, Iterable, List, TypeVar from prometheus_client import Counter @@ -42,12 +42,14 @@ "synapse_notifier_users_woken_by_stream", "", ["stream"] ) +T = TypeVar("T") + # TODO(paul): Should be shared somewhere -def count(func, l): - """Return the number of items in l for which func returns true.""" +def count(func: Callable[[T], bool], it: Iterable[T]) -> int: + """Return the number of items in it for which func returns true.""" n = 0 - for x in l: + for x in it: if func(x): n += 1 return n diff --git a/synapse/push/mailer.py b/synapse/push/mailer.py index 73580c1c6cc8..ab33abbeed83 100644 --- a/synapse/push/mailer.py +++ b/synapse/push/mailer.py @@ -19,6 +19,7 @@ import time from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText +from typing import Iterable, List, TypeVar from six.moves import urllib @@ -41,6 +42,8 @@ logger = logging.getLogger(__name__) +T = TypeVar("T") + MESSAGE_FROM_PERSON_IN_ROOM = ( "You have a message on %(app)s from %(person)s in the %(room)s room..." @@ -638,10 +641,10 @@ def safe_text(raw_text): ) -def deduped_ordered_list(l): +def deduped_ordered_list(it: Iterable[T]) -> List[T]: seen = set() ret = [] - for item in l: + for item in it: if item not in seen: seen.add(item) ret.append(item) diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py index 733c51b75841..39c99a280230 100644 --- a/synapse/python_dependencies.py +++ b/synapse/python_dependencies.py @@ -98,7 +98,9 @@ "sentry": ["sentry-sdk>=0.7.2"], "opentracing": ["jaeger-client>=4.0.0", "opentracing>=2.2.0"], "jwt": ["pyjwt>=1.6.4"], - "redis": ["txredisapi>=1.4.7"], + # hiredis is not a *strict* dependency, but it makes things much faster. + # (if it is not installed, we fall back to slow code.) + "redis": ["txredisapi>=1.4.7", "hiredis"], } ALL_OPTIONAL_REQUIREMENTS = set() # type: Set[str] diff --git a/synapse/replication/tcp/handler.py b/synapse/replication/tcp/handler.py index b14a3d9fcaac..4328b38e9dfe 100644 --- a/synapse/replication/tcp/handler.py +++ b/synapse/replication/tcp/handler.py @@ -131,10 +131,9 @@ def start_replication(self, hs): import txredisapi logger.info( - "Connecting to redis (host=%r port=%r DBID=%r)", + "Connecting to redis (host=%r port=%r)", hs.config.redis_host, hs.config.redis_port, - hs.config.redis_dbid, ) # We need two connections to redis, one for the subscription stream and @@ -145,7 +144,6 @@ def start_replication(self, hs): outbound_redis_connection = txredisapi.lazyConnection( host=hs.config.redis_host, port=hs.config.redis_port, - dbid=hs.config.redis_dbid, password=hs.config.redis.redis_password, reconnect=True, ) diff --git a/synapse/replication/tcp/redis.py b/synapse/replication/tcp/redis.py index db69f9255719..55bfa71dfdc0 100644 --- a/synapse/replication/tcp/redis.py +++ b/synapse/replication/tcp/redis.py @@ -96,7 +96,7 @@ def messageReceived(self, pattern: str, channel: str, message: str): cmd = parse_command_from_line(message) except Exception: logger.exception( - "[%s] failed to parse line: %r", message, + "Failed to parse replication line: %r", message, ) return diff --git a/synapse/rest/admin/users.py b/synapse/rest/admin/users.py index 593ce011e888..326682fbdb67 100644 --- a/synapse/rest/admin/users.py +++ b/synapse/rest/admin/users.py @@ -243,11 +243,11 @@ async def on_PUT(self, request, user_id): else: # create user password = body.get("password") - if password is not None and ( - not isinstance(body["password"], text_type) - or len(body["password"]) > 512 - ): - raise SynapseError(400, "Invalid password") + password_hash = None + if password is not None: + if not isinstance(password, text_type) or len(password) > 512: + raise SynapseError(400, "Invalid password") + password_hash = await self.auth_handler.hash(password) admin = body.get("admin", None) user_type = body.get("user_type", None) @@ -259,7 +259,7 @@ async def on_PUT(self, request, user_id): user_id = await self.registration_handler.register_user( localpart=target_user.localpart, - password=password, + password_hash=password_hash, admin=bool(admin), default_display_name=displayname, user_type=user_type, @@ -298,7 +298,7 @@ class UserRegisterServlet(RestServlet): NONCE_TIMEOUT = 60 def __init__(self, hs): - self.handlers = hs.get_handlers() + self.auth_handler = hs.get_auth_handler() self.reactor = hs.get_reactor() self.nonces = {} self.hs = hs @@ -362,16 +362,16 @@ async def on_POST(self, request): 400, "password must be specified", errcode=Codes.BAD_JSON ) else: - if ( - not isinstance(body["password"], text_type) - or len(body["password"]) > 512 - ): + password = body["password"] + if not isinstance(password, text_type) or len(password) > 512: raise SynapseError(400, "Invalid password") - password = body["password"].encode("utf-8") - if b"\x00" in password: + password_bytes = password.encode("utf-8") + if b"\x00" in password_bytes: raise SynapseError(400, "Invalid password") + password_hash = await self.auth_handler.hash(password) + admin = body.get("admin", None) user_type = body.get("user_type", None) @@ -388,7 +388,7 @@ async def on_POST(self, request): want_mac_builder.update(b"\x00") want_mac_builder.update(username) want_mac_builder.update(b"\x00") - want_mac_builder.update(password) + want_mac_builder.update(password_bytes) want_mac_builder.update(b"\x00") want_mac_builder.update(b"admin" if admin else b"notadmin") if user_type: @@ -407,7 +407,7 @@ async def on_POST(self, request): user_id = await register.registration_handler.register_user( localpart=body["username"].lower(), - password=body["password"], + password_hash=password_hash, admin=bool(admin), user_type=user_type, ) diff --git a/synapse/rest/client/v2_alpha/register.py b/synapse/rest/client/v2_alpha/register.py index af08cc6cce82..c26927f27b9e 100644 --- a/synapse/rest/client/v2_alpha/register.py +++ b/synapse/rest/client/v2_alpha/register.py @@ -426,12 +426,16 @@ async def on_POST(self, request): # we do basic sanity checks here because the auth layer will store these # in sessions. Pull out the username/password provided to us. if "password" in body: - if ( - not isinstance(body["password"], string_types) - or len(body["password"]) > 512 - ): + password = body.pop("password") + if not isinstance(password, string_types) or len(password) > 512: raise SynapseError(400, "Invalid password") - self.password_policy_handler.validate_password(body["password"]) + self.password_policy_handler.validate_password(password) + + # If the password is valid, hash it and store it back on the request. + # This ensures the hashed password is handled everywhere. + if "password_hash" in body: + raise SynapseError(400, "Unexpected property: password_hash") + body["password_hash"] = await self.auth_handler.hash(password) desired_username = None if "username" in body: @@ -484,7 +488,7 @@ async def on_POST(self, request): guest_access_token = body.get("guest_access_token", None) - if "initial_device_display_name" in body and "password" not in body: + if "initial_device_display_name" in body and "password_hash" not in body: # ignore 'initial_device_display_name' if sent without # a password to work around a client bug where it sent # the 'initial_device_display_name' param alone, wiping out @@ -546,11 +550,11 @@ async def on_POST(self, request): registered = False else: # NB: This may be from the auth handler and NOT from the POST - assert_params_in_dict(params, ["password"]) + assert_params_in_dict(params, ["password_hash"]) desired_username = params.get("username", None) guest_access_token = params.get("guest_access_token", None) - new_password = params.get("password", None) + new_password_hash = params.get("password_hash", None) if desired_username is not None: desired_username = desired_username.lower() @@ -583,7 +587,7 @@ async def on_POST(self, request): registered_user_id = await self.registration_handler.register_user( localpart=desired_username, - password=new_password, + password_hash=new_password_hash, guest_access_token=guest_access_token, threepid=threepid, address=client_addr, diff --git a/synapse/storage/data_stores/main/roommember.py b/synapse/storage/data_stores/main/roommember.py index 62ef8bab4a0d..44c586be7f2a 100644 --- a/synapse/storage/data_stores/main/roommember.py +++ b/synapse/storage/data_stores/main/roommember.py @@ -576,7 +576,8 @@ def _get_joined_users_from_context( if key[0] == EventTypes.Member ] for etype, state_key in context.delta_ids: - users_in_room.pop(state_key, None) + if etype == EventTypes.Member: + users_in_room.pop(state_key, None) # We check if we have any of the member event ids in the event cache # before we ask the DB diff --git a/synapse/storage/data_stores/main/ui_auth.py b/synapse/storage/data_stores/main/ui_auth.py index c8eebc937843..1d8ee22fb117 100644 --- a/synapse/storage/data_stores/main/ui_auth.py +++ b/synapse/storage/data_stores/main/ui_auth.py @@ -172,6 +172,27 @@ async def get_completed_ui_auth_stages( return results + async def set_ui_auth_clientdict( + self, session_id: str, clientdict: JsonDict + ) -> None: + """ + Store an updated clientdict for a given session ID. + + Args: + session_id: The ID of this session as returned from check_auth + clientdict: + The dictionary from the client root level, not the 'auth' key. + """ + # The clientdict gets stored as JSON. + clientdict_json = json.dumps(clientdict) + + self.db.simple_update_one( + table="ui_auth_sessions", + keyvalues={"session_id": session_id}, + updatevalues={"clientdict": clientdict_json}, + desc="set_ui_auth_client_dict", + ) + async def set_ui_auth_session_data(self, session_id: str, key: str, value: Any): """ Store a key-value pair into the sessions data associated with this diff --git a/synapse/storage/database.py b/synapse/storage/database.py index 2b635d6ca084..c3d086342976 100644 --- a/synapse/storage/database.py +++ b/synapse/storage/database.py @@ -214,9 +214,9 @@ def execute(self, sql: str, *args: Any): def executemany(self, sql: str, *args: Any): self._do_execute(self.txn.executemany, sql, *args) - def _make_sql_one_line(self, sql): + def _make_sql_one_line(self, sql: str) -> str: "Strip newlines out of SQL so that the loggers in the DB are on one line" - return " ".join(l.strip() for l in sql.splitlines() if l.strip()) + return " ".join(line.strip() for line in sql.splitlines() if line.strip()) def _do_execute(self, func, sql, *args): sql = self._make_sql_one_line(sql) diff --git a/tests/config/test_load.py b/tests/config/test_load.py index b3e557bd6ab1..734a9983e832 100644 --- a/tests/config/test_load.py +++ b/tests/config/test_load.py @@ -122,7 +122,7 @@ def generate_config_and_remove_lines_containing(self, needle): with open(self.file, "r") as f: contents = f.readlines() - contents = [l for l in contents if needle not in l] + contents = [line for line in contents if needle not in line] with open(self.file, "w") as f: f.write("".join(contents)) diff --git a/tests/rest/client/v2_alpha/test_auth.py b/tests/rest/client/v2_alpha/test_auth.py index 587be7b2e714..293ccfba2bb1 100644 --- a/tests/rest/client/v2_alpha/test_auth.py +++ b/tests/rest/client/v2_alpha/test_auth.py @@ -12,16 +12,20 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - +from typing import List, Union from twisted.internet.defer import succeed import synapse.rest.admin from synapse.api.constants import LoginType from synapse.handlers.ui_auth.checkers import UserInteractiveAuthChecker -from synapse.rest.client.v2_alpha import auth, register +from synapse.http.site import SynapseRequest +from synapse.rest.client.v1 import login +from synapse.rest.client.v2_alpha import auth, devices, register +from synapse.types import JsonDict from tests import unittest +from tests.server import FakeChannel class DummyRecaptchaChecker(UserInteractiveAuthChecker): @@ -34,11 +38,15 @@ def check_auth(self, authdict, clientip): return succeed(True) +class DummyPasswordChecker(UserInteractiveAuthChecker): + def check_auth(self, authdict, clientip): + return succeed(authdict["identifier"]["user"]) + + class FallbackAuthTests(unittest.HomeserverTestCase): servlets = [ auth.register_servlets, - synapse.rest.admin.register_servlets_for_client_rest_resource, register.register_servlets, ] hijack_auth = False @@ -59,79 +67,83 @@ def prepare(self, reactor, clock, hs): auth_handler = hs.get_auth_handler() auth_handler.checkers[LoginType.RECAPTCHA] = self.recaptcha_checker - @unittest.INFO - def test_fallback_captcha(self): - + def register(self, expected_response: int, body: JsonDict) -> FakeChannel: + """Make a register request.""" request, channel = self.make_request( - "POST", - "register", - {"username": "user", "type": "m.login.password", "password": "bar"}, - ) + "POST", "register", body + ) # type: SynapseRequest, FakeChannel self.render(request) - # Returns a 401 as per the spec - self.assertEqual(request.code, 401) - # Grab the session - session = channel.json_body["session"] - # Assert our configured public key is being given - self.assertEqual( - channel.json_body["params"]["m.login.recaptcha"]["public_key"], "brokencake" - ) + self.assertEqual(request.code, expected_response) + return channel + + def recaptcha( + self, session: str, expected_post_response: int, post_session: str = None + ) -> None: + """Get and respond to a fallback recaptcha. Returns the second request.""" + if post_session is None: + post_session = session request, channel = self.make_request( "GET", "auth/m.login.recaptcha/fallback/web?session=" + session - ) + ) # type: SynapseRequest, FakeChannel self.render(request) self.assertEqual(request.code, 200) request, channel = self.make_request( "POST", "auth/m.login.recaptcha/fallback/web?session=" - + session + + post_session + "&g-recaptcha-response=a", ) self.render(request) - self.assertEqual(request.code, 200) + self.assertEqual(request.code, expected_post_response) # The recaptcha handler is called with the response given attempts = self.recaptcha_checker.recaptcha_attempts self.assertEqual(len(attempts), 1) self.assertEqual(attempts[0][0]["response"], "a") - # also complete the dummy auth - request, channel = self.make_request( - "POST", "register", {"auth": {"session": session, "type": "m.login.dummy"}} + @unittest.INFO + def test_fallback_captcha(self): + """Ensure that fallback auth via a captcha works.""" + # Returns a 401 as per the spec + channel = self.register( + 401, {"username": "user", "type": "m.login.password", "password": "bar"}, ) - self.render(request) + + # Grab the session + session = channel.json_body["session"] + # Assert our configured public key is being given + self.assertEqual( + channel.json_body["params"]["m.login.recaptcha"]["public_key"], "brokencake" + ) + + # Complete the recaptcha step. + self.recaptcha(session, 200) + + # also complete the dummy auth + self.register(200, {"auth": {"session": session, "type": "m.login.dummy"}}) # Now we should have fulfilled a complete auth flow, including # the recaptcha fallback step, we can then send a # request to the register API with the session in the authdict. - request, channel = self.make_request( - "POST", "register", {"auth": {"session": session}} - ) - self.render(request) - self.assertEqual(channel.code, 200) + channel = self.register(200, {"auth": {"session": session}}) # We're given a registered user. self.assertEqual(channel.json_body["user_id"], "@user:test") - def test_cannot_change_operation(self): + def test_complete_operation_unknown_session(self): """ - The initial requested operation cannot be modified during the user interactive authentication session. + Attempting to mark an invalid session as complete should error. """ - # Make the initial request to register. (Later on a different password # will be used.) - request, channel = self.make_request( - "POST", - "register", - {"username": "user", "type": "m.login.password", "password": "bar"}, + # Returns a 401 as per the spec + channel = self.register( + 401, {"username": "user", "type": "m.login.password", "password": "bar"} ) - self.render(request) - # Returns a 401 as per the spec - self.assertEqual(request.code, 401) # Grab the session session = channel.json_body["session"] # Assert our configured public key is being given @@ -139,85 +151,166 @@ def test_cannot_change_operation(self): channel.json_body["params"]["m.login.recaptcha"]["public_key"], "brokencake" ) + # Attempt to complete the recaptcha step with an unknown session. + # This results in an error. + self.recaptcha(session, 400, session + "unknown") + + +class UIAuthTests(unittest.HomeserverTestCase): + servlets = [ + auth.register_servlets, + devices.register_servlets, + login.register_servlets, + synapse.rest.admin.register_servlets_for_client_rest_resource, + register.register_servlets, + ] + + def prepare(self, reactor, clock, hs): + auth_handler = hs.get_auth_handler() + auth_handler.checkers[LoginType.PASSWORD] = DummyPasswordChecker(hs) + + self.user_pass = "pass" + self.user = self.register_user("test", self.user_pass) + self.user_tok = self.login("test", self.user_pass) + + def get_device_ids(self) -> List[str]: + # Get the list of devices so one can be deleted. request, channel = self.make_request( - "GET", "auth/m.login.recaptcha/fallback/web?session=" + session - ) + "GET", "devices", access_token=self.user_tok, + ) # type: SynapseRequest, FakeChannel self.render(request) + + # Get the ID of the device. self.assertEqual(request.code, 200) + return [d["device_id"] for d in channel.json_body["devices"]] + def delete_device( + self, device: str, expected_response: int, body: Union[bytes, JsonDict] = b"" + ) -> FakeChannel: + """Delete an individual device.""" request, channel = self.make_request( - "POST", - "auth/m.login.recaptcha/fallback/web?session=" - + session - + "&g-recaptcha-response=a", - ) + "DELETE", "devices/" + device, body, access_token=self.user_tok + ) # type: SynapseRequest, FakeChannel self.render(request) - self.assertEqual(request.code, 200) - # The recaptcha handler is called with the response given - attempts = self.recaptcha_checker.recaptcha_attempts - self.assertEqual(len(attempts), 1) - self.assertEqual(attempts[0][0]["response"], "a") + # Ensure the response is sane. + self.assertEqual(request.code, expected_response) - # also complete the dummy auth + return channel + + def delete_devices(self, expected_response: int, body: JsonDict) -> FakeChannel: + """Delete 1 or more devices.""" + # Note that this uses the delete_devices endpoint so that we can modify + # the payload half-way through some tests. request, channel = self.make_request( - "POST", "register", {"auth": {"session": session, "type": "m.login.dummy"}} - ) + "POST", "delete_devices", body, access_token=self.user_tok, + ) # type: SynapseRequest, FakeChannel self.render(request) - # Now we should have fulfilled a complete auth flow, including - # the recaptcha fallback step. Make the initial request again, but - # with a different password. This causes the request to fail since the - # operaiton was modified during the ui auth session. - request, channel = self.make_request( - "POST", - "register", + # Ensure the response is sane. + self.assertEqual(request.code, expected_response) + + return channel + + def test_ui_auth(self): + """ + Test user interactive authentication outside of registration. + """ + device_id = self.get_device_ids()[0] + + # Attempt to delete this device. + # Returns a 401 as per the spec + channel = self.delete_device(device_id, 401) + + # Grab the session + session = channel.json_body["session"] + # Ensure that flows are what is expected. + self.assertIn({"stages": ["m.login.password"]}, channel.json_body["flows"]) + + # Make another request providing the UI auth flow. + self.delete_device( + device_id, + 200, { - "username": "user", - "type": "m.login.password", - "password": "foo", # Note this doesn't match the original request. - "auth": {"session": session}, + "auth": { + "type": "m.login.password", + "identifier": {"type": "m.id.user", "user": self.user}, + "password": self.user_pass, + "session": session, + }, }, ) - self.render(request) - self.assertEqual(channel.code, 403) - def test_complete_operation_unknown_session(self): + def test_can_change_body(self): """ - Attempting to mark an invalid session as complete should error. + The client dict can be modified during the user interactive authentication session. + + Note that it is not spec compliant to modify the client dict during a + user interactive authentication session, but many clients currently do. + + When Synapse is updated to be spec compliant, the call to re-use the + session ID should be rejected. """ + # Create a second login. + self.login("test", self.user_pass) - # Make the initial request to register. (Later on a different password - # will be used.) - request, channel = self.make_request( - "POST", - "register", - {"username": "user", "type": "m.login.password", "password": "bar"}, - ) - self.render(request) + device_ids = self.get_device_ids() + self.assertEqual(len(device_ids), 2) + # Attempt to delete the first device. # Returns a 401 as per the spec - self.assertEqual(request.code, 401) + channel = self.delete_devices(401, {"devices": [device_ids[0]]}) + # Grab the session session = channel.json_body["session"] - # Assert our configured public key is being given - self.assertEqual( - channel.json_body["params"]["m.login.recaptcha"]["public_key"], "brokencake" - ) + # Ensure that flows are what is expected. + self.assertIn({"stages": ["m.login.password"]}, channel.json_body["flows"]) - request, channel = self.make_request( - "GET", "auth/m.login.recaptcha/fallback/web?session=" + session + # Make another request providing the UI auth flow, but try to delete the + # second device. + self.delete_devices( + 200, + { + "devices": [device_ids[1]], + "auth": { + "type": "m.login.password", + "identifier": {"type": "m.id.user", "user": self.user}, + "password": self.user_pass, + "session": session, + }, + }, ) - self.render(request) - self.assertEqual(request.code, 200) - # Attempt to complete an unknown session, which should return an error. - unknown_session = session + "unknown" - request, channel = self.make_request( - "POST", - "auth/m.login.recaptcha/fallback/web?session=" - + unknown_session - + "&g-recaptcha-response=a", + def test_cannot_change_uri(self): + """ + The initial requested URI cannot be modified during the user interactive authentication session. + """ + # Create a second login. + self.login("test", self.user_pass) + + device_ids = self.get_device_ids() + self.assertEqual(len(device_ids), 2) + + # Attempt to delete the first device. + # Returns a 401 as per the spec + channel = self.delete_device(device_ids[0], 401) + + # Grab the session + session = channel.json_body["session"] + # Ensure that flows are what is expected. + self.assertIn({"stages": ["m.login.password"]}, channel.json_body["flows"]) + + # Make another request providing the UI auth flow, but try to delete the + # second device. This results in an error. + self.delete_device( + device_ids[1], + 403, + { + "auth": { + "type": "m.login.password", + "identifier": {"type": "m.id.user", "user": self.user}, + "password": self.user_pass, + "session": session, + }, + }, ) - self.render(request) - self.assertEqual(request.code, 400) diff --git a/tests/storage/test_roommember.py b/tests/storage/test_roommember.py index 00df0ea68e9c..5dd46005e652 100644 --- a/tests/storage/test_roommember.py +++ b/tests/storage/test_roommember.py @@ -22,6 +22,8 @@ from synapse.types import Requester, UserID from tests import unittest +from tests.test_utils import event_injection +from tests.utils import TestHomeServer class RoomMemberStoreTestCase(unittest.HomeserverTestCase): @@ -38,7 +40,7 @@ def make_homeserver(self, reactor, clock): ) return hs - def prepare(self, reactor, clock, hs): + def prepare(self, reactor, clock, hs: TestHomeServer): # We can't test the RoomMemberStore on its own without the other event # storage logic @@ -114,6 +116,52 @@ def test_count_known_servers_stat_counter_enabled(self): # It now knows about Charlie's server. self.assertEqual(self.store._known_servers_count, 2) + def test_get_joined_users_from_context(self): + room = self.helper.create_room_as(self.u_alice, tok=self.t_alice) + bob_event = event_injection.inject_member_event( + self.hs, room, self.u_bob, Membership.JOIN + ) + + # first, create a regular event + event, context = event_injection.create_event( + self.hs, + room_id=room, + sender=self.u_alice, + prev_event_ids=[bob_event.event_id], + type="m.test.1", + content={}, + ) + + users = self.get_success( + self.store.get_joined_users_from_context(event, context) + ) + self.assertEqual(users.keys(), {self.u_alice, self.u_bob}) + + # Regression test for #7376: create a state event whose key matches bob's + # user_id, but which is *not* a membership event, and persist that; then check + # that `get_joined_users_from_context` returns the correct users for the next event. + non_member_event = event_injection.inject_event( + self.hs, + room_id=room, + sender=self.u_bob, + prev_event_ids=[bob_event.event_id], + type="m.test.2", + state_key=self.u_bob, + content={}, + ) + event, context = event_injection.create_event( + self.hs, + room_id=room, + sender=self.u_alice, + prev_event_ids=[non_member_event.event_id], + type="m.test.3", + content={}, + ) + users = self.get_success( + self.store.get_joined_users_from_context(event, context) + ) + self.assertEqual(users.keys(), {self.u_alice, self.u_bob}) + class CurrentStateMembershipUpdateTestCase(unittest.HomeserverTestCase): def prepare(self, reactor, clock, homeserver): diff --git a/tox.ini b/tox.ini index ad4ed8299ec5..d23d25948e59 100644 --- a/tox.ini +++ b/tox.ini @@ -204,6 +204,7 @@ commands = mypy \ synapse/util/caches/stream_change_cache.py \ tests/replication/tcp/streams \ tests/test_utils \ + tests/rest/client/v2_alpha/test_auth.py \ tests/util/test_stream_change_cache.py # To find all folders that pass mypy you run: