Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add tests for blacklisting reactor/agent. #9563

Merged
merged 6 commits into from
Mar 11, 2021
Merged

Conversation

clokep
Copy link
Member

@clokep clokep commented Mar 8, 2021

Related to #9513, this:

  • Adds tests for the blacklisting agent and reactor (we didn't seem to have any?)
  • Fixes some type hint errors related to those classes.

I broke it into a few commits that should be standalone, hopefully they make sense!

@anoadragon453
Copy link
Member

The error that seems to plague the SyTest logs:

2021-03-08 20:46:53,620 - synapse.http.federation.matrix_federation_agent - 289 - INFO - POST-3914 - Failed to connect to localhost:8836: An error occurred while connecting: 99: Cannot assign requested address.
2021-03-08 20:46:53,621 - synapse.http.matrixfederationclient - 552 - INFO - POST-3914 - {PUT-O-30} [localhost:8836] Request failed: PUT matrix://localhost:8836/_matrix/federation/v2/invite/%21NSmQXRsxOFnooAPutw%3Alocalhost%3A8800/%241615236413224mOQKH%3Alocalhost%3A8800: ConnectError('Cannot assign requested address',)
2021-03-08 20:46:54,064 - synapse.metrics - 577 - INFO - None - Collecting gc 1
2021-03-08 20:46:54,668 - synapse.http.federation.matrix_federation_agent - 289 - INFO - POST-3914 - Failed to connect to localhost:8836: An error occurred while connecting: 99: Cannot assign requested address.
2021-03-08 20:46:54,669 - synapse.http.matrixfederationclient - 552 - INFO - POST-3914 - {PUT-O-30} [localhost:8836] Request failed: PUT matrix://localhost:8836/_matrix/federation/v2/invite/%21NSmQXRsxOFnooAPutw%3Alocalhost%3A8800/%241615236413224mOQKH%3Alocalhost%3A8800: ConnectError('Cannot assign requested address',)
2021-03-08 20:46:56,430 - synapse.http.federation.matrix_federation_agent - 289 - INFO - POST-3914 - Failed to connect to localhost:8836: An error occurred while connecting: 99: Cannot assign requested address.
2021-03-08 20:46:56,431 - synapse.http.matrixfederationclient - 552 - INFO - POST-3914 - {PUT-O-30} [localhost:8836] Request failed: PUT matrix://localhost:8836/_matrix/federation/v2/invite/%21NSmQXRsxOFnooAPutw%3Alocalhost%3A8800/%241615236413224mOQKH%3Alocalhost%3A8800: ConnectError('Cannot assign requested address',)
2021-03-08 20:46:58,732 - synapse.http.federation.matrix_federation_agent - 289 - INFO - POST-3914 - Failed to connect to localhost:8836: An error occurred while connecting: 99: Cannot assign requested address.
2021-03-08 20:46:58,733 - synapse.http.matrixfederationclient - 552 - INFO - POST-3914 - {PUT-O-30} [localhost:8836] Request failed: PUT matrix://localhost:8836/_matrix/federation/v2/invite/%21NSmQXRsxOFnooAPutw%3Alocalhost%3A8800/%241615236413224mOQKH%3Alocalhost%3A8800: ConnectError('Cannot assign requested address',)
2021-03-08 20:46:58,735 - synapse.http.server - 98 - ERROR - POST-3914 - Failed handle request via 'RoomCreateRestServlet': <SynapseRequest at 0x7fceebb54da0 method='POST' uri='/_matrix/client/r0/createRoom?access_token=<redacted>' clientproto='HTTP/1.1' site='8800'>
Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/synapse/http/matrixfederationclient.py", line 481, in _send_request
    response = await request_deferred
  File "/venv/lib/python3.5/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/venv/lib/python3.5/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/venv/lib/python3.5/site-packages/synapse/util/patch_inline_callbacks.py", line 142, in check_yield_points_inner
    d = result.throwExceptionIntoGenerator(gen)
  File "/venv/lib/python3.5/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/venv/lib/python3.5/site-packages/synapse/http/federation/matrix_federation_agent.py", line 191, in request
    self._agent.request(method, uri, request_headers, bodyProducer)
  File "/venv/lib/python3.5/site-packages/synapse/util/patch_inline_callbacks.py", line 190, in check_yield_points_inner
    result = yield d
  File "/venv/lib/python3.5/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/venv/lib/python3.5/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/venv/lib/python3.5/site-packages/synapse/http/federation/matrix_federation_agent.py", line 296, in _do_connect
    raise first_exception
  File "/venv/lib/python3.5/site-packages/synapse/http/federation/matrix_federation_agent.py", line 283, in _do_connect
    endpoint.connect(protocol_factory)
twisted.internet.error.ConnectError: An error occurred while connecting: 99: Cannot assign requested address.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/synapse/http/server.py", line 259, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/venv/lib/python3.5/site-packages/synapse/http/server.py", line 447, in _async_render
    callback_return = await raw_callback_return
  File "/venv/lib/python3.5/site-packages/synapse/rest/client/v1/room.py", line 83, in on_POST
    requester, self.get_room_config(request)
  File "/venv/lib/python3.5/site-packages/synapse/handlers/room.py", line 818, in create_room
    content=content,
  File "/venv/lib/python3.5/site-packages/synapse/handlers/room_member.py", line 621, in update_membership_locked
    require_consent=require_consent,
  File "/venv/lib/python3.5/site-packages/synapse/handlers/room_member.py", line 248, in _local_membership_update
    ratelimit=ratelimit,
  File "/venv/lib/python3.5/site-packages/synapse/util/metrics.py", line 92, in measured_func
    r = await func(self, *args, **kwargs)
  File "/venv/lib/python3.5/site-packages/synapse/handlers/message.py", line 989, in handle_new_client_event
    requester, event, context, ratelimit=ratelimit, extra_users=extra_users
  File "/venv/lib/python3.5/site-packages/synapse/handlers/message.py", line 1177, in persist_and_notify_client_event
    invitee.domain, event
  File "/venv/lib/python3.5/site-packages/synapse/handlers/federation.py", line 1312, in send_invite
    pdu=event,
  File "/venv/lib/python3.5/site-packages/synapse/federation/federation_client.py", line 761, in send_invite
    content = await self._do_send_invite(destination, pdu, room_version)
  File "/venv/lib/python3.5/site-packages/synapse/federation/federation_client.py", line 795, in _do_send_invite
    "invite_room_state": pdu.unsigned.get("invite_room_state", []),
  File "/venv/lib/python3.5/site-packages/synapse/federation/transport/client.py", line 311, in send_invite_v2
    destination=destination, path=path, data=content, ignore_backoff=True
  File "/venv/lib/python3.5/site-packages/synapse/http/matrixfederationclient.py", line 720, in put_json
    timeout=timeout,
  File "/venv/lib/python3.5/site-packages/synapse/http/matrixfederationclient.py", line 297, in _send_request_with_optional_trailing_slash
    response = await self._send_request(request, **send_request_args)
  File "/venv/lib/python3.5/site-packages/synapse/http/matrixfederationclient.py", line 485, in _send_request
    raise RequestSendFailed(e, can_retry=True) from e
synapse.api.errors.RequestSendFailed: Failed to send request: ConnectError: An error occurred while connecting: 99: Cannot assign requested address.

Comment on lines +181 to +182
recv.addressResolved(address)
recv.resolutionComplete()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these recvs need to be recv()?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mentioned this in 315061f, but recv does not need to be instantiated. (It actually shouldn't be, it isn't a callable necessarily!) The only reason the current code works is that the implementation of EndpointReceiver happens to use a class with all static methods, so instantiating it essentially has the same methods called as if it isn't instantiated.

tl;dr recv is meant to be an instance, not a class.

This just happens to work since the default implementation (via the
HostnameEndpoint) happens to create a class that only has static
methods, then passes the class (without instantiating it), presumedly
to save memory.
@clokep
Copy link
Member Author

clokep commented Mar 9, 2021

I had a typo in one of the commits which was causing the sytest failure. This should be ready for review now!

Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm other than this bit that I don't quite understand.

agent.request(b"GET", b"http://" + self.unsafe_ip), SynapseError
)

# The safe and unsafe domains and safe IPs should be accepted.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the unsafe domain accepted? Shouldn't it resolve to the unsafe IP and then be blocked?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The agent doesn't do any resolution, it only attempts to block access when IPs are directly accessed. The reactor ensures that resolved IPs can be blocked effectively. This is tested above. You should use both of them in tandem to ensure all cases are blocked.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, and the docstring of BlacklistingAgentWrapper does indeed say as much. Ah well, thank you!

Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

agent.request(b"GET", b"http://" + self.unsafe_ip), SynapseError
)

# The safe and unsafe domains and safe IPs should be accepted.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, and the docstring of BlacklistingAgentWrapper does indeed say as much. Ah well, thank you!

@clokep clokep merged commit e55bd0e into develop Mar 11, 2021
@clokep clokep deleted the clokep/twisted-types-3 branch March 11, 2021 14:15
Half-Shot added a commit that referenced this pull request Mar 16, 2021
Synapse 1.30.0rc1 (2021-03-16)
==============================

Note that this release deprecates the ability for appservices to
call `POST /_matrix/client/r0/register`  without the body parameter `type`. Appservice
developers should use a `type` value of `m.login.application_service` as
per [the spec](https://matrix.org/docs/spec/application_service/r0.1.2#server-admin-style-permissions).
In future releases, calling this endpoint with an access token - but without a `m.login.application_service`
type - will fail.

Features
--------

- Add prometheus metrics for number of users successfully registering and logging in. ([\#9510](#9510), [\#9511](#9511), [\#9573](#9573))
- Add `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time` prometheus metrics, which monitor federation delays by reporting the timestamps of messages sent and received to a set of remote servers. ([\#9540](#9540))
- Add support for generating JSON Web Tokens dynamically for use as OIDC client secrets. ([\#9549](#9549))
- Optimise handling of incomplete room history for incoming federation. ([\#9601](#9601))
- Finalise support for allowing clients to pick an SSO Identity Provider ([MSC2858](matrix-org/matrix-spec-proposals#2858)). ([\#9617](#9617))
- Tell spam checker modules about the SSO IdP a user registered through if one was used. ([\#9626](#9626))

Bugfixes
--------

- Fix long-standing bug when generating thumbnails for some images with transparency: `TypeError: cannot unpack non-iterable int object`. ([\#9473](#9473))
- Purge chain cover indexes for events that were purged prior to Synapse v1.29.0. ([\#9542](#9542), [\#9583](#9583))
- Fix bug where federation requests were not correctly retried on 5xx responses. ([\#9567](#9567))
- Fix re-activating an account via the admin API when local passwords are disabled. ([\#9587](#9587))
- Fix a bug introduced in Synapse 1.20 which caused incoming federation transactions to stack up, causing slow recovery from outages. ([\#9597](#9597))
- Fix a bug introduced in v1.28.0 where the OpenID Connect callback endpoint could error with a `MacaroonInitException`. ([\#9620](#9620))
- Fix Internal Server Error on `GET /_synapse/client/saml2/authn_response` request. ([\#9623](#9623))

Updates to the Docker image
---------------------------

- Use jemalloc if available in docker. ([\#8553](#8553))

Improved Documentation
----------------------

- Add relayd entry to reverse proxy example configurations. ([\#9508](#9508))
- Improve the SAML2 upgrade notes for 1.27.0. ([\#9550](#9550))
- Link to the "List user's media" admin API from the media admin API docs. ([\#9571](#9571))
- Clarify the spam checker modules documentation example to mention that `parse_config` is a required method. ([\#9580](#9580))
- Clarify the sample configuration for `stats` settings. ([\#9604](#9604))

Deprecations and Removals
-------------------------

- The `synapse_federation_last_sent_pdu_age` and `synapse_federation_last_received_pdu_age` prometheus metrics have been removed. They are replaced by `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time`. ([\#9540](#9540))
- Registering an Application Service user without using the `m.login.application_service` login type will be unsupported in an upcoming Synapse release. ([\#9559](#9559))

Internal Changes
----------------

- Add tests to ResponseCache. ([\#9458](#9458))
- Add type hints to purge room and server notice admin API. ([\#9520](#9520))
- Add extra logging to ObservableDeferred when callbacks throw exceptions. ([\#9523](#9523))
- Fix incorrect type hints. ([\#9528](#9528), [\#9543](#9543), [\#9591](#9591), [\#9608](#9608), [\#9618](#9618))
- Add an additional test for purging a room. ([\#9541](#9541))
- Add a `.git-blame-ignore-revs` file with the hashes of auto-formatting. ([\#9560](#9560))
- Increase the threshold before which outbound federation to a server goes into "catch up" mode, which is expensive for the remote server to handle. ([\#9561](#9561))
- Fix spurious errors reported by the `config-lint.sh` script. ([\#9562](#9562))
- Fix type hints and tests for BlacklistingAgentWrapper and BlacklistingReactorWrapper. ([\#9563](#9563))
- Do not have mypy ignore type hints from unpaddedbase64. ([\#9568](#9568))
- Improve efficiency of calculating the auth chain in large rooms. ([\#9576](#9576))
- Convert `synapse.types.Requester` to an `attrs` class. ([\#9586](#9586))
- Add logging for redis connection setup. ([\#9590](#9590))
- Improve logging when processing incoming transactions. ([\#9596](#9596))
- Remove unused `stats.retention` setting, and emit a warning if stats are disabled. ([\#9604](#9604))
- Prevent attempting to bundle aggregations for state events in /context APIs. ([\#9619](#9619))
Half-Shot added a commit that referenced this pull request Mar 22, 2021
Synapse 1.30.0 (2021-03-22)
===========================

Note that this release deprecates the ability for appservices to
call `POST /_matrix/client/r0/register`  without the body parameter `type`. Appservice
developers should use a `type` value of `m.login.application_service` as
per [the spec](https://matrix.org/docs/spec/application_service/r0.1.2#server-admin-style-permissions).
In future releases, calling this endpoint with an access token - but without a `m.login.application_service`
type - will fail.

No significant changes.

Synapse 1.30.0rc1 (2021-03-16)
==============================

Features
--------

- Add prometheus metrics for number of users successfully registering and logging in. ([\#9510](#9510), [\#9511](#9511), [\#9573](#9573))
- Add `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time` prometheus metrics, which monitor federation delays by reporting the timestamps of messages sent and received to a set of remote servers. ([\#9540](#9540))
- Add support for generating JSON Web Tokens dynamically for use as OIDC client secrets. ([\#9549](#9549))
- Optimise handling of incomplete room history for incoming federation. ([\#9601](#9601))
- Finalise support for allowing clients to pick an SSO Identity Provider ([MSC2858](matrix-org/matrix-spec-proposals#2858)). ([\#9617](#9617))
- Tell spam checker modules about the SSO IdP a user registered through if one was used. ([\#9626](#9626))

Bugfixes
--------

- Fix long-standing bug when generating thumbnails for some images with transparency: `TypeError: cannot unpack non-iterable int object`. ([\#9473](#9473))
- Purge chain cover indexes for events that were purged prior to Synapse v1.29.0. ([\#9542](#9542), [\#9583](#9583))
- Fix bug where federation requests were not correctly retried on 5xx responses. ([\#9567](#9567))
- Fix re-activating an account via the admin API when local passwords are disabled. ([\#9587](#9587))
- Fix a bug introduced in Synapse 1.20 which caused incoming federation transactions to stack up, causing slow recovery from outages. ([\#9597](#9597))
- Fix a bug introduced in v1.28.0 where the OpenID Connect callback endpoint could error with a `MacaroonInitException`. ([\#9620](#9620))
- Fix Internal Server Error on `GET /_synapse/client/saml2/authn_response` request. ([\#9623](#9623))

Updates to the Docker image
---------------------------

- Make use of an improved malloc implementation (`jemalloc`) in the docker image. ([\#8553](#8553))

Improved Documentation
----------------------

- Add relayd entry to reverse proxy example configurations. ([\#9508](#9508))
- Improve the SAML2 upgrade notes for 1.27.0. ([\#9550](#9550))
- Link to the "List user's media" admin API from the media admin API docs. ([\#9571](#9571))
- Clarify the spam checker modules documentation example to mention that `parse_config` is a required method. ([\#9580](#9580))
- Clarify the sample configuration for `stats` settings. ([\#9604](#9604))

Deprecations and Removals
-------------------------

- The `synapse_federation_last_sent_pdu_age` and `synapse_federation_last_received_pdu_age` prometheus metrics have been removed. They are replaced by `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time`. ([\#9540](#9540))
- Registering an Application Service user without using the `m.login.application_service` login type will be unsupported in an upcoming Synapse release. ([\#9559](#9559))

Internal Changes
----------------

- Add tests to ResponseCache. ([\#9458](#9458))
- Add type hints to purge room and server notice admin API. ([\#9520](#9520))
- Add extra logging to ObservableDeferred when callbacks throw exceptions. ([\#9523](#9523))
- Fix incorrect type hints. ([\#9528](#9528), [\#9543](#9543), [\#9591](#9591), [\#9608](#9608), [\#9618](#9618))
- Add an additional test for purging a room. ([\#9541](#9541))
- Add a `.git-blame-ignore-revs` file with the hashes of auto-formatting. ([\#9560](#9560))
- Increase the threshold before which outbound federation to a server goes into "catch up" mode, which is expensive for the remote server to handle. ([\#9561](#9561))
- Fix spurious errors reported by the `config-lint.sh` script. ([\#9562](#9562))
- Fix type hints and tests for BlacklistingAgentWrapper and BlacklistingReactorWrapper. ([\#9563](#9563))
- Do not have mypy ignore type hints from unpaddedbase64. ([\#9568](#9568))
- Improve efficiency of calculating the auth chain in large rooms. ([\#9576](#9576))
- Convert `synapse.types.Requester` to an `attrs` class. ([\#9586](#9586))
- Add logging for redis connection setup. ([\#9590](#9590))
- Improve logging when processing incoming transactions. ([\#9596](#9596))
- Remove unused `stats.retention` setting, and emit a warning if stats are disabled. ([\#9604](#9604))
- Prevent attempting to bundle aggregations for state events in /context APIs. ([\#9619](#9619))
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Apr 28, 2021
Synapse 1.32.2 (2021-04-22)
===========================

This release includes a fix for a regression introduced in 1.32.0.

Bugfixes
--------

- Fix a regression in Synapse 1.32.0 and 1.32.1 which caused `LoggingContext` errors in plugins. ([\#9857](matrix-org/synapse#9857))


Synapse 1.32.1 (2021-04-21)
===========================

This release fixes [a regression](matrix-org/synapse#9853)
in Synapse 1.32.0 that caused connected Prometheus instances to become unstable.

However, as this release is still subject to the `LoggingContext` change in 1.32.0,
it is recommended to remain on or downgrade to 1.31.0.

Bugfixes
--------

- Fix a regression in Synapse 1.32.0 which caused Synapse to report large numbers of Prometheus time series, potentially overwhelming Prometheus instances. ([\#9854](matrix-org/synapse#9854))


Synapse 1.32.0 (2021-04-20)
===========================

**Note:** This release introduces [a regression](matrix-org/synapse#9853)
that can overwhelm connected Prometheus instances. This issue was not present in
1.32.0rc1. If affected, it is recommended to downgrade to 1.31.0 in the meantime, and
follow [these instructions](matrix-org/synapse#9854 (comment))
to clean up any excess writeahead logs.

**Note:** This release also mistakenly included a change that may affected Synapse
modules that import `synapse.logging.context.LoggingContext`, such as
[synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider).
This will be fixed in a later Synapse version.

**Note:** This release requires Python 3.6+ and Postgres 9.6+ or SQLite 3.22+.

This release removes the deprecated `GET /_synapse/admin/v1/users/<user_id>` admin API. Please use the [v2 API](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/user_admin_api.rst#query-user-account) instead, which has improved capabilities.

This release requires Application Services to use type `m.login.application_service` when registering users via the `/_matrix/client/r0/register` endpoint to comply with the spec. Please ensure your Application Services are up to date.

If you are using the `packages.matrix.org` Debian repository for Synapse packages,
note that we have recently updated the expiry date on the gpg signing key. If you see an
error similar to `The following signatures were invalid: EXPKEYSIG F473DD4473365DE1`, you
will need to get a fresh copy of the keys. You can do so with:

```sh
sudo wget -O /usr/share/keyrings/matrix-org-archive-keyring.gpg https://packages.matrix.org/debian/matrix-org-archive-keyring.gpg
```

Bugfixes
--------

- Fix the log lines of nested logging contexts. Broke in 1.32.0rc1. ([\#9829](matrix-org/synapse#9829))


Synapse 1.32.0rc1 (2021-04-13)
==============================

Features
--------

- Add a Synapse module for routing presence updates between users. ([\#9491](matrix-org/synapse#9491))
- Add an admin API to manage ratelimit for a specific user. ([\#9648](matrix-org/synapse#9648))
- Include request information in structured logging output. ([\#9654](matrix-org/synapse#9654))
- Add `order_by` to the admin API `GET /_synapse/admin/v2/users`. Contributed by @dklimpel. ([\#9691](matrix-org/synapse#9691))
- Replace the `room_invite_state_types` configuration setting with `room_prejoin_state`. ([\#9700](matrix-org/synapse#9700))
- Add experimental support for [MSC3083](matrix-org/matrix-spec-proposals#3083): restricting room access via group membership. ([\#9717](matrix-org/synapse#9717), [\#9735](matrix-org/synapse#9735))
- Update experimental support for Spaces: include `m.room.create` in the room state sent with room-invites. ([\#9710](matrix-org/synapse#9710))
- Synapse now requires Python 3.6 or later. It also requires Postgres 9.6 or later or SQLite 3.22 or later. ([\#9766](matrix-org/synapse#9766))


Bugfixes
--------

- Prevent `synapse_forward_extremities` and `synapse_excess_extremity_events` Prometheus metrics from initially reporting zero-values after startup. ([\#8926](matrix-org/synapse#8926))
- Fix recently added ratelimits to correctly honour the application service `rate_limited` flag. ([\#9711](matrix-org/synapse#9711))
- Fix longstanding bug which caused `duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"` errors. ([\#9725](matrix-org/synapse#9725))
- Fix bug where sharded federation senders could get stuck repeatedly querying the DB in a loop, using lots of CPU. ([\#9770](matrix-org/synapse#9770))
- Fix duplicate logging of exceptions thrown during federation transaction processing. ([\#9780](matrix-org/synapse#9780))


Updates to the Docker image
---------------------------

- Move opencontainers labels to the final Docker image such that users can inspect them. ([\#9765](matrix-org/synapse#9765))


Improved Documentation
----------------------

- Make the `allowed_local_3pids` regex example in the sample config stricter. ([\#9719](matrix-org/synapse#9719))


Deprecations and Removals
-------------------------

- Remove old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9401](matrix-org/synapse#9401))
- Make `/_matrix/client/r0/register` expect a type of `m.login.application_service` when an Application Service registers a user, to align with [the relevant spec](https://spec.matrix.org/unstable/application-service-api/#server-admin-style-permissions). ([\#9548](matrix-org/synapse#9548))


Internal Changes
----------------

- Replace deprecated `imp` module with successor `importlib`. Contributed by Cristina Muñoz. ([\#9718](matrix-org/synapse#9718))
- Experiment with GitHub Actions for CI. ([\#9661](matrix-org/synapse#9661))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9682](matrix-org/synapse#9682))
- Update `scripts-dev/complement.sh` to use a local checkout of Complement, allow running a subset of tests and have it use Synapse's Complement test blacklist. ([\#9685](matrix-org/synapse#9685))
- Improve Jaeger tracing for `to_device` messages. ([\#9686](matrix-org/synapse#9686))
- Add release helper script for automating part of the Synapse release process. ([\#9713](matrix-org/synapse#9713))
- Add type hints to expiring cache. ([\#9730](matrix-org/synapse#9730))
- Convert various testcases to `HomeserverTestCase`. ([\#9736](matrix-org/synapse#9736))
- Start linting mypy with `no_implicit_optional`. ([\#9742](matrix-org/synapse#9742))
- Add missing type hints to federation handler and server. ([\#9743](matrix-org/synapse#9743))
- Check that a `ConfigError` is raised, rather than simply `Exception`, when appropriate in homeserver config file generation tests. ([\#9753](matrix-org/synapse#9753))
- Fix incompatibility with `tox` 2.5. ([\#9769](matrix-org/synapse#9769))
- Enable Complement tests for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary API. ([\#9771](matrix-org/synapse#9771))
- Use mock from the standard library instead of a separate package. ([\#9772](matrix-org/synapse#9772))
- Update Black configuration to target Python 3.6. ([\#9781](matrix-org/synapse#9781))
- Add option to skip unit tests when building Debian packages. ([\#9793](matrix-org/synapse#9793))


Synapse 1.31.0 (2021-04-06)
===========================

**Note:** As announced in v1.25.0, and in line with the deprecation policy for platform dependencies, this is the last release to support Python 3.5 and PostgreSQL 9.5. Future versions of Synapse will require Python 3.6+ and PostgreSQL 9.6+, as per our [deprecation policy](docs/deprecation_policy.md).

This is also the last release that the Synapse team will be publishing packages for Debian Stretch and Ubuntu Xenial.


Improved Documentation
----------------------

- Add a document describing the deprecation policy for platform dependencies. ([\#9723](matrix-org/synapse#9723))


Internal Changes
----------------

- Revert using `dmypy run` in lint script. ([\#9720](matrix-org/synapse#9720))
- Pin flake8-bugbear's version. ([\#9734](matrix-org/synapse#9734))


Synapse 1.31.0rc1 (2021-03-30)
==============================

Features
--------

- Add support to OpenID Connect login for requiring attributes on the `userinfo` response. Contributed by Hubbe King. ([\#9609](matrix-org/synapse#9609))
- Add initial experimental support for a "space summary" API. ([\#9643](matrix-org/synapse#9643), [\#9652](matrix-org/synapse#9652), [\#9653](matrix-org/synapse#9653))
- Add support for the busy presence state as described in [MSC3026](matrix-org/matrix-spec-proposals#3026). ([\#9644](matrix-org/synapse#9644))
- Add support for credentials for proxy authentication in the `HTTPS_PROXY` environment variable. ([\#9657](matrix-org/synapse#9657))


Bugfixes
--------

- Fix a longstanding bug that could cause issues when editing a reply to a message. ([\#9585](matrix-org/synapse#9585))
- Fix the `/capabilities` endpoint to return `m.change_password` as disabled if the local password database is not used for authentication. Contributed by @dklimpel. ([\#9588](matrix-org/synapse#9588))
- Check if local passwords are enabled before setting them for the user. ([\#9636](matrix-org/synapse#9636))
- Fix a bug where federation sending can stall due to `concurrent access` database exceptions when it falls behind. ([\#9639](matrix-org/synapse#9639))
- Fix a bug introduced in Synapse 1.30.1 which meant the suggested `pip` incantation to install an updated `cryptography` was incorrect. ([\#9699](matrix-org/synapse#9699))


Updates to the Docker image
---------------------------

- Speed up Docker builds and make it nicer to test against Complement while developing (install all dependencies before copying the project). ([\#9610](matrix-org/synapse#9610))
- Include [opencontainers labels](https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys) in the Docker image. ([\#9612](matrix-org/synapse#9612))


Improved Documentation
----------------------

- Clarify that `register_new_matrix_user` is present also when installed via non-pip package. ([\#9074](matrix-org/synapse#9074))
- Update source install documentation to mention platform prerequisites before the source install steps. ([\#9667](matrix-org/synapse#9667))
- Improve worker documentation for fallback/web auth endpoints. ([\#9679](matrix-org/synapse#9679))
- Update the sample configuration for OIDC authentication. ([\#9695](matrix-org/synapse#9695))


Internal Changes
----------------

- Preparatory steps for removing redundant `outlier` data from `event_json.internal_metadata` column. ([\#9411](matrix-org/synapse#9411))
- Add type hints to the caching module. ([\#9442](matrix-org/synapse#9442))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9499](matrix-org/synapse#9499), [\#9659](matrix-org/synapse#9659))
- Add additional type hints to the Homeserver object. ([\#9631](matrix-org/synapse#9631), [\#9638](matrix-org/synapse#9638), [\#9675](matrix-org/synapse#9675), [\#9681](matrix-org/synapse#9681))
- Only save remote cross-signing and device keys if they're different from the current ones. ([\#9634](matrix-org/synapse#9634))
- Rename storage function to fix spelling and not conflict with another function's name. ([\#9637](matrix-org/synapse#9637))
- Improve performance of federation catch up by sending the latest events in the room to the remote, rather than just the last event sent by the local server. ([\#9640](matrix-org/synapse#9640), [\#9664](matrix-org/synapse#9664))
- In the `federation_client` commandline client, stop automatically adding the URL prefix, so that servlets on other prefixes can be tested. ([\#9645](matrix-org/synapse#9645))
- In the `federation_client` commandline client, handle inline `signing_key`s in `homeserver.yaml`. ([\#9647](matrix-org/synapse#9647))
- Fixed some antipattern issues to improve code quality. ([\#9649](matrix-org/synapse#9649))
- Add a storage method for pulling all current user presence state from the database. ([\#9650](matrix-org/synapse#9650))
- Import `HomeServer` from the proper module. ([\#9665](matrix-org/synapse#9665))
- Increase default join ratelimiting burst rate. ([\#9674](matrix-org/synapse#9674))
- Add type hints to third party event rules and visibility modules. ([\#9676](matrix-org/synapse#9676))
- Bump mypy-zope to 0.2.13 to fix "Cannot determine consistent method resolution order (MRO)" errors when running mypy a second time. ([\#9678](matrix-org/synapse#9678))
- Use interpreter from `$PATH` via `/usr/bin/env` instead of absolute paths in various scripts. ([\#9689](matrix-org/synapse#9689))
- Make it possible to use `dmypy`. ([\#9692](matrix-org/synapse#9692))
- Suppress "CryptographyDeprecationWarning: int_from_bytes is deprecated". ([\#9698](matrix-org/synapse#9698))
- Use `dmypy run` in lint script for improved performance in type-checking while developing. ([\#9701](matrix-org/synapse#9701))
- Fix undetected mypy error when using Python 3.6. ([\#9703](matrix-org/synapse#9703))
- Fix type-checking CI on develop. ([\#9709](matrix-org/synapse#9709))


Synapse 1.30.1 (2021-03-26)
===========================

This release is identical to Synapse 1.30.0, with the exception of explicitly
setting a minimum version of Python's Cryptography library to ensure that users
of Synapse are protected from the recent [OpenSSL security advisories](https://mta.openssl.org/pipermail/openssl-announce/2021-March/000198.html),
especially CVE-2021-3449.

Note that Cryptography defaults to bundling its own statically linked copy of
OpenSSL, which means that you may not be protected by your operating system's
security updates.

It's also worth noting that Cryptography no longer supports Python 3.5, so
admins deploying to older environments may not be protected against this or
future vulnerabilities. Synapse will be dropping support for Python 3.5 at the
end of March.


Updates to the Docker image
---------------------------

- Ensure that the docker container has up to date versions of openssl. ([\#9697](matrix-org/synapse#9697))


Internal Changes
----------------

- Enforce that `cryptography` dependency is up to date to ensure it has the most recent openssl patches. ([\#9697](matrix-org/synapse#9697))


Synapse 1.30.0 (2021-03-22)
===========================

Note that this release deprecates the ability for appservices to
call `POST /_matrix/client/r0/register`  without the body parameter `type`. Appservice
developers should use a `type` value of `m.login.application_service` as
per [the spec](https://matrix.org/docs/spec/application_service/r0.1.2#server-admin-style-permissions).
In future releases, calling this endpoint with an access token - but without a `m.login.application_service`
type - will fail.


No significant changes.


Synapse 1.30.0rc1 (2021-03-16)
==============================

Features
--------

- Add prometheus metrics for number of users successfully registering and logging in. ([\#9510](matrix-org/synapse#9510), [\#9511](matrix-org/synapse#9511), [\#9573](matrix-org/synapse#9573))
- Add `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time` prometheus metrics, which monitor federation delays by reporting the timestamps of messages sent and received to a set of remote servers. ([\#9540](matrix-org/synapse#9540))
- Add support for generating JSON Web Tokens dynamically for use as OIDC client secrets. ([\#9549](matrix-org/synapse#9549))
- Optimise handling of incomplete room history for incoming federation. ([\#9601](matrix-org/synapse#9601))
- Finalise support for allowing clients to pick an SSO Identity Provider ([MSC2858](matrix-org/matrix-spec-proposals#2858)). ([\#9617](matrix-org/synapse#9617))
- Tell spam checker modules about the SSO IdP a user registered through if one was used. ([\#9626](matrix-org/synapse#9626))


Bugfixes
--------

- Fix long-standing bug when generating thumbnails for some images with transparency: `TypeError: cannot unpack non-iterable int object`. ([\#9473](matrix-org/synapse#9473))
- Purge chain cover indexes for events that were purged prior to Synapse v1.29.0. ([\#9542](matrix-org/synapse#9542), [\#9583](matrix-org/synapse#9583))
- Fix bug where federation requests were not correctly retried on 5xx responses. ([\#9567](matrix-org/synapse#9567))
- Fix re-activating an account via the admin API when local passwords are disabled. ([\#9587](matrix-org/synapse#9587))
- Fix a bug introduced in Synapse 1.20 which caused incoming federation transactions to stack up, causing slow recovery from outages. ([\#9597](matrix-org/synapse#9597))
- Fix a bug introduced in v1.28.0 where the OpenID Connect callback endpoint could error with a `MacaroonInitException`. ([\#9620](matrix-org/synapse#9620))
- Fix Internal Server Error on `GET /_synapse/client/saml2/authn_response` request. ([\#9623](matrix-org/synapse#9623))


Updates to the Docker image
---------------------------

- Make use of an improved malloc implementation (`jemalloc`) in the docker image. ([\#8553](matrix-org/synapse#8553))


Improved Documentation
----------------------

- Add relayd entry to reverse proxy example configurations. ([\#9508](matrix-org/synapse#9508))
- Improve the SAML2 upgrade notes for 1.27.0. ([\#9550](matrix-org/synapse#9550))
- Link to the "List user's media" admin API from the media admin API docs. ([\#9571](matrix-org/synapse#9571))
- Clarify the spam checker modules documentation example to mention that `parse_config` is a required method. ([\#9580](matrix-org/synapse#9580))
- Clarify the sample configuration for `stats` settings. ([\#9604](matrix-org/synapse#9604))


Deprecations and Removals
-------------------------

- The `synapse_federation_last_sent_pdu_age` and `synapse_federation_last_received_pdu_age` prometheus metrics have been removed. They are replaced by `synapse_federation_last_sent_pdu_time` and `synapse_federation_last_received_pdu_time`. ([\#9540](matrix-org/synapse#9540))
- Registering an Application Service user without using the `m.login.application_service` login type will be unsupported in an upcoming Synapse release. ([\#9559](matrix-org/synapse#9559))


Internal Changes
----------------

- Add tests to ResponseCache. ([\#9458](matrix-org/synapse#9458))
- Add type hints to purge room and server notice admin API. ([\#9520](matrix-org/synapse#9520))
- Add extra logging to ObservableDeferred when callbacks throw exceptions. ([\#9523](matrix-org/synapse#9523))
- Fix incorrect type hints. ([\#9528](matrix-org/synapse#9528), [\#9543](matrix-org/synapse#9543), [\#9591](matrix-org/synapse#9591), [\#9608](matrix-org/synapse#9608), [\#9618](matrix-org/synapse#9618))
- Add an additional test for purging a room. ([\#9541](matrix-org/synapse#9541))
- Add a `.git-blame-ignore-revs` file with the hashes of auto-formatting. ([\#9560](matrix-org/synapse#9560))
- Increase the threshold before which outbound federation to a server goes into "catch up" mode, which is expensive for the remote server to handle. ([\#9561](matrix-org/synapse#9561))
- Fix spurious errors reported by the `config-lint.sh` script. ([\#9562](matrix-org/synapse#9562))
- Fix type hints and tests for BlacklistingAgentWrapper and BlacklistingReactorWrapper. ([\#9563](matrix-org/synapse#9563))
- Do not have mypy ignore type hints from unpaddedbase64. ([\#9568](matrix-org/synapse#9568))
- Improve efficiency of calculating the auth chain in large rooms. ([\#9576](matrix-org/synapse#9576))
- Convert `synapse.types.Requester` to an `attrs` class. ([\#9586](matrix-org/synapse#9586))
- Add logging for redis connection setup. ([\#9590](matrix-org/synapse#9590))
- Improve logging when processing incoming transactions. ([\#9596](matrix-org/synapse#9596))
- Remove unused `stats.retention` setting, and emit a warning if stats are disabled. ([\#9604](matrix-org/synapse#9604))
- Prevent attempting to bundle aggregations for state events in /context APIs. ([\#9619](matrix-org/synapse#9619))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants