Skip to content

Commit

Permalink
Merge tag 'v1.64.0' into merge_v1_64_0
Browse files Browse the repository at this point in the history
Synapse 1.64.0 (2022-08-02)
===========================

No significant changes since 1.64.0rc2.

Deprecation Warning
-------------------

Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf.
[Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email)

Synapse 1.64.0rc2 (2022-07-29)
==============================

This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406))

Synapse 1.64.0rc1 (2022-07-26)
==============================

This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu.

Features
--------

- Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038))
- Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205))
- Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208))
- Add support for room version 10. ([\matrix-org#13220](matrix-org#13220))
- Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276))
- Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263))
- Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270))
- Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278))
- Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296))
- Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350))
- Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352))

Improved Documentation
----------------------

- Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231))
- Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261))
- Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271))
- Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314))
- Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333))
- Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338))
- Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344))

Deprecations and Removals
-------------------------

- Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967))
- Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192))
- Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239))
- Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326))

Internal Changes
----------------

- Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942))
- Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943))
- Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094))
- Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172))
- Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175))
- Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198))
- Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215))
- Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218))
- Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224))
- Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240))
- Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308))
- Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251))
- Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257))
- Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258))
- Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260))
- Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266))
- Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274))
- Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279))
- Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307))
- Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285))
- Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292))
- Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297))
- Improve performance of query  `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299))
- Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300))
- Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303))
- Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310))
- Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311))
- Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313))
- Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318))
- Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320))
- Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323))
- Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324))
- Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362))
- Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349))
- When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342))
- Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354))
  • Loading branch information
Alladin9393 committed Aug 8, 2022
2 parents db00818 + c2f4871 commit b4863c7
Show file tree
Hide file tree
Showing 1,441 changed files with 151,240 additions and 78,361 deletions.
13 changes: 0 additions & 13 deletions .buildkite/.env

This file was deleted.

35 changes: 0 additions & 35 deletions .buildkite/merge_base_branch.sh

This file was deleted.

37 changes: 0 additions & 37 deletions .buildkite/scripts/create_postgres_db.py

This file was deleted.

16 changes: 0 additions & 16 deletions .buildkite/scripts/test_old_deps.sh

This file was deleted.

36 changes: 0 additions & 36 deletions .buildkite/scripts/test_synapse_port_db.sh

This file was deleted.

10 changes: 0 additions & 10 deletions .buildkite/worker-blacklist

This file was deleted.

93 changes: 93 additions & 0 deletions .ci/complement_package.gotpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
{{- /*gotype: github.com/haveyoudebuggedit/gotestfmt/parser.Package*/ -}}
{{- /*
This template contains the format for an individual package. GitHub actions does not currently support nested groups so
we are creating a stylized header for each package.

This template is based on https://github.com/haveyoudebuggedit/gotestfmt/blob/f179b0e462a9dcf7101515d87eec4e4d7e58b92a/.gotestfmt/github/package.gotpl
which is under the Unlicense licence.
*/ -}}
{{- $settings := .Settings -}}
{{- if and (or (not $settings.HideSuccessfulPackages) (ne .Result "PASS")) (or (not $settings.HideEmptyPackages) (ne .Result "SKIP") (ne (len .TestCases) 0)) -}}
{{- if eq .Result "PASS" -}}
{{ "\033" }}[0;32m
{{- else if eq .Result "SKIP" -}}
{{ "\033" }}[0;33m
{{- else -}}
{{ "\033" }}[0;31m
{{- end -}}
📦 {{ .Name }}{{- "\033" }}[0m
{{- with .Coverage -}}
{{- "\033" -}}[0;37m ({{ . }}% coverage){{- "\033" -}}[0m
{{- end -}}
{{- "\n" -}}
{{- with .Reason -}}
{{- " " -}}🛑 {{ . -}}{{- "\n" -}}
{{- end -}}
{{- with .Output -}}
{{- . -}}{{- "\n" -}}
{{- end -}}
{{- with .TestCases -}}
{{- /* Failing tests are first */ -}}
{{- range . -}}
{{- if and (ne .Result "PASS") (ne .Result "SKIP") -}}
::group::{{ "\033" }}[0;31m❌{{ " " }}{{- .Name -}}
{{- "\033" -}}[0;37m ({{if $settings.ShowTestStatus}}{{.Result}}; {{end}}{{ .Duration -}}
{{- with .Coverage -}}
, coverage: {{ . }}%
{{- end -}})
{{- "\033" -}}[0m
{{- "\n" -}}

{{- with .Output -}}
{{- formatTestOutput . $settings -}}
{{- "\n" -}}
{{- end -}}

::endgroup::{{- "\n" -}}
{{- end -}}
{{- end -}}


{{- /* Then skipped tests are second */ -}}
{{- range . -}}
{{- if eq .Result "SKIP" -}}
::group::{{ "\033" }}[0;33m🚧{{ " " }}{{- .Name -}}
{{- "\033" -}}[0;37m ({{if $settings.ShowTestStatus}}{{.Result}}; {{end}}{{ .Duration -}}
{{- with .Coverage -}}
, coverage: {{ . }}%
{{- end -}})
{{- "\033" -}}[0m
{{- "\n" -}}

{{- with .Output -}}
{{- formatTestOutput . $settings -}}
{{- "\n" -}}
{{- end -}}

::endgroup::{{- "\n" -}}
{{- end -}}
{{- end -}}


{{- /* Then passing tests are last */ -}}
{{- range . -}}
{{- if eq .Result "PASS" -}}
::group::{{ "\033" }}[0;32m✅{{ " " }}{{- .Name -}}
{{- "\033" -}}[0;37m ({{if $settings.ShowTestStatus}}{{.Result}}; {{end}}{{ .Duration -}}
{{- with .Coverage -}}
, coverage: {{ . }}%
{{- end -}})
{{- "\033" -}}[0m
{{- "\n" -}}

{{- with .Output -}}
{{- formatTestOutput . $settings -}}
{{- "\n" -}}
{{- end -}}

::endgroup::{{- "\n" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- "\n" -}}
{{- end -}}
4 changes: 4 additions & 0 deletions .ci/latest_deps_build_failed_issue_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: CI run against latest deps is failing
---
See https://github.com/{{env.GITHUB_REPOSITORY}}/actions/runs/{{env.GITHUB_RUN_ID}}
8 changes: 3 additions & 5 deletions .buildkite/postgres-config.yaml → .ci/postgres-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,17 @@
# CI's Docker setup at the point where this file is considered.
server_name: "localhost:8800"

signing_key_path: "/src/.buildkite/test.signing.key"
signing_key_path: ".ci/test.signing.key"

report_stats: false

database:
name: "psycopg2"
args:
user: postgres
host: postgres
host: localhost
password: postgres
database: synapse

# Suppress the key server warning.
trusted_key_servers:
- server_name: "matrix.org"
suppress_key_server_warning: true
trusted_key_servers: []
25 changes: 25 additions & 0 deletions .ci/scripts/checkout_complement.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/bin/bash
#
# Fetches a version of complement which best matches the current build.
#
# The tarball is unpacked into `./complement`.

set -e
mkdir -p complement

# Pick an appropriate version of complement. Depending on whether this is a PR or release,
# etc. we need to use different fallbacks:
#
# 1. First check if there's a similarly named branch (GITHUB_HEAD_REF
# for pull requests, otherwise GITHUB_REF).
# 2. Attempt to use the base branch, e.g. when merging into release-vX.Y
# (GITHUB_BASE_REF for pull requests).
# 3. Use the default complement branch ("HEAD").
for BRANCH_NAME in "$GITHUB_HEAD_REF" "$GITHUB_BASE_REF" "${GITHUB_REF#refs/heads/}" "HEAD"; do
# Skip empty branch names and merge commits.
if [[ -z "$BRANCH_NAME" || $BRANCH_NAME =~ ^refs/pull/.* ]]; then
continue
fi

(wget -O - "https://github.com/matrix-org/complement/archive/$BRANCH_NAME.tar.gz" | tar -xz --strip-components=1 -C complement) && break
done
27 changes: 16 additions & 11 deletions ...e/replication/slave/storage/appservice.py → .ci/scripts/postgres_exec.py
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# -*- coding: utf-8 -*-
# Copyright 2015, 2016 OpenMarket Ltd
# Copyright 2018 New Vector Ltd
#!/usr/bin/env python
# Copyright 2019 The Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -14,13 +13,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.

from synapse.storage.databases.main.appservice import (
ApplicationServiceTransactionWorkerStore,
ApplicationServiceWorkerStore,
)
import sys

import psycopg2

# a very simple replacment for `psql`, to make up for the lack of the postgres client
# libraries in the synapse docker image.

class SlavedApplicationServiceStore(
ApplicationServiceTransactionWorkerStore, ApplicationServiceWorkerStore
):
pass
# We use "postgres" as a database because it's bound to exist and the "synapse" one
# doesn't exist yet.
db_conn = psycopg2.connect(
user="postgres", host="localhost", password="postgres", dbname="postgres"
)
db_conn.autocommit = True
cur = db_conn.cursor()
for c in sys.argv[1:]:
cur.execute(c)
36 changes: 36 additions & 0 deletions .ci/scripts/setup_complement_prerequisites.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/sh
#
# Common commands to set up Complement's prerequisites in a GitHub Actions CI run.
#
# Must be called after Synapse has been checked out to `synapse/`.
#
set -eu

alias block='{ set +x; } 2>/dev/null; func() { echo "::group::$*"; set -x; }; func'
alias endblock='{ set +x; } 2>/dev/null; func() { echo "::endgroup::"; set -x; }; func'

block Set Go Version
# The path is set via a file given by $GITHUB_PATH. We need both Go 1.17 and GOPATH on the path to run Complement.
# See https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#adding-a-system-path

# Add Go 1.17 to the PATH: see https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-Readme.md#environment-variables-2
echo "$GOROOT_1_17_X64/bin" >> $GITHUB_PATH
# Add the Go path to the PATH: We need this so we can call gotestfmt
echo "~/go/bin" >> $GITHUB_PATH
endblock

block Install Complement Dependencies
sudo apt-get -qq update && sudo apt-get install -qqy libolm3 libolm-dev
go get -v github.com/haveyoudebuggedit/gotestfmt/v2/cmd/gotestfmt@latest
endblock

block Install custom gotestfmt template
mkdir .gotestfmt/github -p
cp synapse/.ci/complement_package.gotpl .gotestfmt/github/package.gotpl
endblock

block Check out Complement
# Attempt to check out the same branch of Complement as the PR. If it
# doesn't exist, fallback to HEAD.
synapse/.ci/scripts/checkout_complement.sh
endblock
Loading

0 comments on commit b4863c7

Please sign in to comment.