Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Lock Dependencies in Synapse #11537

Closed
callahad opened this issue Dec 8, 2021 · 46 comments
Closed

Lock Dependencies in Synapse #11537

callahad opened this issue Dec 8, 2021 · 46 comments
Labels
T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. z-p2 (Deprecated Label)

Comments

@callahad
Copy link
Contributor

callahad commented Dec 8, 2021

Hi folks,

Right now most of Synapse's dependencies only declare a minimum version bound, like Twisted>=18.9.0 (cite). This means that every time we build release artifacts or install Synapse from source, we unconditionally pull in the latest versions of our dependencies at that moment.

This creates unnecessary risk, as our dependencies can change out from under us without warning. For example, installing Synapse 1.49.0 may work today but fail tomorrow if one of our dependencies releases a new version overnight.

This exact scenario bit us with the release of attrs 21.1.0 (synapse#9936) on May 6th. We were forced to release Synapse 1.33.1 less than a day after 1.33.0 as the attrs release broke our ability to install Synapse from source, build working Debian packages, or create functioning Docker images, even though nothing in our repositories had changed.

The absence of locked dependencies also impedes our ability to pursue continuous delivery, maintain LTS releases, or easily backport security fixes as we cannot recreate older release artifacts without also implicitly updating all of the dependencies included therein.

Definition of Done

  • For any given Synapse commit, it is possible to repeatably create identical virtualenvs.

Further Discussion / Constraints

Resolving this implies that it must be possible to enumerate exact versions of all dependencies included in any given upstream release of Synapse, using only a clone of the Synapse repository. This is important for auditing, as it allows us to easily answer questions like "did we ever ship a release with a vulnerable version of that dependency?"

Moreover, any solution must record hash digests to protect against remote tampering, such as with pip's hash-checking mode.

To ease maintenance burden (and avail of GitHub's supply chain security features), it would be nice if whatever solution we arrived at integrated with Dependabot. Supported package ecosystems for Python are requirements.txt (pip / pip-tools), pipfile.lock (Pipenv), and poetry.lock (Poetry).

@callahad callahad added P2 T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. labels Dec 8, 2021
@richvdh
Copy link
Member

richvdh commented Dec 8, 2021

"did we ever ship a release with a vulnerable version of that dependency?"

this appears to assume that our releases include their dependencies. We do not vendor our dependencies into the pypi release, with good reason.

I think what you mean is "did we ever ship docker images for a release with a vulnerable version of that dependency?".

It's a bit of a terminology nit, but an important one imho: Synapse releases are distinct from those of our dependencies.

@callahad
Copy link
Contributor Author

callahad commented Dec 8, 2021

I think what you mean is "did we ever ship docker images for a release with a vulnerable version of that dependency?".

Sure, Docker images or Debian packages. Our built release artifacts / things we're responsible for as a team. Downstream packagers and people directly consuming a Synapse wheel or sdist are outside of the scope of this proposal.

@clokep
Copy link
Member

clokep commented Dec 8, 2021

For any given Synapse commit, it is possible to repeatably create identical virtualenvs.

From previous conversations my understanding was that it was important to be able to create an identical virtualenv for any release of Synapse, not any commit. (I think this is related to what @richvdh is saying though.) Those aren't quite the same as described though.

@richvdh
Copy link
Member

richvdh commented Dec 8, 2021

This is fine, and as you note has many advantages, but I do have a counter-requirement, which is:

We should not fall into the trap of only supporting the pinned version of a dependency. To take an arbitrary example, we currently require cryptography>=3.4.7. That's eight months old now, and being a crypto thing, if we were pinning, we would probably pin to the most recent version. However, doing so would make packaging Synapse for distributions effectively impossible.

I therefore think that any "pinned" versions should be in addition to the current "minimum" versions, where the minimum versions are also subject to a certain amount of testing (as we have presently).

(Exactly how you you would choose whether to use the "pinned" version or the "minimum" version is very much up for debate)

@callahad
Copy link
Contributor Author

callahad commented Dec 8, 2021

my understanding was that it was important to be able to create an identical virtualenv for any release of Synapse, not any commit.

@clokep: Is this a distinction without a difference? Even if we were only dumping pip freeze into the first commit of a release branch, we'd end up with that frozen requirements.txt file back on develop once the release branch merged back in. This would nominally satisfy the requirement of being able to create identical virtualenvs for any commit from that point on.

Functionally, I suspect we'll prefer to actively maintain the lockfile with the rest of our source during development, rather than only touching it when making release branches, but we'll see how things shake out.

We should not fall into the trap of only supporting the pinned version of a dependency. [...] I therefore think that any "pinned" versions should be in addition to the current "minimum" versions, where the minimum versions are also subject to a certain amount of testing (as we have presently).

@richvdh Happy to go along with that here. The standard practice, across languages, is to separate declared minimums from locked versions (setup.py vs requirements.txt, package.json vs package-lock.json, Cargo.toml vs Cargo.lock, Gemfile vs Gemfile.lock, etc.), so we'll be able to do that.

The question of what our minimums should be (or when to raise them) is a separate question, and one we already face regardless of this proposal.

@clokep
Copy link
Member

clokep commented Dec 8, 2021

my understanding was that it was important to be able to create an identical virtualenv for any release of Synapse, not any commit.

@clokep: Is this a distinction without a difference? Even if we were only dumping pip freeze into the first commit of a release branch, we'd end up with that frozen requirements.txt file back on develop once the release branch merged back in. This would nominally satisfy the requirement of being able to create identical virtualenvs for any commit from that point on.

It is different, yes. There's an assumption being made that develop should have this on it. I'm unsure if I agree with that.

Functionally, I suspect we'll prefer to actively maintain the lockfile with the rest of our source during development, rather than only touching it when making release branches, but we'll see how things shake out.

Perhaps, but that doesn't seem to be one of the requirements.

@callahad
Copy link
Contributor Author

callahad commented Dec 8, 2021

Perhaps, but that doesn't seem to be one of the requirements.

Let's take the requirement as stated: any commit, not just releases, must have enough information (a lockfile) to reliably recreate identical virtualenvs.

This is a slight leap, but one I believe is inevitable as maintaining a lockfile on the main branch is a prerequisite for availing of Dependabot's features, which are genuinely compelling.

Enabling repeatable builds for arbitrary commits is also necessary for continuous delivery, but that's a distinct concern best left for the future.

@callahad
Copy link
Contributor Author

callahad commented Dec 9, 2021

There's general consensus that we're fine going ahead with this. The question is how. Next month we'll pick a victim volunteer to research Poetry, Pipenv, and pip-tools and make a recommendation.

We'd likely benefit from proactively simplifying setup.py and synapse/python_dependencies.py in advance as Poetry replaces setuptools entirely, while pip-tools and Pipenv overlap with it.

@callahad
Copy link
Contributor Author

callahad commented Dec 9, 2021

(Braindump) A few questions to ask of a given solution:

  • How does it handle locking dependencies with environment markers like when we specify psycopg2>=2.8;platform_python_implementation!='PyPy'?
    • Are those dependencies resolved and present in the lockfile even if they're not currently installed or wouldn't be installed in the current environment?
      This is important if we have conditional dependencies (including indirect ones!) since their absence would mean our lockfile wasn't complete and deterministic.
  • What about groups like lint, dev, etc?
  • Does it help us know whether we're up to date with upstream releases?
  • How easy is it to determine where a transitive dependency is coming from?
  • Is the lockfile reasonably legible / intelligible? What about the diffs when it gets updated?

Also worth noting that PyCharm supports both Pipenv and Poetry

@callahad
Copy link
Contributor Author

(PDM and Conda are also in the packaging space, but still somewhat esoteric / specialized)

@DMRobertson
Copy link
Contributor

DMRobertson commented Jan 12, 2022

Other writings on the topic of Pipenv vs Poetry:

While reading around I also came across pyflow, micropipenv and pip-tools.

Frankly I feel paralysed by too much choice. I'll try to see if I can summarise this into something digestible.

@DMRobertson DMRobertson self-assigned this Jan 12, 2022
@callahad
Copy link
Contributor Author

Frankly I feel paralysed by too much choice. I'll try to see if I can summarise this into something digestible.

If integrating with GitHub's Depenabot security stuff is a hard requirement (and it might as well be), our only choices are Poetry, Pipenv, and pip-tools. I'd recommend stepping back from others' opinions and blog posts and just diving in on your own:

  • Package up a Hello World example with each tool.
  • Then try packaging a smaller library like signedjson.
  • Then try Sydent or Sygnal.

Don't worry about CI or anything... just get a feel for how each tool works, how its docs are, etc.

After, go and try to answer the questions from this comment for each.

@DMRobertson
Copy link
Contributor

DMRobertson commented Jan 12, 2022

Right, I've started at a lot of the above and web-pages. I concentrated on the pip-tools, pipenv and poetry because they were dependabot supported. There's an info dump coming up, but TL;DR:

  • I reckon we could use any of these to manage our deps.
  • From reading webpages, docs and --help alone, I have a slight preference for pip-tools.

Agree it needs to be put into practice.

@DMRobertson
Copy link
Contributor

DMRobertson commented Jan 12, 2022

Preliminary infodump

Requirement pip-tools pipenv poetry
new venv from lock pip-sync requirements.txt [...] pipenv install poetry install
checksumming hashes checked by pip automatic, but not sure of details automatic I think
add dependency add to requirements.in; pip-compile pipenv install PACKAGE~=VER poetry add PACKAGE
edit dependency bound edit requirements.in; pip-compile; edit Pipfile and pipenv lock, or pipenv install edit pyproject.toml, poetry lock; or poetry add
get newest package ver pip-compile -P PACKAGE pipenv update PACKAGE poetry update PACKAGE
remove dependency remove from requirements.in; pip-compile pipenv uninstall PACKAGE poetry remove PACKAGE
lock with checksum pip-compile --generate-hashes pipenv lock poetry lock
editable installations Unsure; -e . in requirements.in, but bugs? pipenv install -e . editable by default
minimum ver while pinning
environment markers run once per environment, multiple requirements.txt own syntax in Pipfile. Unsure how it's treated in lock. supported, dunno about lock
Multiple dep categories pip-compile --extra GROUPNAME reads setup.py; multiple infiles? Just --dev and --dev-only in various cmds supported, dunno about lock
Detect newer versions pip-compile --dry-run and diff? pipenv update --outdated poetry show --outdated
Detect security problems none, pipe to safety pipenv check; uses Pyup safety none, pipe to safety
Dependabot support 6.1.0 <= 2021-05-29 v1
Explain given dependency pip-compile --annotate pipenv graph poetry show --tree
PyCharm support None, just requirements.txt Yes Yes
Override bad dep metadata Unsure--what does pip allow? Unsure No. Called out in HN comment.
Dry run pip-compile -n; pip-sync -n pipenv update --dry-run --dry-run everywhere
Conflict resolution Defers to pip Unsure Home-grown, claimed as better.
Venv access normal venv? pipenv run or pipenv shell poetry run or poetry shell
Other Can pin distribute, pip, setuptools Historical drama, inactivity; build and publish shortcuts
"Ideally, you should only have one target Python version." Trying to be all-in-one?
Unsure how to expose depencies on PyPI Seems trendier/more active/more favoured?
Custom Pipfile syntax? Manages multiple venvs?

Notes

  • all three have a separate specfile and lockfile
  • piptools simplest. You specify a requirements.in file; this gets processed into a requirements.txt file. Sync that to your venv.
  • The other two have commands that manage the spec file and the lock file simultaneously
  • poetry seems to be a one-stop shop closer to cargo (also builds, publishes)
  • poetry seemed to have more dependencies when I installed it
  • the chatter online seems to be slightly pro-poetry. Claims that pipenv is/was slower, buggier.
  • pipenv suffered a reputational hit with historical drama. Later, there was a 1.5 year period without a release.
  • pipenv focused on applications, poetry applications and libraries? Unsure here---they all seem to cover our use case
  • does poetry make setup.py redundant? does it aim to?

Which to pick?

Ultimately I think we could choose any one of these and there's not much between them for our purposes. On the face of it, I think I'd recommend pip-tools > poetry > pipenv. Rationale:

Pip-tools pro:

  • pip-tools is simplest, and leaves us in full control of the "spec file", not the tool
  • it doesn't try to manage a venv for you --- I think this would result in the least friction for the team
  • it relies on pip for its package resolving, so is "closer" somehow to what you'll get by naively installing synapse from pypi
  • There were poetry warnings on hacker news about not being able to override specific dependencies, but I couldn't see anything enabling this in the other two. If anything, being closer to pip might give pip-tools an edge here? (I think you can override pip if you think you know best.)

pip-tools con:

  • need to test what happens with editable installs
  • if we want to have lots of dev test lint environments, I think we'll need to have separate inputs and output requirements files which refer to each other.

Poetry pro:

  • has community momentum
  • dependency tree output
  • claims to have better and less buggy dep resolution
  • I think I slightly preferred its --help to pipenv's

Poetry cons:

  • might have a hard time if our dependencies' dependencies are incorrectly labelled
  • poetry would have us put lots more stuff into pyproject.toml. Unwieldy?

@callahad
Copy link
Contributor Author

does poetry make setup.py redundant? does it aim to?

Yes to both; in addition to managing environments, Poetry is a complete replacement for Setuptools. This includes related build + publishing tools (twine, build, etc.).

Pipenv and pip-tools are concerned only with installing dependencies; they want nothing to do with packaging. (Pipenv also manages venvs for you, pip-tools expects you to manage your own venvs).

Detect newer versions [...] poetry update --dry-run

I'll also note that poetry show --outdated does an excellent job of summarizing the state of the world:

Screen Shot 2022-01-12 at 22 10 36

The red vs yellow coloring is based on whether the upgrade would be "safe" according to semver (e.g., the leftmost part of the version didn't change). This makes it much easier to keep up to date if manually updating things: the versions in red are likely to be the quickest / easiest to knock out, while the yellow ones might have backwards compatibility concerns and thus take more work.

@DMRobertson
Copy link
Contributor

DMRobertson commented Jan 14, 2022

https://github.com/DMRobertson/test-packaging has a branch for each tool under consideration.

@DMRobertson
Copy link
Contributor

Right, here's wot I think.

Of the three tools supported by dependabot:

  • we could use any of these to pin dependencies.
  • each include sha256 hashes from pypi and verifies them at install time
  • each has some kind of integration with pycharm
  • lockfiles and spec files all reasonably human readable
  • all three export a requirements.txt, so in the worst case scenario we can reuse the lock files with pip or pip-tools.
  • all three can take over an existing virtual environment

TL;DR

I reckon we should use poetry because its lock file is comprehensive, it manages a single source of truth, and has pretty good documentation with a nice CLI UI. pip-tools is more awkward if you want to make use of environment markers. pipenv left me confused: its interplay with setuptools was unclear, I think partly because pipenv is intended for managing application environments, not libraries. Also, many of pipenv's commands lack a --dry-run option.

In more detail:

pipenv

To start, I want to complain about pipenv. AFAICS it's designed for wrangling applications alone: a single set of fixed packages and versions, ideally running on a single fixed interpreter on a fixed host. If we were only supporting our debian packages and docker images, that'd be fine, but... we want to give our community packagers a reasonable range of lower bounds too.

Instead, for packaging a library (and publishing to PyPI?) you're expected to use existing setuptools infrastructure. Seeking an example, I went to see how pipenv itself used Pipfile and setup.py and setup.py. I was confused that

  • their Pipfile defines no dependencies, but setup.py does;
  • the two files seem to define different sets of dev dependencies

Perhaps pipenv will read from setup.py/cfg much like pip-tools can? It must do, since the packages in setup.py appear in the lockfile. If so, this isn't clear from the documentation. And while I'm at it, generally speaking I found their documentation to be more of a tutorial than a reference; I found it hard to use it to answer my questions.

Pros:

  • multiple environments in one lockfile
  • built in safety check. Other tools have to export and pipe to safety
  • shows up in lots of examples on https://packaging.python.org/en/latest/ . Pseudo-official.
  • editable installs work with one config line (and are part of the lockfile)

Cons:

  • I'm pretty confused about how I'm supposed to use it overall. Wasn't clear if we'd need to duplicate info in setup.py in a Pipfile.
  • Didn't always have a --dry-run option.
  • Seemed a little sluggish (though not really a fair evaluation).

Other:

  • what does the colour coding in pipenv help denote? no idea.

pip-tools

Pip-tools was my initial preference. I thought it would be simplest. Given a virtualenv, you manually control the spec file, run one command to make the lock file, and then one command to update your virtualenv. Regrettably it took a bit of exploring to work a few things out. I had to opt-in to have the tool --generate-hashes. It took me a moment to realise that pip-compile can parse setup.cfg (and maybe setup.py)? Once I'd realised that, great---we have a single source of truth for our package requirements.

Unfortuntely the lockfile only caters to a single environment at a time. That means a separate lockfile for a dev environment, a separate one for a pypy environment, a separate one for a python 3.8-with-backport-exclusive-to-3.8, ... it seemed a little messy and the other tools could account for this.

I think the only other thing worth saying is that the CLI isn't as fancy. There isn't an easy way to query the state of the lockfile: what dependencies could be updated?

Pros:

  • single source of truth for requirements
  • Simple. The most lightweight.
  • I didn't feel like the tool was trying to do anything magic.
  • Unixy: do one thing well.

Cons:

  • The lockfile corresponds to one environment (i.e. interpreter + OS).
  • unclear if pip-tools can version itself. I had warnings when trying to do so.
  • No nice way to see if anything is outdated; re-lock and git diff.
  • BAD THINGS HAPPENED when I tried to pip install --user pip-tools. (sync tried to nuke my system virtual env)
  • couldn't get editable installs working.
  • scary error message when I messed up setup.cfg

poetry

If I'm being honest, by this time I was tired of the packaging swamp that Python has created for itself. Poetry covers the same ground as pipenv, but it's more ambitious. I think poetry is trying to be cargo for the Python world.

One major difference: it eschews setuptools---the equivalent metadata has to go in pyproject.toml, and poetry itself will try to build the project. I had a quick look at our wheel build for synapse, and we don't seem to have any fancy C dependencies. I think this means that we're not doing anything clever in setup.py, and that using poetry instead should be fairly straightforward. (There's no reason we have to poetry for this, but we wouldn't want to maintain parallel config for two systems). I wasn't sure if things would get complicated if we wanted to build things in rust and link them with Py03. But it looks like https://github.com/pyo3/maturin#mixed-rustpython-projects might be a different tool to use for this? I'm not sure.

The killer feature for me though was that the lockfile describes a locked version of dependencies for all environments. Take for instance this one:

  [[package]]
  name = "black"
  #...
    
  [package.dependencies]
  #... 
  typing-extensions = [
     {version = ">=3.10.0.0", markers = "python_version < \"3.10\""},
     {version = "!=3.10.0.1", markers = "python_version >= \"3.10\""},
  ]

The CLI/UI is nice enough but basically comparable to pipenv. (I liked the more prevalent --dry-run and interactive options). Documentation was a good enough reference, if a little barebones.

Pros:

  • multiple environments in the lockfile.
  • single source of truth for the requirements.
  • felt snappy.
  • Error messages were helpful (but probably didn't need to include a python traceback)
  • --dry-run everywhere.
  • Has its own foo^=1.2.3 syntax for soft-pining to 1.2.3 or anything semver compatible. That's slightly more flexible than what pip supports (foo~=1.2.3).

Cons:

  • Quite a lot of options on the main --help menu. A little overwhelming.
  • Biggest of the three tools (in terms of its dependencies)
  • More work for us to migrate, since it replaces setuptools
  • Potentially inflexible if there is a conflict between transitive dependencies. (Worst case: export requirements.txt and fall back to pip-tools?)

Other:

speed

No-op lock + update venvs. Same requirements + the tool itself.

  • 3s to pip-compile setup.cfg --generate-hashes --extra dev,test,lint,meta -o requirements-dev.txt && pip-sync requirements-dev.txt
  • 12s to pipenv update --dev, mostly in locking
  • 6s to poetry lock && poetry install

closing thoughts

I would really like the core developers to have sorted this out (c.f. cargo) so there isn't fragmentation. Live and learn, I suppose.

@clokep
Copy link
Member

clokep commented Jan 19, 2022

@DMRobertson It isn't 100% clear to me, but do any of these tools handle "extras"? It seems like for pip-tools it would be completely manual, but does poetry? (E.g. right now you can pip install synapse[oidc] -- how does this work with pinning?)

@reivilibre
Copy link
Contributor

Thanks David — that was quite a good run-down.

Since I had to look it up: safety check seems to refer to https://pypi.org/project/safety/ — a way to list known security vulnerabilities in your dependencies.

@DMRobertson
Copy link
Contributor

@DMRobertson It isn't 100% clear to me, but do any of these tools handle "extras"? It seems like for pip-tools it would be completely manual, but does poetry? (E.g. right now you can pip install synapse[oidc] -- how does this work with pinning?)

There's syntax for this but let me work through an example to see how it works with a lock file.

@richvdh
Copy link
Member

richvdh commented Jan 19, 2022

This is a great write-up, thanks so much @DMRobertson!

I don't think it's sufficiently important that it should influence our decision, but currently synapse does a run-time check on all the dependencies, as a way to warn people if their virtualenvs are outdated (and to stop them reporting bugs caused by outdated deps). Is such a check possible under poetry?

@callahad
Copy link
Contributor Author

There's no expectation that Poetry is present at runtime; it's primarily used during development and packaging. Running poetry build produces a normal Python wheel which includes requirements metadata in a standard format. You don't need Poetry to pip install synapse.

Thus, if we want run-time checks, we should re-implement those functions from python_dependencies.py in terms of importlib.metadata (backport) from the standard library. We should do this regardless (preferring standard approaches where possible), but we must do so if we move to Poetry, since it will replace our use of setuptools and thus the information used by those functions won't be present where they expect it to be.

@DMRobertson
Copy link
Contributor

Dan beat me to it. But it's nice to finally understand why python_dependencies exists as an executable python file in the first place.

@callahad
Copy link
Contributor Author

If we do want to maintain the runtime checks, we may also find packaging to be a useful library for parsing / comparing version specifiers (setuptools itself vendors packaging for this reason).

@DMRobertson
Copy link
Contributor

DMRobertson commented Jan 19, 2022

To address the recent batch of comments:

@DMRobertson It isn't 100% clear to me, but do any of these tools handle "extras"? It seems like for pip-tools it would be completely manual, but does poetry? (E.g. right now you can pip install synapse[oidc] -- how does this work with pinning?)

"extras" are supported by poetry. (Not pipenv, as far as I can tell?) I believe that the pinning will pick a version and hashes for anything that could possibly be installed with any combination of extras, but I'll check this.

I've just checked that the lockfile contains dependencies for python versions or other environment markers that you're not using. So I'm hopeful the same will ring true of extras.

(I think the top of the lockfile represents a dump of poetry's solver's output---across all extras, environments, etc and that gets used at install time to work out what to install. The rest of the lockfile is then the hashes to guard the supply chain.)

currently synapse does a run-time check on all the dependencies, as a way to warn people if their virtualenvs are outdated (and to stop them reporting bugs caused by outdated deps). I Is such a check possible under poetry?

@richvdh poetry wouldn't be used for this at runtime as Dan says. I'm haven't seen a way to ask poetry "does this environment meet the constraints, even if it doesn't match with the locked file"? It's concerned with producing an environment that matches a lockfile exactly, so only ever seems to compare an environment against a lockfile.


In other news, I noticed today that that poetry makes it pretty convenient to switch between a venv for different python versions. That could occasionally be useful for tricky backwards compatibility stuff.

I am slightly anxious at the number of open issues/prs on poetry's repository. I think it might be prudent to pin the version of poetry we use in CI. (The same applies to whichever tool we use, but I think poetry is the one that has the most activity going on.)

@DMRobertson
Copy link
Contributor

believe that the pinning will pick a version and hashes for anything that could possibly be installed with any combination of extras, but I'll check this.

AFAICS this is true. I defined an extras group here. Lockfile included the new packages, but poetry install didn't do anything until I asked for poetry install -E name_of_extra.

Are there any outstanding questions, comments or concerns? If not I'd like to try converting signedjson to use poetry and see how that goes.

DMRobertson pushed a commit that referenced this issue Mar 30, 2022
AFAICS these aren't used any more. They're going to break when I do
poetry work (#11537), because tox won't (easily) recreate the locked
poetry environment. I think it's less effort to throw it away.
DMRobertson pushed a commit that referenced this issue Mar 30, 2022
AFAICS these aren't used any more. They're going to break when I do
poetry work (#11537), because tox won't (easily) recreate the locked
poetry environment. I think it's less effort to throw it away.
DMRobertson pushed a commit that referenced this issue Mar 30, 2022
It only checks the manifest. Once #11537 lands there won't be a manifest
any more.
DMRobertson pushed a commit that referenced this issue Mar 30, 2022
It only checks the manifest. Once #11537 lands there won't be a manifest
any more.
DMRobertson pushed a commit that referenced this issue Mar 30, 2022
AFAICS these aren't used any more. They're going to break when I do
poetry work (#11537), because tox won't (easily) recreate the locked
poetry environment. I think it's less effort to throw it away than to
fix it.
babolivier added a commit to matrix-org/synapse-dinsic that referenced this issue Apr 28, 2022
Synapse 1.58.0rc1 (2022-04-26)
==============================

As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](matrix-org/synapse#11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61.

Features
--------

- Implement [MSC3383](matrix-org/matrix-spec-proposals#3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](matrix-org/synapse#11398))
- Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](matrix-org/synapse#11537))
- Enable processing of device list updates asynchronously. ([\#12365](matrix-org/synapse#12365), [\#12465](matrix-org/synapse#12465))
- Implement [MSC2815](matrix-org/matrix-spec-proposals#2815) to allow room moderators to view redacted event content. Contributed by @tulir. ([\#12427](matrix-org/synapse#12427))
- Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](matrix-org/synapse#12543))

Bugfixes
--------

- Prevent a sync request from removing a user's busy presence status. ([\#12213](matrix-org/synapse#12213))
- Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper. ([\#12319](matrix-org/synapse#12319))
- Fix a long-standing bug which incorrectly caused `GET /_matrix/client/v3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original. ([\#12476](matrix-org/synapse#12476))
- Fix a bug introduced in Synapse 1.27.0 where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1, no matter how many extremities were deleted. ([\#12496](matrix-org/synapse#12496))
- Fix a long-standing bug where the image thumbnails embedded into email notifications were broken. ([\#12510](matrix-org/synapse#12510))
- Fix a bug in the implementation of [MSC3202](matrix-org/matrix-spec-proposals#3202) where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`. ([\#12520](matrix-org/synapse#12520))
- Fix a bug introduced in Synapse 0.99.3 which could cause Synapse to consume large amounts of RAM when back-paginating in a large room. ([\#12522](matrix-org/synapse#12522))

Improved Documentation
----------------------

- Fix rendering of the documentation site when using the 'print' feature. ([\#12340](matrix-org/synapse#12340))
- Add a manual documenting config file options. ([\#12368](matrix-org/synapse#12368), [\#12527](matrix-org/synapse#12527))
- Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental. ([\#12451](matrix-org/synapse#12451))
- Update worker documentation and replace old `federation_reader` with `generic_worker`. ([\#12457](matrix-org/synapse#12457))
- Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](matrix-org/synapse#12475))
- Add some example configurations for workers and update architectural diagram. ([\#12492](matrix-org/synapse#12492))
- Fix a broken link in `README.rst`. ([\#12495](matrix-org/synapse#12495))
- Add HAProxy delegation example with CORS headers to docs. ([\#12501](matrix-org/synapse#12501))
- Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari. ([\#12533](matrix-org/synapse#12533))

Deprecations and Removals
-------------------------

- The groups/communities feature in Synapse is now disabled by default. ([\#12344](matrix-org/synapse#12344))
- Remove unstable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#12382](matrix-org/synapse#12382))

Internal Changes
----------------

- Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join. ([\#12394](matrix-org/synapse#12394))
- Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive. ([\#12399](matrix-org/synapse#12399))
- Remove an unstable identifier from [MSC3083](matrix-org/matrix-spec-proposals#3083). ([\#12395](matrix-org/synapse#12395))
- Run CI in the locked [Poetry](https://python-poetry.org/) environment, and remove corresponding `tox` jobs. ([\#12425](matrix-org/synapse#12425), [\#12434](matrix-org/synapse#12434), [\#12438](matrix-org/synapse#12438), [\#12441](matrix-org/synapse#12441), [\#12449](matrix-org/synapse#12449), [\#12478](matrix-org/synapse#12478), [\#12514](matrix-org/synapse#12514), [\#12472](matrix-org/synapse#12472))
- Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current iteration of [MSC2666](matrix-org/matrix-spec-proposals#2666). ([\#12445](matrix-org/synapse#12445))
- Fix typo in the release script help string. ([\#12450](matrix-org/synapse#12450))
- Fix a minor typo in the Debian changelogs generated by the release script. ([\#12497](matrix-org/synapse#12497))
- Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development. ([\#12455](matrix-org/synapse#12455))
- Limit length of `device_id` to less than 512 characters. ([\#12454](matrix-org/synapse#12454))
- Dockerfile-workers: reduce the amount we install in the image. ([\#12464](matrix-org/synapse#12464))
- Dockerfile-workers: give the master its own log config. ([\#12466](matrix-org/synapse#12466))
- complement-synapse-workers: factor out separate entry point script. ([\#12467](matrix-org/synapse#12467))
- Back out experimental implementation of [MSC2314](matrix-org/matrix-spec-proposals#2314). ([\#12474](matrix-org/synapse#12474))
- Fix grammatical error in federation error response when the room version of a room is unknown. ([\#12483](matrix-org/synapse#12483))
- Remove unnecessary configuration overrides in tests. ([\#12511](matrix-org/synapse#12511))
- Refactor the relations code for clarity. ([\#12519](matrix-org/synapse#12519))
- Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`. ([\#12528](matrix-org/synapse#12528))
- Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s. ([\#12468](matrix-org/synapse#12468))
- Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db`. ([\#12529](matrix-org/synapse#12529))
@DMRobertson DMRobertson added this to the Revisit: Next Month milestone May 3, 2022
babolivier added a commit to matrix-org/synapse-dinsic that referenced this issue May 3, 2022
Synapse 1.58.0 (2022-05-03)
===========================

As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](matrix-org/synapse#11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61.

No significant changes since 1.58.0rc2.

Synapse 1.58.0rc2 (2022-04-26)
==============================

This release candidate fixes bugs related to Synapse 1.58.0rc1's logic for handling device list updates.

Bugfixes
--------

- Fix a bug introduced in Synapse 1.58.0rc1 where the main process could consume excessive amounts of CPU and memory while handling sentry logging failures. ([\#12554](matrix-org/synapse#12554))
- Fix a bug introduced in Synapse 1.58.0rc1 where opentracing contexts were not correctly sent to whitelisted remote servers with device lists updates. ([\#12555](matrix-org/synapse#12555))

Internal Changes
----------------

- Reduce unnecessary work when handling remote device list updates. ([\#12557](matrix-org/synapse#12557))

Synapse 1.58.0rc1 (2022-04-26)
==============================

Features
--------

- Implement [MSC3383](matrix-org/matrix-spec-proposals#3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](matrix-org/synapse#11398))
- Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](matrix-org/synapse#11537))
- Enable processing of device list updates asynchronously. ([\#12365](matrix-org/synapse#12365), [\#12465](matrix-org/synapse#12465))
- Implement [MSC2815](matrix-org/matrix-spec-proposals#2815) to allow room moderators to view redacted event content. Contributed by @tulir @ Beeper. ([\#12427](matrix-org/synapse#12427))
- Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](matrix-org/synapse#12543))

Bugfixes
--------

- Prevent a sync request from removing a user's busy presence status. ([\#12213](matrix-org/synapse#12213))
- Fix bug with incremental sync missing events when rejoining/backfilling. Contributed by Nick @ Beeper. ([\#12319](matrix-org/synapse#12319))
- Fix a long-standing bug which incorrectly caused `GET /_matrix/client/v3/rooms/{roomId}/event/{eventId}` to return edited events rather than the original. ([\#12476](matrix-org/synapse#12476))
- Fix a bug introduced in Synapse 1.27.0 where the admin API for [deleting forward extremities](https://github.com/matrix-org/synapse/blob/erikj/fix_delete_event_response_count/docs/admin_api/rooms.md#deleting-forward-extremities) would always return a count of 1, no matter how many extremities were deleted. ([\#12496](matrix-org/synapse#12496))
- Fix a long-standing bug where the image thumbnails embedded into email notifications were broken. ([\#12510](matrix-org/synapse#12510))
- Fix a bug in the implementation of [MSC3202](matrix-org/matrix-spec-proposals#3202) where Synapse would use the field name `device_unused_fallback_keys`, rather than `device_unused_fallback_key_types`. ([\#12520](matrix-org/synapse#12520))
- Fix a bug introduced in Synapse 0.99.3 which could cause Synapse to consume large amounts of RAM when back-paginating in a large room. ([\#12522](matrix-org/synapse#12522))

Improved Documentation
----------------------

- Fix rendering of the documentation site when using the 'print' feature. ([\#12340](matrix-org/synapse#12340))
- Add a manual documenting config file options. ([\#12368](matrix-org/synapse#12368), [\#12527](matrix-org/synapse#12527))
- Update documentation to reflect that both the `run_background_tasks_on` option and the options for moving stream writers off of the main process are no longer experimental. ([\#12451](matrix-org/synapse#12451))
- Update worker documentation and replace old `federation_reader` with `generic_worker`. ([\#12457](matrix-org/synapse#12457))
- Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](matrix-org/synapse#12475))
- Add some example configurations for workers and update architectural diagram. ([\#12492](matrix-org/synapse#12492))
- Fix a broken link in `README.rst`. ([\#12495](matrix-org/synapse#12495))
- Add HAProxy delegation example with CORS headers to docs. ([\#12501](matrix-org/synapse#12501))
- Remove extraneous comma in User Admin API's device deletion section so that the example JSON is actually valid and works. Contributed by @olmari. ([\#12533](matrix-org/synapse#12533))

Deprecations and Removals
-------------------------

- The groups/communities feature in Synapse is now disabled by default. ([\#12344](matrix-org/synapse#12344))
- Remove unstable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#12382](matrix-org/synapse#12382))

Internal Changes
----------------

- Preparation for faster-room-join work: start a background process to resynchronise the room state after a room join. ([\#12394](matrix-org/synapse#12394))
- Preparation for faster-room-join work: Implement a tracking mechanism to allow functions to wait for full room state to arrive. ([\#12399](matrix-org/synapse#12399))
- Remove an unstable identifier from [MSC3083](matrix-org/matrix-spec-proposals#3083). ([\#12395](matrix-org/synapse#12395))
- Run CI in the locked [Poetry](https://python-poetry.org/) environment, and remove corresponding `tox` jobs. ([\#12425](matrix-org/synapse#12425), [\#12434](matrix-org/synapse#12434), [\#12438](matrix-org/synapse#12438), [\#12441](matrix-org/synapse#12441), [\#12449](matrix-org/synapse#12449), [\#12478](matrix-org/synapse#12478), [\#12514](matrix-org/synapse#12514), [\#12472](matrix-org/synapse#12472))
- Change Mutual Rooms' `unstable_features` flag to `uk.half-shot.msc2666.mutual_rooms` which matches the current iteration of [MSC2666](matrix-org/matrix-spec-proposals#2666). ([\#12445](matrix-org/synapse#12445))
- Fix typo in the release script help string. ([\#12450](matrix-org/synapse#12450))
- Fix a minor typo in the Debian changelogs generated by the release script. ([\#12497](matrix-org/synapse#12497))
- Reintroduce the list of targets to the linter script, to avoid linting unwanted local-only directories during development. ([\#12455](matrix-org/synapse#12455))
- Limit length of `device_id` to less than 512 characters. ([\#12454](matrix-org/synapse#12454))
- Dockerfile-workers: reduce the amount we install in the image. ([\#12464](matrix-org/synapse#12464))
- Dockerfile-workers: give the master its own log config. ([\#12466](matrix-org/synapse#12466))
- complement-synapse-workers: factor out separate entry point script. ([\#12467](matrix-org/synapse#12467))
- Back out experimental implementation of [MSC2314](matrix-org/matrix-spec-proposals#2314). ([\#12474](matrix-org/synapse#12474))
- Fix grammatical error in federation error response when the room version of a room is unknown. ([\#12483](matrix-org/synapse#12483))
- Remove unnecessary configuration overrides in tests. ([\#12511](matrix-org/synapse#12511))
- Refactor the relations code for clarity. ([\#12519](matrix-org/synapse#12519))
- Add type hints so `docker` and `stubs` directories pass `mypy --disallow-untyped-defs`. ([\#12528](matrix-org/synapse#12528))
- Update `delay_cancellation` to accept any awaitable, rather than just `Deferred`s. ([\#12468](matrix-org/synapse#12468))
- Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db`. ([\#12529](matrix-org/synapse#12529))
PiotrKozimor added a commit to globekeeper/synapse that referenced this issue May 5, 2022
Synapse 1.57.0 (2022-04-19)
===========================

This version includes a [change](matrix-org#12209) to the way transaction IDs are managed for application services. If your deployment uses a dedicated worker for application service traffic, **it must be stopped** when the database is upgraded (which normally happens when the main process is upgraded), to ensure the change is made safely without any risk of reusing transaction IDs.

See the [upgrade notes](https://github.com/matrix-org/synapse/blob/v1.57.0rc1/docs/upgrade.md#upgrading-to-v1570) for more details.

No significant changes since 1.57.0rc1.

Synapse 1.57.0rc1 (2022-04-12)
==============================

Features
--------

- Send device list changes to application services as specified by [MSC3202](matrix-org/matrix-spec-proposals#3202), using unstable prefixes. The `msc3202_transaction_extensions` experimental homeserver config option must be enabled and `org.matrix.msc3202: true` must be present in the application service registration file for device list changes to be sent. The "left" field is currently always empty. ([\matrix-org#11881](matrix-org#11881))
- Optimise fetching large quantities of missing room state over federation. ([\matrix-org#12040](matrix-org#12040))
- Offload the `update_client_ip` background job from the main process to the background worker, when using Redis-based replication. ([\matrix-org#12251](matrix-org#12251))
- Move `update_client_ip` background job from the main process to the background worker. ([\matrix-org#12252](matrix-org#12252))
- Add a module callback to react to new 3PID (email address, phone number) associations. ([\matrix-org#12302](matrix-org#12302))
- Add a configuration option to remove a specific set of rooms from sync responses. ([\matrix-org#12310](matrix-org#12310))
- Add a module callback to react to account data changes. ([\matrix-org#12327](matrix-org#12327))
- Allow setting user admin status using the module API. Contributed by Famedly. ([\matrix-org#12341](matrix-org#12341))
- Reduce overhead of restarting synchrotrons. ([\matrix-org#12367](matrix-org#12367), [\matrix-org#12372](matrix-org#12372))
- Update `/messages` to use historic pagination tokens if no `from` query parameter is given. ([\matrix-org#12370](matrix-org#12370))
- Add a module API for reading and writing global account data. ([\matrix-org#12391](matrix-org#12391))
- Support the stable `v1` endpoint for `/relations`, per [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\matrix-org#12403](matrix-org#12403))
- Include bundled aggregations in search results
  ([MSC3666](matrix-org/matrix-spec-proposals#3666)). ([\matrix-org#12436](matrix-org#12436))

Bugfixes
--------

- Fix a long-standing bug where updates to the server notices user profile (display name/avatar URL) in the configuration would not be applied to pre-existing rooms. Contributed by Jorge Florian. ([\matrix-org#12115](matrix-org#12115))
- Fix a long-standing bug where events from ignored users were still considered for bundled aggregations. ([\matrix-org#12235](matrix-org#12235), [\matrix-org#12338](matrix-org#12338))
- Fix non-member state events not resolving for historical events when used in [MSC2716](matrix-org/matrix-spec-proposals#2716) `/batch_send` `state_events_at_start`. ([\matrix-org#12329](matrix-org#12329))
- Fix a long-standing bug affecting URL previews that would generate a 500 response instead of a 403 if the previewed URL includes a port that isn't allowed by the relevant blacklist. ([\matrix-org#12333](matrix-org#12333))
- Default to `private` room visibility rather than `public` when a client does not specify one, according to spec. ([\matrix-org#12350](matrix-org#12350))
- Fix a spec compliance issue where requests to the `/publicRooms` federation API would specify `limit` as a string. ([\matrix-org#12364](matrix-org#12364), [\matrix-org#12410](matrix-org#12410))
- Fix a bug introduced in Synapse 1.49.0 which caused the `synapse_event_persisted_position` metric to have invalid values. ([\matrix-org#12390](matrix-org#12390))

Updates to the Docker image
---------------------------

- Bundle locked versions of dependencies into the Docker image. ([\matrix-org#12385](matrix-org#12385), [\matrix-org#12439](matrix-org#12439))
- Fix up healthcheck generation for workers docker image. ([\matrix-org#12405](matrix-org#12405))

Improved Documentation
----------------------

- Clarify documentation for running SyTest against Synapse, including use of Postgres and worker mode. ([\matrix-org#12271](matrix-org#12271))
- Document the behaviour of `LoggingTransaction.call_after` and `LoggingTransaction.call_on_exception` methods when transactions are retried. ([\matrix-org#12315](matrix-org#12315))
- Update dead links in `check-newsfragment.sh` to point to the correct documentation URL. ([\matrix-org#12331](matrix-org#12331))
- Upgrade the version of `mdbook` in CI to 0.4.17. ([\matrix-org#12339](matrix-org#12339))
- Updates to the Room DAG concepts development document to clarify that we mark events as outliers because we don't have any state for them. ([\matrix-org#12345](matrix-org#12345))
- Update the link to Redis pub/sub documentation in the workers documentation. ([\matrix-org#12369](matrix-org#12369))
- Remove documentation for converting a legacy structured logging configuration to the new format. ([\matrix-org#12392](matrix-org#12392))

Deprecations and Removals
-------------------------

- Remove the unused and unstable `/aggregations` endpoint which was removed from [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\matrix-org#12293](matrix-org#12293))

Internal Changes
----------------

- Remove lingering unstable references to MSC2403 (knocking). ([\matrix-org#12165](matrix-org#12165))
- Avoid trying to calculate the state at outlier events. ([\matrix-org#12191](matrix-org#12191), [\matrix-org#12316](matrix-org#12316), [\matrix-org#12330](matrix-org#12330), [\matrix-org#12332](matrix-org#12332), [\matrix-org#12409](matrix-org#12409))
- Omit sending "offline" presence updates to application services after they are initially configured. ([\matrix-org#12193](matrix-org#12193))
- Switch to using a sequence to generate AS transaction IDs. Contributed by Nick @ Beeper. If running synapse with a dedicated appservice worker, this MUST be stopped before upgrading the main process and database. ([\matrix-org#12209](matrix-org#12209))
- Add missing type hints for storage. ([\matrix-org#12267](matrix-org#12267))
- Add missing type definitions for scripts in docker folder. Contributed by Jorge Florian. ([\matrix-org#12280](matrix-org#12280))
- Move [MSC2654](matrix-org/matrix-spec-proposals#2654) support behind an experimental configuration flag. ([\matrix-org#12295](matrix-org#12295))
- Update docstrings to explain how to decipher live and historic pagination tokens. ([\matrix-org#12317](matrix-org#12317))
- Add ground work for speeding up device list updates for users in large numbers of rooms. ([\matrix-org#12321](matrix-org#12321))
- Fix typechecker problems exposed by signedjson 1.1.2. ([\matrix-org#12326](matrix-org#12326))
- Remove the `tox` packaging job: it will be redundant once matrix-org#11537 lands. ([\matrix-org#12334](matrix-org#12334))
- Ignore `.envrc` for `direnv` users. ([\matrix-org#12335](matrix-org#12335))
- Remove the (broadly unused, dev-only) dockerfile for pg tests. ([\matrix-org#12336](matrix-org#12336))
- Remove redundant `get_success` calls in test code. ([\matrix-org#12346](matrix-org#12346))
- Add type annotations for `tests/unittest.py`. ([\matrix-org#12347](matrix-org#12347))
- Move single-use methods out of `TestCase`. ([\matrix-org#12348](matrix-org#12348))
- Remove broken and unused development scripts. ([\matrix-org#12349](matrix-org#12349), [\matrix-org#12351](matrix-org#12351), [\matrix-org#12355](matrix-org#12355))
- Convert `Linearizer` tests from `inlineCallbacks` to async. ([\matrix-org#12353](matrix-org#12353))
- Update docstrings for `ReadWriteLock` tests. ([\matrix-org#12354](matrix-org#12354))
- Refactor `Linearizer`, convert methods to async and use an async context manager. ([\matrix-org#12357](matrix-org#12357))
- Fix a long-standing bug where `Linearizer`s could get stuck if a cancellation were to happen at the wrong time. ([\matrix-org#12358](matrix-org#12358))
- Make `StreamToken.from_string` and `RoomStreamToken.parse` propagate cancellations instead of replacing them with `SynapseError`s. ([\matrix-org#12366](matrix-org#12366))
- Add type hints to tests files. ([\matrix-org#12371](matrix-org#12371))
- Allow specifying the Postgres database's port when running unit tests with Postgres. ([\matrix-org#12376](matrix-org#12376))
- Remove temporary pin of signedjson<=1.1.1 that was added in Synapse 1.56.0. ([\matrix-org#12379](matrix-org#12379))
- Add opentracing spans to calls to external cache. ([\matrix-org#12380](matrix-org#12380))
- Lay groundwork for using `poetry` to manage Synapse's dependencies. ([\matrix-org#12381](matrix-org#12381), [\matrix-org#12407](matrix-org#12407), [\matrix-org#12412](matrix-org#12412), [\matrix-org#12418](matrix-org#12418))
- Make missing `importlib_metadata` dependency explicit. ([\matrix-org#12384](matrix-org#12384), [\matrix-org#12400](matrix-org#12400))
- Update type annotations for compatiblity with prometheus_client 0.14. ([\matrix-org#12389](matrix-org#12389))
- Remove support for the unstable identifiers specified in [MSC3288](matrix-org/matrix-spec-proposals#3288). ([\matrix-org#12398](matrix-org#12398))
- Add missing type hints to configuration classes. ([\matrix-org#12402](matrix-org#12402))
- Add files used to build the Docker image used for complement testing into the Synapse repository. ([\matrix-org#12404](matrix-org#12404))
- Do not include groups in the sync response when disabled. ([\matrix-org#12408](matrix-org#12408))
- Improve type hints related to HTTP query parameters. ([\matrix-org#12415](matrix-org#12415))
- Stop maintaining a list of lint targets. ([\matrix-org#12420](matrix-org#12420))
- Make `synapse._scripts` pass type checks. ([\matrix-org#12421](matrix-org#12421), [\matrix-org#12422](matrix-org#12422))
- Add some type hints to datastore. ([\matrix-org#12423](matrix-org#12423))
- Enable certificate checking during complement tests. ([\matrix-org#12435](matrix-org#12435))
- Explicitly specify the `tls` extra for Twisted dependency. ([\matrix-org#12444](matrix-org#12444))
@DMRobertson DMRobertson removed this from the Revisit: Next Month milestone May 5, 2022
@DMRobertson DMRobertson removed their assignment May 5, 2022
@DMRobertson
Copy link
Contributor

This partially made its way into Synapse 1.57 (which had a parallel pyproject.toml/poetry.lock and setup.py). Synapse 1.58 has removed the setup.py completely. I'm going to close this issue as done and file new ones for any final fixups.

See also the project https://github.com/orgs/matrix-org/projects/54.

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Jun 16, 2022
Change to wheel/poetry from egg.  Port remediation of upstream's
cryptography version demands to new build system.

Upstream no longer installs synmark.

Upstream NEWS, less bugfixes and minor updates:

Synapse 1.58.1 (2022-05-05)
===========================

[Debian packaging bugfix]


Synapse 1.58.0 (2022-05-03)
===========================

As of this release, the groups/communities feature in Synapse is now disabled by default. See [\#11584](matrix-org/synapse#11584) for details. As mentioned in [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1580), this feature will be removed in Synapse 1.61.

Synapse 1.58.0rc1 (2022-04-26)
==============================

Features
--------

- Implement [MSC3383](matrix-org/matrix-spec-proposals#3383) for including the destination in server-to-server authentication headers. Contributed by @Bubu and @jcgruenhage for Famedly. ([\#11398](matrix-org/synapse#11398))
- Docker images and Debian packages from matrix.org now contain a locked set of Python dependencies, greatly improving build reproducibility. ([Board](https://github.com/orgs/matrix-org/projects/54), [\#11537](matrix-org/synapse#11537))
- Enable processing of device list updates asynchronously. ([\#12365](matrix-org/synapse#12365), [\#12465](matrix-org/synapse#12465))
- Implement [MSC2815](matrix-org/matrix-spec-proposals#2815) to allow room moderators to view redacted event content. Contributed by @tulir @ Beeper. ([\#12427](matrix-org/synapse#12427))
- Build Debian packages for Ubuntu 22.04 "Jammy Jellyfish". ([\#12543](matrix-org/synapse#12543))


Improved Documentation
----------------------

- Strongly recommend [Poetry](https://python-poetry.org/) for development. ([\#12475](matrix-org/synapse#12475))


Deprecations and Removals
-------------------------

- The groups/communities feature in Synapse is now disabled by default. ([\#12344](matrix-org/synapse#12344))
- Remove unstable identifiers from [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#12382](matrix-org/synapse#12382))
@kittykat kittykat added the z-p2 (Deprecated Label) label Sep 6, 2022
@qknight
Copy link

qknight commented Sep 20, 2022

Just tried to build v1.61.0 on Linux and Windows and both fail.

I know this worked a few weeks ago and I basically think that OP was right with the version pinning.

Motivation: I want to bisect and thus build v1.61.0 .. v1.62.0 for identifying a patch which destroyed the generate call for us (I'll open a new ticket for that one).

Here is the error:

git checkout tags/v1.61.0
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile -t fred .
[+] Building 76.9s (14/25)
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.1s
 => => transferring dockerfile: 5.28kB                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                       0.1s
 => => transferring context: 170B                                                                                                                                                                                                       0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                                                              1.3s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                                                                        0.0s
 => docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                1.0s
 => => resolve docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                    0.0s
 => => sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 2.00kB / 2.00kB                                                                                                                                          0.0s
 => => sha256:ad87fb03593d1b71f9a1cfc1406c4aafcb253b1dabebf569768d6e6166836f34 528B / 528B                                                                                                                                              0.0s
 => => sha256:1e8a16826fd1c80a63fa6817a9c7284c94e40cded14a9b0d0d3722356efa47bd 2.37kB / 2.37kB                                                                                                                                          0.0s
 => => sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 9.94MB / 9.94MB                                                                                                                                          0.7s
 => => extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546                                                                                                                                               0.2s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                                                                       0.3s
 => => transferring context: 7.41MB                                                                                                                                                                                                     0.2s
 => [stage-2 1/5] FROM docker.io/library/python:3.9-slim                                                                                                                                                                                0.2s
 => [builder 2/7] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update && apt-get install -y     build-essential     libffi-dev     libjpeg-dev     51.4s
 => [stage-2 2/5] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked   apt-get update && apt-get install -y     curl     gosu     libjpeg62-turbo     libpq5      70.1s
 => [requirements 2/6] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update && apt-get install -y git     && rm -rf /var/lib/apt/lists/*            60.6s
 => ERROR [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"                     13.0s
------
 > [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5":
#13 3.856 Collecting git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 3.858   Cloning https://github.com/python-poetry/poetry.git (to revision fb13b3a676f476177f7937ffa480ee5cff9a90a5) to /tmp/pip-req-build-tufn8q4p
#13 3.866   Running command git clone --filter=blob:none --quiet https://github.com/python-poetry/poetry.git /tmp/pip-req-build-tufn8q4p
#13 7.242   Running command git rev-parse -q --verify 'sha^fb13b3a676f476177f7937ffa480ee5cff9a90a5'
#13 7.250   Running command git fetch -q https://github.com/python-poetry/poetry.git fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 7.839   Running command git checkout -q fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 9.253   Resolved https://github.com/python-poetry/poetry.git to commit fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 9.720   Installing build dependencies: started
#13 12.36   Installing build dependencies: finished with status 'done'
#13 12.37   Getting requirements to build wheel: started
#13 12.46   Getting requirements to build wheel: finished with status 'done'
#13 12.46   Preparing metadata (pyproject.toml): started
#13 12.77   Preparing metadata (pyproject.toml): finished with status 'error'
#13 12.77   error: subprocess-exited-with-error
#13 12.77
#13 12.77   × Preparing metadata (pyproject.toml) did not run successfully.
#13 12.77   │ exit code: 1
#13 12.77   ╰─> [26 lines of output]
#13 12.77       Traceback (most recent call last):
#13 12.77         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
#13 12.77           main()
#13 12.77         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
#13 12.77           json_out['return_val'] = hook(**hook_input['kwargs'])
#13 12.77         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
#13 12.77           return hook(metadata_directory, config_settings)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/masonry/api.py", line 40, in prepare_metadata_for_build_wheel
#13 12.77           poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 62, in create_poetry
#13 12.77           package = self.configure_package(
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 156, in configure_package
#13 12.77           cls._add_package_group_dependencies(
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 102, in _add_package_group_dependencies
#13 12.77           cls.create_dependency(
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 366, in create_dependency
#13 12.77           dependency = Dependency(name, constraint, groups=groups)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/packages/dependency.py", line 66, in __init__
#13 12.77           self.constraint = constraint  # type: ignore[assignment]
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/packages/dependency.py", line 110, in constraint
#13 12.77           self._constraint = parse_constraint(constraint)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/semver/helpers.py", line 31, in parse_constraint
#13 12.77           constraint_objects.append(parse_single_constraint(constraint))
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/semver/helpers.py", line 147, in parse_single_constraint
#13 12.77           raise ParseConstraintError(f"Could not parse version constraint: {constraint}")
#13 12.77       poetry.core.semver.exceptions.ParseConstraintError: Could not parse version constraint: (>=20.4.3
#13 12.77       [end of output]
#13 12.77
#13 12.77   note: This error originates from a subprocess, and is likely not a problem with pip.
#13 12.78 error: metadata-generation-failed
#13 12.78
#13 12.78 × Encountered error while generating package metadata.
#13 12.78 ╰─> See above for output.
#13 12.78
#13 12.78 note: This is an issue with the package mentioned above, not pip.
#13 12.78 hint: See above for details.
#13 12.79 WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
#13 12.79 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"]: exit code: 1

@DMRobertson
Copy link
Contributor

qknight's issue was pulled out to #13849.


For completeness: there was some discussion about how to get tox and poetry to interact, starting roughly #11537 (comment)

I note that Poetry's docs (now?) has a section on tox configurations. TL;DR: if you want to get tox to install locked versions, you have to get it to invoke poetry.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. z-p2 (Deprecated Label)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants