ImportError after upgrade to Synapse 1.55.{1,2}

[richvdh]

I upgraded Synapse to 1.55.1, and got this error:

Mar 24 18:04:24 river synapse[16993]: Traceback (most recent call last):
Mar 24 18:04:24 river synapse[16993]:   File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
Mar 24 18:04:24 river synapse[16993]:     "__main__", mod_spec)
Mar 24 18:04:24 river synapse[16993]:   File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
Mar 24 18:04:24 river synapse[16993]:     exec(code, run_globals)
Mar 24 18:04:24 river synapse[16993]:   File "/home/synapse/src/synapse/app/homeserver.py", line 28, in <module>
Mar 24 18:04:24 river synapse[16993]:     import synapse.config.logger
Mar 24 18:04:24 river synapse[16993]:   File "/home/synapse/src/synapse/config/__init__.py", line 15, in <module>
Mar 24 18:04:24 river synapse[16993]:     from ._base import ConfigError, find_config_files
Mar 24 18:04:24 river synapse[16993]:   File "/home/synapse/src/synapse/config/_base.py", line 37, in <module>
Mar 24 18:04:24 river synapse[16993]:     import jinja2
Mar 24 18:04:24 river synapse[16993]:   File "/home/synapse/env/lib/python3.7/site-packages/jinja2/__init__.py", line 12, in <module>
Mar 24 18:04:24 river synapse[16993]:     from .environment import Environment
Mar 24 18:04:24 river synapse[16993]:   File "/home/synapse/env/lib/python3.7/site-packages/jinja2/environment.py", line 25, in <module>
Mar 24 18:04:24 river synapse[16993]:     from .defaults import BLOCK_END_STRING
Mar 24 18:04:24 river synapse[16993]:   File "/home/synapse/env/lib/python3.7/site-packages/jinja2/defaults.py", line 3, in <module>
Mar 24 18:04:24 river synapse[16993]:     from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
Mar 24 18:04:24 river synapse[16993]:   File "/home/synapse/env/lib/python3.7/site-packages/jinja2/filters.py", line 13, in <module>
Mar 24 18:04:24 river synapse[16993]:     from markupsafe import soft_unicode
Mar 24 18:04:24 river synapse[16993]: ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/home/synapse/env/lib/python3.7/site-packages/markupsafe/__init__.py)

[richvdh]

This turns out to be due to the new dependency on Markupsafe. The current Markupsafe is incompatible with Jinja 2.x.

The workaround was to downgrade Markupsafe to 2.0.1.

[babolivier]

Related #12025

[richvdh]

More details:

#12289 added a dependency on Markupsafe. I previously had no Markupsafe installed, hence, when I ran pip install, I got the current version of Markupsafe (2.1.1).

Jinja2, prior to v3.0.0, includes the line:

from markupsafe import soft_unicode

If Markupsafe is not installed, this uses the internal jinja2.markupsafe module, which works fine; but once Markupsafe is installed it picks that up instead. Now, that is fine for Markupsafe v2.0.x, but Markupsafe v2.1.0 removed markupsafe.soft_unicode.

So, that gives us a compatibility matrix:

jinja2 version	Markupsafe version	result
2.x	none	ok
2.x	2.0.x	ok
2.x	2.1.x	ImportError
3.x	none	n/a (Markupsafe is pulled in transitively)
3.x	2.0.x	ok
3.x	2.1.x	ok

[happenpappen]

Upgrading with "pip install --upgrade matrix-synapse" to version 1.55.2 in a venv ended with the ImportError staying the same as before. I had to downgrade MarkupSafe manually by issuing "pip install 'MarkupSafe<2.1.0'" which solved the issue.

[richvdh]

evidently this is still happening in 1.55.2. I don't really understand why.

If anyone who sees this could share the output from pip during the upgrade, it might help us track down the problem.

[richvdh]

That said: If you previously upgraded to 1.55.1, a subsequent upgrade to 1.55.2 won't fix the problems introduced by 1.55.1, and a manual downgrade of Markupsafe is to be expected.

The part that surprises me is that this affects people who upgrade directly from 1.55.0 (or earlier) to 1.55.2.

[DoM1niC]

Try make a virtualenv with Python 3.9, in my case no issues here....
you can try my simple Upgrade Script 👍🏻 The LDAP Stuff is optional

MATRIX_PATH=/opt/matrix
cd $MATRIX_PATH
su matrix -c rm -rf bin/ lib/; virtualenv -p /usr/bin/python3 .
su matrix -c "
source $MATRIX_PATH/bin/activate; \
pip install --upgrade setuptools; \
pip install --upgrade matrix-synapse==$2; \
pip install --upgrade https://github.com/matrix-org/matrix-synapse-ldap3/archive/master.zip; \
pip install --upgrade psycopg2-binary; \
pip install --upgrade hiredis; \
pip install --upgrade 'lxml>=3.5.0'; \
pip install --upgrade txredisapi; \
pip install --upgrade treq; \
pip install --upgrade twisted"

[richvdh]

I think what we ultimately need to do here is reintroduce #12289 for compatibility with Jinja 3.1, but also to pin our Jinja2 dependency to ~=3.0, for compatibility with Markupsafe 2.1. That latter step may be inconvenient for downstream packagers seeking to package synapse for distributions which do not yet have Jinja 3.x, but they can patch out the dependency, and it's probably the least bad compromise.

Soon, our work on #11537 will put an end to this madness.

[DMRobertson]

Soon, our work on #11537 will put an end to this madness.

...for people using our debs, our docker images, or installing with poetry from source.

[richvdh]

...for people using our debs, our docker images, or installing with poetry from source.

well, the madness here is solely to support people installing from source (currently via pip, soon via poetry). Everyone else is fine.

(The task of building our debs and docker images is just a special case of installing from source.)