“lxml & xmlsec libxml2 library version mismatch” error under uWSGI

[andersk]

If libxml2-dev was installed when uWSGI was built, then importing xmlsec within uWSGI leads to an incorrect error xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch'). There’s no such error outside of uWSGI.

(I can work around this error with --no-binary=lxml --no-binary=xmlsec, but that wastes a lot more CI time and shouldn’t be necessary.)

Reproduction in a fresh container:

$ docker run --rm -it ubuntu:22.04
root@076333186bac:/# apt update
…
root@076333186bac:/# apt install -y gcc libxml2-dev python3-dev python3-venv
…
root@076333186bac:/# python3 -m venv venv
root@076333186bac:/# . venv/bin/activate
(venv) root@076333186bac:/# pip install uWSGI xmlsec
…
Successfully installed lxml-5.2.1 uWSGI-2.0.25.1 xmlsec-1.3.14
(venv) root@076333186bac:/# cat > app.py <<EOF
import xmlsec

def application(env, start_response):
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [b"Hello, world!\n"]

EOF
(venv) root@076333186bac:/# python -c 'import xmlsec'
(venv) root@076333186bac:/# uwsgi --http :9090 --wsgi-file app.py
*** Starting uWSGI 2.0.25.1 (64bit) on [Mon May  6 22:18:16 2024] ***
compiled with version: 11.4.0 on 06 May 2024 22:17:53
os: Linux-6.8.7 #1-NixOS SMP PREEMPT_DYNAMIC Wed Apr 17 09:23:43 UTC 2024
nodename: 076333186bac
machine: x86_64
clock source: unix
detected number of CPU cores: 12
current working directory: /
detected binary path: /venv/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :9090 fd 4
spawned uWSGI http 1 (pid: 4203)
uwsgi socket 0 bound to TCP address 127.0.0.1:46691 (port auto-assigned) fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x55a23eea0bc0
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72904 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
Traceback (most recent call last):
  File "app.py", line 1, in <module>
    import xmlsec
xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 4202, cores: 1)

[andersk]

Some debugging information:

>>> import ctypes, lxml, xmlsec
>>> libxml2 = ctypes.CDLL("libxml2.so.2")
>>> ctypes.c_char_p.in_dll(libxml2, "xmlParserVersion").value
b'20913'
>>> lxml.etree.LIBXML_VERSION
(2, 12, 6)
>>> lxml.etree.LIBXML_COMPILED_VERSION
(2, 12, 6)
>>> xmlsec.get_libxml_version()
(2, 12, 6)
>>> xmlsec.get_libxml_compiled_version()
(2, 12, 6)

xmlParserVersion, LIBXML_VERSION, and LIBXML_COMPILED_VERSION are the same inside and outside uWSGI.

[eliasmazur]

I'm getting this same error on a container that I just re-compiled and it was working fine. I use python3-saml and that's breaking with this error when I run the container.

Here is the log:

[2024-05-20 02:54:20 +0000] [11] [ERROR] Exception in worker process
2024-05-19 22:54:20 Traceback (most recent call last):
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2024-05-19 22:54:20     worker.init_process()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/ggevent.py", line 203, in init_process
2024-05-19 22:54:20     super(GeventWorker, self).init_process()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base.py", line 129, in init_process
2024-05-19 22:54:20     self.load_wsgi()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2024-05-19 22:54:20     self.wsgi = self.app.wsgi()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/base.py", line 67, in wsgi
2024-05-19 22:54:20     self.callable = self.load()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
2024-05-19 22:54:20     return self.load_wsgiapp()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2024-05-19 22:54:20     return util.import_app(self.app_uri)
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/util.py", line 350, in import_app
2024-05-19 22:54:20     __import__(module)
2024-05-19 22:54:20   File "/opt/app/tinyflask/__init__.py", line 11, in <module>
2024-05-19 22:54:20     import tinyflask.views
2024-05-19 22:54:20   File "/opt/app/tinyflask/views.py", line 24, in <module>
2024-05-19 22:54:20     from onelogin.saml2.auth import OneLogin_Saml2_Auth
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/onelogin/saml2/auth.py", line 12, in <module>
2024-05-19 22:54:20     import xmlsec
2024-05-19 22:54:20 xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')
2024-05-19 22:54:20 [2024-05-20 02:54:20 +0000] [11] [INFO] Worker exiting (pid: 11)

Here is my Dockerfile:

FROM ubuntu:bionic

RUN apt update
RUN apt-get update
RUN apt-get -y upgrade
RUN apt-get install -y pkg-config
RUN apt-get install -y software-properties-common apt-utils
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt-get install -y vim
RUN apt-get install -y python3.7
RUN apt-get install -y python3-pip

RUN apt-get install -y libxml2-dev libxmlsec1-dev

RUN mkdir -p /opt/app
WORKDIR /opt/app/
COPY . .
RUN pip3 install --upgrade pip
RUN pip3 install --no-cache-dir -r requirements.txt

RUN pip3 install python3-saml

RUN touch /var/log/cron.log

RUN echo "35 6 * * * root /usr/bin/python3 /opt/app/last_login.py >> /opt/app/audit.log" >> /etc/crontab

CMD (cron) && exec gunicorn tinyflask:app -b 0.0.0.0:80 --workers 3 -k gevent

And this is the requirements.txt:

Click==7.0
Flask==1.1.1
Flask-Cors==3.0.8
gevent==1.4.0
greenlet==0.4.15
gunicorn==19.9.0
itsdangerous==1.1.0
Jinja2==2.10.1
MarkupSafe==1.1.1
Werkzeug==0.15.5
flask-compress==1.4.0
requests==2.24.0
python-dateutil==2.8.0
pytz==2021.1
pickleshare==0.7.5
xmltodict

I'm stuck. Any help is appreciated.

Thanks

[andres-holvi]

It seems like [email protected] uses [email protected]. However, [email protected] has been released, and [email protected] breaks because of this libxml2 version mismatch (?)

Not sure who should be in charge of fixing this, but I suppose using [email protected] could work.

[gregbrant2]

@eliasmazur I have the same use case as you with python3-saml. Based on what @andersk said, I've restricted the package version of LXML like so

lxml >= 4.6.5, !=4.7.0, <=5.2.1 

My build is working again now.

[eliasmazur]

Thanks for the help. I realized I was restricting all packages except xmlsec. I restricted to an earlier version and it now works. Looks like python3-saml was installing the latest xmlsec and that was causing the mismatch.

[mstuttgart]

In my work, we use sigxml and xmlsec. Both install lxml with binaries, causing the same error.

xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')

To resolve this, I install using the --no-binary pip flag. (work on Ubuntu 22.04, Python 3.10 and Pip 22.0.2)

pip install --no-binary lxml==4.6.3 lxml==4.6.3 --force-reinstall
pip install --no-binary xmlsec==1.3.13 xmlsec==1.3.13
pip install --no-binary signxml==3.2.2 signxml==3.2.2

[buddhi-vamsi13]

Thanks for the help. I realized I was restricting all packages except xmlsec. I restricted to an earlier version and it now works. Looks like python3-saml was installing the latest xmlsec and that was causing the mismatch.

Hi,

we have the similiar issue can you provide us the version which u have mentioned

[eliasmazur]

For the libxml in Dockerfile:
RUN apt-get install -y libxml2-dev=2.9.4+dfsg1-6.1ubuntu1.9 libxmlsec1-dev=1.2.25-1build1

For the lxml and xmlsec in requirements.txt:
lxml==4.9.3
xmlsec==1.3.13

Hope this helps.

[sergei-maertens]

Would this mean that a new release/wheel is needed also built with [email protected]? I was very excited about being able to install binary wheels again with xmlsec 1.3.14, but it seems that a patch release of libxml broke that again?

[sergei-maertens]

@eliasmazur I have the same use case as you with python3-saml. Based on what @andersk said, I've restricted the package version of LXML like so

lxml >= 4.6.5, !=4.7.0, <=5.2.1 

My build is working again now.

Trying this, the problem is not solved for us:

defusedxml==0.7.1
et-xmlfile==1.1.0
lxml==5.2.1
lxml_html_clean==0.1.1
xmlsec==1.3.14
xmltodict==0.12.0

still produces the uwsgi startup crash, while a normal python interpreter is fine (just as reported in the initial description). I have tried uwsgi 2.0.23 and 2.0.26 so far.

Making sure that libxml2-dev is NOT present in the system packages when uwsgi is being installed/built resolves the issue.

[D3X]

I came up with another workaround - compile uwsgi against libexpat instead of libxml2. You can do that by setting the UWSGI_PROFILE_OVERRIDE environment variable to xml=expat (or adding that part to it if you already have it set to something) for whatever command you use to install uwsgi, for example:

UWSGI_PROFILE_OVERRIDE=xml=expat pip install uwsgi

or

UWSGI_PROFILE_OVERRIDE="ssl=false;xml=expat" pip install uwsgi

Caveat: libexpat is supposedly slower than libxml2 (based on some old benchmarks found around the Internet), so if you're a heavy user of uWSGI features that depend on an XML parsing library, this may affect performance of your application. I am not sure what these features are, though.

Once your uwsgi is decoupled from libxml2, you should be able to use binary wheels for xmlsec and lxml. You should also be able to install the older version of xmlsec that didn't provide binary wheels, as long as you still have libxmlsec1-dev / libxml2-dev installed.

[jarshwah]

Thanks @D3X - that was a really good steer. IF you don't need XML support at all (it looks like it's only for supporting xml configuration of uwsgi, and... who does that??), you can:

UWSGI_PROFILE_OVERRIDE="xml=no" and it'll strip xml support altogether. From the uwsgi build system: https://github.com/unbit/uwsgi/blob/89cb161cda959697a4afe013f348b06646b960aa/uwsgiconfig.py#L1328-L1363

This is the route I've just taken, and everything seems fine to me.