Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QGIS 3.28.1 + gdal<3.6 causes error loading shared libraries #276

Closed
1 task done
danschef opened this issue Dec 1, 2022 · 15 comments · Fixed by #277 · May be fixed by conda-forge/conda-forge-repodata-patches-feedstock#372
Closed
1 task done
Labels

Comments

@danschef
Copy link

danschef commented Dec 1, 2022

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

I just tried to install to install the latest qgis 3.28.1 Linux build together with enpt. Somehow enpt forces to install gdal 3.5.3 instead of gdal 3.6, which does not seem to work. When I start QGIS, I only get:

qgis: error while loading shared libraries: libgdal.so.32: cannot open shared object file: No such file or directory

Not sure if this is related to qgis but I had no problem with with qgis 3.22 + enpt.

When I create an environment with mamba create -n myenv "qgis=3.28.1", gdal 3.6 is installed and everything works fine. However, when I downgrade gdal to 3.5.3, it fails to load libgdal.so.32 (only libgdal.so.31 is present).

-> Reproduce it with mamba create -n myenv "qgis=3.28.1" "gdal<3.6", see output of conda list below.

Any idea what goes wrong here?

Installed packages

(myenv) me@geoms:~$ conda list
# packages in environment at /home/me/mambaforge/envs/myenv:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
alsa-lib                  1.2.8                h166bdaf_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
blosc                     1.21.1               h83bc5f7_3    conda-forge
boost-cpp                 1.78.0               h75c5d50_1    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.9.24            ha878542_0    conda-forge
cairo                     1.16.0            ha61ee94_1014    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_2    conda-forge
cfitsio                   4.1.0                hd9d235c_0    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
coverage                  6.5.0           py310h5764c6d_1    conda-forge
cryptography              38.0.4          py310h597c629_0    conda-forge
curl                      7.86.0               h7bff187_1    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
draco                     1.5.5                h924138e_0    conda-forge
eigen                     3.4.0                h4bd325d_0    conda-forge
exiv2                     0.27.5               h848de5d_0    conda-forge
expat                     2.5.0                h27087fc_0    conda-forge
fftw                      3.3.10          nompi_hf0379b8_105    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.1               hc2a2eb6_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
freexl                    1.0.6                h166bdaf_1    conda-forge
future                    0.18.2             pyhd8ed1ab_6    conda-forge
gdal                      3.5.3           py310hb7951cf_1    conda-forge
geos                      3.11.0               h27087fc_0    conda-forge
geotiff                   1.7.1                ha76d385_4    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
glib                      2.74.1               h6239696_1    conda-forge
glib-tools                2.74.1               h6239696_1    conda-forge
gsl                       2.7                  he838d99_0    conda-forge
gst-plugins-base          1.21.2               h3e40eee_0    conda-forge
gstreamer                 1.21.2               hd4edc92_0    conda-forge
gstreamer-orc             0.4.33               h166bdaf_0    conda-forge
hdf4                      4.2.15               h9772cbc_5    conda-forge
hdf5                      1.12.2          nompi_h2386368_100    conda-forge
httplib2                  0.21.0             pyhd8ed1ab_0    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
jack                      1.9.21               he978b8e_1    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
json-c                    0.16                 hc379101_0    conda-forge
kealib                    1.4.15               ha7026e8_1    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
laz-perf                  3.0.0                h4bd325d_0    conda-forge
lcms2                     2.14                 h6ed2654_0    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libcap                    2.66                 ha37c62d_0    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libclang                  15.0.6          default_h2e3cab8_0    conda-forge
libclang13                15.0.6          default_h3a83d3e_0    conda-forge
libcups                   2.3.3                h3e49a29_2    conda-forge
libcurl                   7.86.0               h7bff187_1    conda-forge
libdap4                   3.20.6               hd7c4107_2    conda-forge
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.14                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.2                h27087fc_0    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgcrypt                 1.10.1               h166bdaf_0    conda-forge
libgdal                   3.5.3                h275edd5_1    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libglib                   2.74.1               h606061b_1    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libgpg-error              1.45                 hc0c96e0_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libkml                    1.3.0             h37653c0_1015    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libllvm15                 15.0.6               h63197d8_0    conda-forge
libnetcdf                 4.8.1           nompi_h261ec11_106    conda-forge
libnghttp2                1.47.0               hdcd2b5c_1    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     14.5                 hd77ab85_1    conda-forge
libprotobuf               3.21.10              h6239696_0    conda-forge
librttopo                 1.1.0               hf730bdb_11    conda-forge
libsecret                 0.18.8               h329b89f_2    conda-forge
libsndfile                1.1.0                h27087fc_0    conda-forge
libspatialindex           1.9.3                h9c3ff4c_4    conda-forge
libspatialite             5.0.1               hfbd986c_21    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libsystemd0               252                  h2a991cd_0    conda-forge
libtiff                   4.4.0                h55922b4_4    conda-forge
libtool                   2.4.6             h9c3ff4c_1008    conda-forge
libudev1                  252                  h166bdaf_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp                   1.2.4                h522a892_0    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libxslt                   1.1.37               h873f0b0_0    conda-forge
libzip                    1.9.2                hc869a4a_1    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
markupsafe                2.1.1           py310h5764c6d_2    conda-forge
mock                      4.0.3              pyhd8ed1ab_4    conda-forge
mpg123                    1.30.2               h27087fc_1    conda-forge
mysql-common              8.0.31               haf5c9bc_0    conda-forge
mysql-libs                8.0.31               h28c427c_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
nitro                     2.7.dev6             h27087fc_5    conda-forge
nose2                     0.9.2                      py_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.82                 he02c5a1_0    conda-forge
numpy                     1.23.5          py310h53a5b5f_0    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   1.1.1s               h166bdaf_0    conda-forge
owslib                    0.27.2             pyhd8ed1ab_1    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pdal                      2.4.3                hbecf0f4_2    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
plotly                    5.11.0             pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
poppler                   22.11.0              h92391eb_0    conda-forge
poppler-data              0.4.11               hd8ed1ab_0    conda-forge
postgresql                14.5                 hdeef612_1    conda-forge
proj                      9.1.0                h93bde94_0    conda-forge
psycopg2                  2.9.3           py310h5764c6d_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pulseaudio                16.1                 h4a94279_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.13.0             pyhd8ed1ab_0    conda-forge
pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyproj                    3.4.0           py310hb1338dc_2    conda-forge
pyqt                      5.15.7          py310h29803b5_2    conda-forge
pyqt5-sip                 12.11.0         py310hd8f1fbe_2    conda-forge
pyqtwebkit                5.15.7          py310h29803b5_3    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.8          h257c98d_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytz                      2022.6             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py310h5764c6d_5    conda-forge
qca                       2.3.5                h51e5271_0    conda-forge
qgis                      3.28.1          py310h3c18fa3_0    conda-forge
qjson                     0.9.0             h51e5271_1007    conda-forge
qscintilla2               2.13.3          py310h29803b5_4    conda-forge
qt-main                   5.15.6               h7acdfc8_2    conda-forge
qtkeychain                0.13.2               h1601faa_0    conda-forge
qtwebkit                  5.212                h3383a02_6    conda-forge
qwt                       6.2.0                h1d9fb53_4    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
setuptools                65.5.1             pyhd8ed1ab_0    conda-forge
sip                       6.7.5           py310hd8f1fbe_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.9                hbd366e4_2    conda-forge
sqlite                    3.40.0               h4ff8645_0    conda-forge
tenacity                  8.1.0              pyhd8ed1ab_0    conda-forge
tiledb                    2.11.3               h1e4a385_1    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzcode                    2022g                h166bdaf_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
urllib3                   1.26.13            pyhd8ed1ab_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xcb-util                  0.4.0                h516909a_0    conda-forge
xcb-util-image            0.4.0                h166bdaf_0    conda-forge
xcb-util-keysyms          0.4.0                h516909a_0    conda-forge
xcb-util-renderutil       0.3.9                h166bdaf_0    conda-forge
xcb-util-wm               0.4.1                h516909a_0    conda-forge
xerces-c                  3.2.4                h55805fa_1    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge

Environment info

(myenv) gfz-fe@geoms:~$ conda info

     active environment : myenv
    active env location : /home/me/mambaforge/envs/myenv
            shell level : 5
       user config file : /home/me/.condarc
 populated config files : /home/me/mambaforge/.condarc
                          /home/me/.condarc
          conda version : 22.9.0
    conda-build version : not installed
         python version : 3.10.8.final.0
       virtual packages : __linux=5.4.0=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/me/mambaforge  (writable)
      conda av data dir : /home/me/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : file:///home/me/.conda_channels_local/pygfz/linux-64
                          file:///home/me/.conda_channels_local/pygfz/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/me/mambaforge/pkgs
                          /home/me/.conda/pkgs
       envs directories : /home/me/mambaforge/envs
                          /home/me/.conda/envs
               platform : linux-64
             user-agent : conda/22.9.0 requests/2.28.1 CPython/3.10.8 Linux/5.4.0-52-generic ubuntu/20.04.5 glibc/2.31
                UID:GID : 1009:38755
             netrc file : None
           offline mode : False
@danschef danschef added the bug label Dec 1, 2022
@gillins
Copy link
Contributor

gillins commented Dec 1, 2022

QGIS has done the GDAL 3.6.0 migration so anything that downgrades GDAL to an earlier version will break the environment... Can enpt be updated (probably needs a re-render so all its dependencies are the latest versions)?

@gillins
Copy link
Contributor

gillins commented Dec 2, 2022

Actually @ocefpaf is this an error with the qgis recipe? Could be be caused by depending on gdal rather than libgdal? I think this is why my LTR tests are also failing...

@gillins gillins mentioned this issue Dec 2, 2022
4 tasks
@danschef
Copy link
Author

danschef commented Dec 2, 2022

It is fiona that downgrades gdal to <3.6. They are currently waiting for a new release compatible with gdal 3.6: conda-forge/fiona-feedstock#202. However, an incomptibility of qgis 3.28.1 and gdal<3.6 should be indicated somehow in the recipe so that conda does not install both together.

I built the Linux Python 3.9 version from #277 locally and can confirm that this fixes the issue as it refuses to install gdal<3.6. Instead, conda jumps to build 0 if I try to install gdal<3.6:

  Package    Version  Build           Channel                    Size
───────────────────────────────────────────────────────────────────────
  Change:
───────────────────────────────────────────────────────────────────────

  - pdal       2.4.3  hbf2fe72_3      conda-forge
  + pdal       2.4.3  hbecf0f4_2      conda-forge/linux-64     Cached
  - qgis      3.28.1  py39h5e55f13_1  danschef
  + qgis      3.28.1  py39ha366a5a_0  conda-forge/linux-64     Cached

  Downgrade:
───────────────────────────────────────────────────────────────────────

  - gdal       3.6.0  py39h92c1d47_1  conda-forge
  + gdal       3.5.3  py39h92c1d47_1  conda-forge/linux-64     Cached
  - libgdal    3.6.0  h275edd5_1      conda-forge
  + libgdal    3.5.3  h275edd5_1      conda-forge/linux-64     Cached

  Summary:

  Change: 2 packages
  Downgrade: 2 packages

  Total download: 0 B

───────────────────────────────────────────────────────────────────────

@gillins
Copy link
Contributor

gillins commented Dec 2, 2022

I guess we can get build 0 marked as broken?

@SrNetoChan
Copy link
Contributor

I don't think this is a QGIS problem. should the downgrade push the qgis version to 3.22 too?

@danschef
Copy link
Author

danschef commented Dec 2, 2022

So far as I understand, this is the latest version compatible with gdal<3.6. Not marking build 0 as broken directly leads to the above issue when installing any package that is not yet compatible with gdal 3.6.

@SrNetoChan
Copy link
Contributor

How about merging this #277 see if it corrects the problem and then mark build 0 as broken?

@SrNetoChan
Copy link
Contributor

Because you mention fiona I have tried installing geopandas together with QGIS 3.28.1 and I got this:

The following packages will be DOWNGRADED:

gdal 3.6.0-py311hf6158c2_1 --> 3.5.3-py311hf6158c2_1
libgdal 3.6.0-h275edd5_1 --> 3.5.3-h275edd5_1
pdal 2.4.3-hbf2fe72_3 --> 2.4.3-hbecf0f4_2

Which seems similar problem.

@danschef
Copy link
Author

danschef commented Dec 2, 2022

Yes, because geopandas depends on fiona: https://github.com/conda-forge/geopandas-feedstock/blob/a45dcce2f2798f912e16aaf7ed4d363a35cd3059/recipe/meta.yaml#L48

I already tested if #277 fixes the issue: #276 (comment)
The problem is just that conda then jumps back to qgis 3.28.1 build 0, which breaks with gdal<3.6.

I am building the Linux versions for #277.

@danschef
Copy link
Author

danschef commented Dec 2, 2022

Maybe a stupid question: Providing compatibility of qgis 3.28.1 to gdal>=3.5.3 is not realistic, right?

@SrNetoChan
Copy link
Contributor

This was one of the reasons why I wanted to keep a LTR build with the main dependencies pinned (gdal, geos, proj). Make as stable and retro compatible with other libraries.

@gillins
Copy link
Contributor

gillins commented Dec 3, 2022

Providing compatibility of qgis 3.28.1 to gdal>=3.5.3 is not realistic, right?

It's a bit tricky since everything is always moving forward and we have already done the GDAL 3.6 migration. Is it possible to have fiona etc in a different environment?
@SrNetoChan, do you want to see if you can pin the LTR to GDAL 3.5.*?

@SrNetoChan
Copy link
Contributor

Yes, I can try that. But I didn't have 5he time yet.

@weiji14
Copy link
Member

weiji14 commented Dec 6, 2022

The problem is just that conda then jumps back to qgis 3.28.1 build 0, which breaks with gdal<3.6.

Instead of marking the build as broken, there's a way to submit a repodata patch to fix the metadata (i.e. set an explicit pin to GDAL 3.6), see example PR at conda-forge/conda-forge-repodata-patches-feedstock#366. Instructions at https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/tree/main/recipe.

Oh and by the way, the error while loading shared libraries: libgdal.so.32 issue is the same as GenericMappingTools/pygmt#2215. We've fixed it in the gmt conda-forge package (after many attempts) at conda-forge/gmt-feedstock#232 by using - {{ pin_compatible('gdal', max_pin='x.x') }} in the recipe/meta.yaml file.

@gillins
Copy link
Contributor

gillins commented Dec 8, 2022

Thanks @weiji14 !

@danschef any chance you have some time to submit a repodata patch for this? I'm completely out of bandwidth ATM...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants