Installation failure in University of Leipzig cluster #3589

jumin94 · 2024-05-10T14:59:39Z

I am installing esmvaltool using mamba at the Leipzig Institute for Meteorology and once I complete all the steps and apparently finishing a successful instalation I activate the enviroment and try "esmvaltool -h" but get the message below. Does anyone know what this is related to or what I can do to solve it?

(esmvaltool) jmindlin@barat:~$ esmvaltool -h
ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/cf_units/init.py", line 186, in
_ud_system = _ud.read_xml()
^^^^^^^^^^^^^^
File "cf_units/_udunits2.pyx", line 194, in cf_units._udunits2.read_xml
File "cf_units/_udunits2.pyx", line 104, in cf_units._udunits2.wrap_system
File "cf_units/_udunits2.pyx", line 184, in cf_units._udunits2._raise_error
cf_units._udunits2.UdunitsError: UT_PARSE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/cf_units/init.py", line 189, in
_ud_system = _ud.read_xml(config.get_xml_path())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "cf_units/_udunits2.pyx", line 194, in cf_units._udunits2.read_xml
File "cf_units/_udunits2.pyx", line 104, in cf_units._udunits2.wrap_system
File "cf_units/_udunits2.pyx", line 184, in cf_units._udunits2._raise_error
cf_units._udunits2.UdunitsError: UT_PARSE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_main.py", line 518, in run
fire.Fire(ESMValTool())
^^^^^^^^^^^^
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_main.py", line 319, in init
self.setattr(entry_point.name, entry_point.load()())
^^^^^^^^^^^^^^^^^^
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/importlib/metadata/init.py", line 202, in load
module = import_module(match.group('module'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/esmvaltool/cmorizers/data/cmorizer.py", line 17, in
from esmvalcore._task import write_ncl_settings
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_task.py", line 24, in
from ._citation import _write_citation_files
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_citation.py", line 10, in
from .config._diagnostics import DIAGNOSTICS
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/config/init.py", line 11, in
from ._config_object import CFG, Config, Session
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/config/_config_object.py", line 12, in
from esmvalcore.cmor.check import CheckLevels
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/cmor/check.py", line 11, in
import cf_units
File "/home/jmindlin/mambaforge/envs/esmvaltool/lib/python3.11/site-packages/cf_units/init.py", line 192, in
raise OSError(
OSError: [UT_PARSE] Failed to open UDUNITS-2 XML unit database

bouweandela · 2024-05-14T15:02:55Z

It is related to udunits2, which is apparently not able to find its units database (the file udunits2.xml). You can manually set the path with the environmental variable UDUNITS2_XML_PATH, but usually this works out of the box when installing everything with mamba.

jumin94 · 2024-05-14T15:57:22Z

Thank you for your answer @bouweandela , I did install everything with mamba though. Will try again and also try manually setting the path as well.

valeriupredoi · 2024-05-16T12:26:50Z

this is most probably because there was no conda/mamba activate $ENV where $ENV is the name of your environment - all the system variables are set at activation point, including UDUNITS2_XML_PATH and ESMFMKFILE (the initializer file for ESMF) - could you please tell us how you initialized the environment, and also what version of cf_units you are using by pasting the output of conda list cf-units please 🍺

jumin94 · 2024-06-13T14:22:13Z

hello @valeriupredoi below is the version of cf_units that I am using

packages in environment at /opt/miniforge3/envs/esmvaltool:

Name Version Build Channel
cf-units 3.2.0 py311h18e1886_5 conda-forge

jumin94 · 2024-07-01T14:00:15Z

Hello

I still haven't found a solution for this. A colleague has made a global installation of esmvaltool that works for him either by logging in with his credentials or with mine, but when I try to run esmvaltool I get the exact same issue as I posted above. Does anyone know what might be happening?

Thank you

jumin94 · 2024-07-05T12:49:34Z

It is related to udunits2, which is apparently not able to find its units database (the file udunits2.xml). You can manually set the path with the environmental variable UDUNITS2_XML_PATH, but usually this works out of the box when installing everything with mamba.

@bouweandela when I activate the esmvaltool enviroment the path exists, and I get the same error evwen when I set the path manualy:


(base) jmindlin@barat:~$ conda activate esmvaltool
(esmvaltool) jmindlin@barat:~$ ls $UDUNITS2_XML_PATH
/opt/miniforge3/envs/esmvaltool/share/udunits/udunits2.xml
(esmvaltool) jmindlin@barat:~$ esmvaltool
ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/cf_units/__init__.py", line 186, in <module>
    _ud_system = _ud.read_xml()
                 ^^^^^^^^^^^^^^
  File "cf_units/_udunits2.pyx", line 194, in cf_units._udunits2.read_xml
  File "cf_units/_udunits2.pyx", line 104, in cf_units._udunits2.wrap_system
  File "cf_units/_udunits2.pyx", line 184, in cf_units._udunits2._raise_error
cf_units._udunits2.UdunitsError: UT_PARSE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/cf_units/__init__.py", line 189, in <module>
    _ud_system = _ud.read_xml(config.get_xml_path())
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cf_units/_udunits2.pyx", line 194, in cf_units._udunits2.read_xml
  File "cf_units/_udunits2.pyx", line 104, in cf_units._udunits2.wrap_system
  File "cf_units/_udunits2.pyx", line 184, in cf_units._udunits2._raise_error
cf_units._udunits2.UdunitsError: UT_PARSE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_main.py", line 518, in run
    fire.Fire(ESMValTool())
              ^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_main.py", line 319, in __init__
    self.__setattr__(entry_point.name, entry_point.load()())
                                       ^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvaltool/cmorizers/data/cmorizer.py", line 17, in <module>
    from esmvalcore._task import write_ncl_settings
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_task.py", line 24, in <module>
    from ._citation import _write_citation_files
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_citation.py", line 10, in <module>
    from .config._diagnostics import DIAGNOSTICS
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/config/__init__.py", line 11, in <module>
    from ._config_object import CFG, Config, Session
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/config/_config_object.py", line 12, in <module>
    from esmvalcore.cmor.check import CheckLevels
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/cmor/check.py", line 11, in <module>
    import cf_units
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/cf_units/__init__.py", line 192, in <module>
    raise OSError(
OSError: [UT_PARSE] Failed to open UDUNITS-2 XML unit database

@valeriupredoi , I get the error when I have activated the environment with conda, this is what I get from 'conda list cf-units':

(esmvaltool) jmindlin@barat:~$ conda list cf-units
# packages in environment at /opt/miniforge3/envs/esmvaltool:
#
# Name                    Version                   Build  Channel
cf-units                  3.2.0           py311h18e1886_5    conda-forge

Nobody in my research group is being able to help - we're all puzzled! thanks!

valeriupredoi · 2024-07-05T13:14:37Z

@jumin94 am looking at this in more detail right now 👍 Could you pls tell me what OS you running on, and what version of udunits2 is installed in the env? Cheers!

valeriupredoi · 2024-07-05T13:22:19Z

it appears this has been encountered by others, but it's a ghost bug ie nobody apart from the OPs can reproduce the issue pydata/xarray#4257 - suggestions range from recreating the env, clearing conda cache etc - I honestly think this is an actual system-specific bug, and is fully reproducible until they patch it. Let's get the ball rolling by getting the info on OS, udunits2 version, and contents of the .condarc file please 🍺

jumin94 · 2024-07-09T12:42:48Z

Thank you @valeriupredoi - it's interesting to see that others have this issue too, below is the information:

OS:

jmindlin@bora:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

cf-units

# packages in environment at /opt/miniforge3/envs/esmvaltool:
#
# Name                    Version                   Build  Channel
cf-units                  3.2.0           py311h18e1886_5    conda-forge

However, something very very weird happened, today I installed the VPN to the cluster in a different notebook and connected - now it works. I don't understand why or how. If you want to close this, I'm OK with it, this is certainly a problem with our cluster or with my personal computer. It still remains a mystery to me and I'm happy to continue trying things in the other notebook to see what happended.

Cheers

jumin94 · 2024-07-10T10:48:17Z

If you want to close this, please go ahead, although it would be amazing to know how to fix this, given that now I'm only able to run esmvaltool on this server if I connect from my personal computer!

valeriupredoi · 2024-07-10T10:52:11Z

Hi @jumin94 give me a few, not closing it just yet, would like to see if I can replicate the issue - what exactly did the VPN install you mentioned do, please? Looking at this toddy 👍🍺

jumin94 · 2024-07-10T11:06:54Z

The VPN is the Leipzig University VPN: https://www.urz.uni-leipzig.de/unsere-services/servicedetail/service/vpn-zugang-zum-uni-netz

But I find that this is not relevan, what is relevant is that wehn I connect to the cluster from a differnt notebook (using VPN or wired connection) I can run esmvaltool with no problems, but when I do it from one particular laptop I can't. I have no idea how this is possible, given that I understood that once I did ssh to a server it didn't matter what laptop I was doing it from.

bouweandela · 2024-07-10T13:12:47Z

From your description, it sounds like this may be an issue with your locale settings. We have some advice on how to correct that here: https://docs.esmvaltool.org/en/latest/quickstart/installation.html#problems-when-using-ssh

As a bit of background: any error messages when loading the units database are hidden by this code:
https://github.com/SciTools/cf-units/blob/f57a7f1fdba4e9424e4dbbc01b614e521a88086c/cf_units/__init__.py#L181-L182. If you comment out the suppress_errors line, you will get more information on the error. I am able to reproduce at least a similar issue by setting export LC_NUMERIC=nl_NL.UTF-8 (a language in which the decimal number separator is a , instead of a . as used in the e.g. this file: https://github.com/Unidata/UDUNITS-2/blob/c83da987387db1174cd2266b73dd5dd556f4476b/lib/udunits2-prefixes.xml#L46, but for me the issue only appears when trying the build the documentation for some reason.

valeriupredoi · 2024-07-10T13:15:43Z

indeed, after reading @jumin94 's latest comments I was heading over to messing with the locale see if I can reproduce the problem 👍

valeriupredoi · 2024-07-10T13:21:57Z

From your description, it sounds like this may be an issue with your locale settings. We have some advice on how to correct that here: https://docs.esmvaltool.org/en/latest/quickstart/installation.html#problems-when-using-ssh

As a bit of background: any error messages when loading the units database are hidden by this code: https://github.com/SciTools/cf-units/blob/f57a7f1fdba4e9424e4dbbc01b614e521a88086c/cf_units/__init__.py#L181-L182. If you comment out the suppress_errors line, you will get more information on the error. I am able to reproduce at least a similar issue by setting export LC_NUMERIC=nl_NL.UTF-8 (a language in which the decimal number separator is a , instead of a . as used in the e.g. this file: https://github.com/Unidata/UDUNITS-2/blob/c83da987387db1174cd2266b73dd5dd556f4476b/lib/udunits2-prefixes.xml#L46, but for me the issue only appears when trying the build the documentation for some reason.

perhaps this exact issue here? pydata/xarray#4257

jumin94 · 2024-09-25T09:20:59Z

Based on your last comment, and that it seems that the issue stems from the LC_NUMERIC locale variable is being set to a locale where the decimal point is a comma (,) instead of a period (.), causing conflicts with cf_units. We added the following line to the ~/.bashrc file, which fixes the issue:

export LC_NUMERIC=C

do you think this is a good solution @valeriupredoi ?

valeriupredoi · 2024-09-25T11:39:42Z

@jumin94 good find! As @bouweandela aslo points out in SciTools/cf-units#435 that is indeed the problem causing such troubles; setting the locale externally vs handling it internally eg Bouwe's PR SciTools/cf-units#436 is a choice of approach, I'd do it externally like you are, if you asked me, so that there are no issues from other packages 🍺

valeriupredoi added the installation label May 16, 2024

valeriupredoi mentioned this issue Jul 5, 2024

local build of docs fails with OSError: [UT_PARSE] Failed to open UDUNITS-2 XML unit database pydata/xarray#4257

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installation failure in University of Leipzig cluster #3589

Installation failure in University of Leipzig cluster #3589

jumin94 commented May 10, 2024

bouweandela commented May 14, 2024 •

edited

Loading

jumin94 commented May 14, 2024

valeriupredoi commented May 16, 2024

jumin94 commented Jun 13, 2024 •

edited

Loading

jumin94 commented Jul 1, 2024

jumin94 commented Jul 5, 2024

valeriupredoi commented Jul 5, 2024 •

edited

Loading

valeriupredoi commented Jul 5, 2024

jumin94 commented Jul 9, 2024

jumin94 commented Jul 10, 2024

valeriupredoi commented Jul 10, 2024 •

edited

Loading

jumin94 commented Jul 10, 2024

bouweandela commented Jul 10, 2024 •

edited

Loading

valeriupredoi commented Jul 10, 2024

valeriupredoi commented Jul 10, 2024

jumin94 commented Sep 25, 2024 •

edited

Loading

valeriupredoi commented Sep 25, 2024

Installation failure in University of Leipzig cluster #3589

Installation failure in University of Leipzig cluster #3589

Comments

jumin94 commented May 10, 2024

bouweandela commented May 14, 2024 • edited Loading

jumin94 commented May 14, 2024

valeriupredoi commented May 16, 2024

jumin94 commented Jun 13, 2024 • edited Loading

jumin94 commented Jul 1, 2024

jumin94 commented Jul 5, 2024

valeriupredoi commented Jul 5, 2024 • edited Loading

valeriupredoi commented Jul 5, 2024

jumin94 commented Jul 9, 2024

jumin94 commented Jul 10, 2024

valeriupredoi commented Jul 10, 2024 • edited Loading

jumin94 commented Jul 10, 2024

bouweandela commented Jul 10, 2024 • edited Loading

valeriupredoi commented Jul 10, 2024

valeriupredoi commented Jul 10, 2024

jumin94 commented Sep 25, 2024 • edited Loading

valeriupredoi commented Sep 25, 2024

bouweandela commented May 14, 2024 •

edited

Loading

jumin94 commented Jun 13, 2024 •

edited

Loading

valeriupredoi commented Jul 5, 2024 •

edited

Loading

valeriupredoi commented Jul 10, 2024 •

edited

Loading

bouweandela commented Jul 10, 2024 •

edited

Loading

jumin94 commented Sep 25, 2024 •

edited

Loading