FileNotFoundError: cfunits requires UNIDATA UDUNITS-2. Can't find the 'udunits2' library. on M2 Macbook Pro

[wknoben]

I'm having some trouble getting cfunits to work on my M2 Macbook Pro. Installing cfunits 3.3.6 reports no errors, but trying to import the package returns a library not found error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/[snip]/lib/python3.11/site-packages/cfunits/__init__.py", line 47, in <module>
    from .units import Units  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[snip]/lib/python3.11/site-packages/cfunits/units.py", line 43, in <module>
    raise FileNotFoundError(
FileNotFoundError: cfunits requires UNIDATA UDUNITS-2. Can't find the 'udunits2' library.

udunits is present on the system:

brew info udunits
==> udunits: stable 2.2.28 (bottled), HEAD
Unidata unit conversion library
https://www.unidata.ucar.edu/software/udunits/
/opt/homebrew/Cellar/udunits/2.2.28 (29 files, 659.5KB) *
  Poured from bottle using the formulae.brew.sh API on 2024-01-19 at 15:24:29
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/u/udunits.rb
==> Dependencies
Build: texinfo ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 105 (30 days), 477 (90 days), 1,702 (365 days)
install-on-request: 67 (30 days), 293 (90 days), 957 (365 days)
build-error: 0 (30 days)

I (sequentially) tried setting paths and reinstalling cfunits to no avail:

export LD_LIBRARY_PATH=/opt/homebrew/lib/libudunits2.0.dylib`

and

export UDUNITS2_INCDIR=/opt/homebrew/Cellar/udunits/2.2.28/include
export UDUNITS2_LIBDIR=/opt/homebrew/Cellar/udunits/2.2.28/lib
export UDUNITS2_XML_PATH=/opt/homebrew/Cellar/udunits/2.2.28/share/udunits/udunits2-common.xml

Further digging brings up to this related issue: https://github.com/orgs/Homebrew/discussions/3424.

Summary: ctypes.util.find_library() will look in /usr/lib and /usr/local/lib to find udunits2 but on newer Apple hardware libraries will be installed in /opt/homebrew instead

cfunits/cfunits/units.py

Line 39 in fac74a5

_libpath = ctypes.util.find_library("udunits2")

The linked thread suggest setting a symlink will solve this, but on institutional machines (like mine), I wasn't able to make this work.

I don't know if this is solvable on cfunits's side of things, but it seemed good to at least document this in case anyone else runs into this.

[sadielbartholomew]

Thanks for reporting this and for all the detail which is really helpful, @wknoben. On our small dev team we don't have much Mac OS experience (we are hardcore Linux users) which means we can't readily investigate this, but we do have access to a Mac in the office so we may be able to look into it in due course (though I am currently working fully remotely for the foreseeable so this is unlikely to be soon). (And unless I am mistake, one can't legally get a VM for Mac OS.)

I see from your report that you have followed the advice as quoted from the cfunits installation page, namely:

If the UDUNITS-2 shared library file (libudunits2.so.0 on GNU/Linux or libudunits2.0.dylibfile on MacOS) is in a non-standard location then its directory path should be added to the LD_LIBRARY_PATH environment variable. It may also be necessary to specify the location (directory path and file name) of the udunits2.xml file in the UDUNITS2_XML_PATH environment variable, although the default location is usually correct. For example, export UDUNITS2_XML_PATH=/home/user/anaconda3/share/udunits/udunits2.xml.

so there's not anything else I could suggest without the ability to play around on Mac OS myself,

As a workaround for installation, I would otherwise suggest you use conda (or wrappers to it such as mamba), but since you mention you are on an institutional machine, I guess this is not an option you can take? If not, then I'm afraid on our side there is not anything we can immediately offer to make it work (unless someone on my team else chips in with an idea).

I don't know if this is solvable on cfunits's side of things, but it seemed good to at least document this in case anyone else runs into this.

Indeed, regardless having this report up will be helpful to other Mac users, so thanks again for the write up. Please do let us know here if you find a way to get it working with Mac OS, or a workaround that can be applied without conda.

[bnlawrence]

I have this working on apple silicon ... but I can't remember what I did! Which isn't much help right now. ... I do have another Mac which I know doesn't have cf-python on it, so I'll have a go on that and see if I can a) recreate your problem, and b) work out what I must have done.

Meanwhile, is the issue that you can't even import cf because of this error, or does it manifest under some particular circumstances?

[bnlawrence]

@davidhassell I vaguely remember something about installing it from the GitHub version, can you recall what we did? I remember that I was stymied about something in the cf-python install, asked you to come in, and more or less instantly we got it going ... it was so instant that I never thought to document it :-(

[wknoben]

@sadielbartholomew Thanks for chiming in. Installing through conda prevents this issue for me. Unfortunately it doesn't solve my entire problem because I can't get the package I'm actually after (which has cfunits as a requirement) installed with conda for a different reason. The specific problem documented here can be avoided this way though.

Solution code for those looking for it:

conda create -n cfunits_test 
conda activate cfunits_test
conda install -c conda-forge cfunits

[wknoben]

@bnlawrence Thanks for taking a look. This prevents me from importing the package at all. import cfunits immediately reports the error in the top post.

[bnlawrence]

@wknoben The memory archaeology I've done so far suggests that I probably ended up using mamba and the problem went away ... can you not install cfunits with conda, then pip install the package you want into that environment?

[wknoben]

Possibly, I haven't tried that yet. In my experience that can get messy but it may be worth it if it's the only way to make things work.

[DocOtak]

Should it be useful, I got around this by setting the DYLD_LIBRARY_PATH envar from inside python before trying to import the package.

import os
os.environ["DYLD_LIBRARY_PATH"] = "/opt/homebrew/Cellar/udunits/2.2.28/lib/"
from cfunits import Units

I think this is SIP that is sanitizing the LD_LIBRARY_PATH and DYLD_LIBRARY_PATH envars because the executable (the python interpreter) does not have the required entitlements.