macOS universal2 package installer #1570

nrnhines · 2022-01-02T22:25:06Z

nrn/bldnrnmacpkgcmake.sh on Apple M1 by default generates pkg installer for arm64 x86_64 for Python 3.8, 3,9, 3.10

Updated docs

Useful error message generated if dlopen fails because libpython, libmpi, or libnrnmech has only an available architecture that differs from the architecture in use by libnrniv

I'm guessing it would be possible to do universal builds on a macOS x86_64. But all libraries linked against likely have to be universal. (note that openmpi is not linked against during the build). In particular, python.org distributes universal builds for python3.8, 3.9, and 3.10.

Try nrn-8.0a-718-g6e80a61cf-osx-arm64-x86_64-py-38-39-310.pkg at https://github.com/neuronsimulator/nrn/releases/tag/8.0a

codecov-commenter · 2022-01-02T22:41:23Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 45.20%. Comparing base (9624c96) to head (a09fe20).
Report is 1877 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1570   +/-   ##
=======================================
  Coverage   45.20%   45.20%           
=======================================
  Files         551      551           
  Lines      111184   111184           
=======================================
  Hits        50256    50256           
  Misses      60928    60928

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pramodk · 2022-01-03T19:52:43Z

@nrnhines : could you share / upload the installer here that you have built & tested?

nrnhines · 2022-01-04T00:41:43Z

@pramodk See the last line of the introductory comment above.
It turned out to be straightforward to do universal builds on a mac powerbook x86_64. (at least to the point where lipo -archs libnrniv.dylib outputs x86_64 arm64. I'll try my hand at building a full pkg and if successful will update this PR so that a universal build does not have an arm64 build machine as a requirement. I'm now thinking that universal macOS wheels will not be out of reach as long as one can install universal pythons from python.org

alexsavulescu

Overall LGTM

src/mac/CMakeLists.txt

CMakeLists.txt

alexsavulescu · 2022-01-05T10:34:11Z

Try nrn-8.0a-718-g6e80a61cf-osx-arm64-x86_64-py-38-39-310.pkg at https://github.com/neuronsimulator/nrn/releases/tag/8.0a

tested on my intel macbook air and seems to run just fine. Just one remark for rxd:

bb-fvfzj1tzlywp:bin savulesc$ python3.9 -c "import neuron; neuron.test_rxd(); quit()"
test_ecs_diffusion_fixed_step (neuron.tests.test_rxd.RxDTestCase) ... ok
test_ecs_diffusion_variable_step_coarse (neuron.tests.test_rxd.RxDTestCase) ... ok
test_ecs_diffusion_variable_step_fine (neuron.tests.test_rxd.RxDTestCase) ... ok
test_rxd (neuron.tests.test_rxd.RxDTestCase) ... ld: warning: dylib (/Applications/NEURON/share/nrn/../../lib/librxdmath.dylib) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: dylib (/Applications/NEURON/share/nrn/../../lib/librxdmath.dylib) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: dylib (/Applications/NEURON/share/nrn/../../lib/librxdmath.dylib) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: dylib (/Applications/NEURON/share/nrn/../../lib/librxdmath.dylib) was built for newer macOS version (12.0) than being linked (11.0)
ld: warning: dylib (/Applications/NEURON/share/nrn/../../lib/librxdmath.dylib) was built for newer macOS version (12.0) than being linked (11.0)
ok

----------------------------------------------------------------------
Ran 4 tests in 4.986s

OK

Co-authored-by: Alexandru Săvulescu <[email protected]>

alexsavulescu · 2022-01-05T12:01:11Z

Also tested on an M1 (arm64 + rosetta) and things also look fine.

nrnhines · 2022-01-05T12:14:48Z

was built for newer macOS version (12.0) than being linked (11.0)

I'm presently a bit confused about what to do about this. I need to experiment a bit.
On the M1 (which I recently upgraded to Monterey) I have not been setting MACOSX_DEPLOYMENT_TARGET, so for that build it ended up by default as 12.0. The bldnrnmac... script contains the legacy fragment

#10.7 possible if one builds with pythons that are consistent with that.
if test "$CPU" = "x86_64" ; then  
  export MACOSX_DEPLOYMENT_TARGET=10.9
fi

If I use the 10.9 value on M1, I'm getting

error: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.9" but "11" during configure

I guess because the python3.8 that was being used was being used during that compile stage was
creating build/temp.macosx-11-universal2-3.8

alexsavulescu · 2022-01-05T12:24:22Z

error: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.9" but "11" during configure

was that a clean build?

nrnhines · 2022-01-05T15:09:32Z

was that a clean build

Yes. But there may not be a perfect solution to the problem. I created a new pkg distribution from the latest 54e9009 and uploaded to https://github.com/neuronsimulator/nrn/releases/tag/8.0a as nrn-8.0a-725-g54e9009fd-osx-arm64-x86_64-py-38-39-310.pkg
On my x86_64 macos both my old Python 10.8 and 10.9 (after having to pip install --upgrade numpy) generate the warnings you saw, but now as the form warning: ... built for newer macOS version (11.0) than being linked (10.9)

So it seems to be way too easy to get into a situation where the python macOS version will mismatch the NEURON macOS version. The only solution I can think of is that the user would have to install a more recent python or a python with a more recent build.

alexsavulescu · 2022-01-06T18:27:37Z

Indeed the official universal2 installers come in with macOS 11.0.

The only solution I can think of is that the user would have to install a more recent python or a python with a more recent build.

can't think of any other solution either. We'd also need to rename our pkgs following official installers i.e.:
nrn-*-osx-arm64-x86_64-py-38-39-310.pkg -> nrn-*-macos11-arm64-x86_64-py-38-39-310.pkg

alexsavulescu · 2022-01-12T14:42:10Z

I'm wondering what strategy we could adopt here. A way to move forward (as a transition?) would be to continue to provide original pkg + universal2 ?

nrn-*-macos11-arm64-x86_64-py-38-39-310.pkg
nrn-*-macosx10.9-x86_64-py-36-37-38-39-310.pkg

nrnhines · 2022-01-12T18:10:53Z

original pkg + universal2

That seems like a reasonable way to cover things. I note that uname -a does not specify but with the Apple icon, clicking on "About This Mac", click on "System Report", click on "Software" does show the item ```System Version: macOS 12.1 (21C52)

edit: missed this. The Apple Icon "About This Mac" shows the version number under the name. eg.

macOS Monterey
Version 12.1

Anyway. I'll adjust the installer builder script to go either way.

nrnhines · 2022-01-16T15:09:34Z

Selecting a MACOSX_DEPLOYMENT_TARGET has become confusing. I'm experimenting with the following fragment in bldnrnmacpkgcmake.sh on may Apple M1

# Choose MACOSX_DEPLOYMENT_TARGET consistent with all Pythons
macosver=""
for i in $args ; do
  mver=`$i -c 'import sysconfig; print(sysconfig.get_config_var("MACOSX_DEPLOYMENT_TARGET")); quit()'`
  echo "macos version for $i is $mver"
  if test "$macosver" = "" ; then
    macosver=$mver
  fi
  if test "$macosver" != "$mver" ; then
    echo "$i macos $mver differs from previous python macos $macosver"
#    exit 1
  fi
done
export MACOSX_DEPLOYMENT_TARGET=$macosver
echo "MACOSX_DEPLOYMENT_TARGET=$MACOSX_DEPLOYMENT_TARGET"
exit 0

and the output is

% NRN_SRC=`pwd` bash bldnrnmacpkgcmake.sh
macos version for python3.8 is 11
macos version for python3.9 is 10.9
python3.9 macos 10.9 differs from previous python macos 11
macos version for python3.10 is 10.9
python3.10 macos 10.9 differs from previous python macos 11
MACOSX_DEPLOYMENT_TARGET=11

I could have sworn that all the universal pythons I installed from http://python.org were for macOS 11. However there does not seem to be any conflict between the pythons with macOS 10.9 and the NEURON with macOS 11.
Note that forcing MACOSX_DEPLOYMENT_TARGET=10.9 explicitly will fail when building the python3.8 related library.

Perhaps a clue to all this is that the current python.org distribution for 3.9.10 has the note:
macOS 64-bit universal2 installer ... for macOS 10.9 and later
even though the name of the pkg file is python-3.9.10-macos11.pkg. On running the installer the front page states
This package will install Python 3.9.10 for macOS 10.9 or later.

Now, there is no current binary distribution for 3.8.12 and the latest one I can find is for 3.8.10 where the download page says

macOS 64-bit Intel installer | macOS | for macOS 10.9 and later
macOS 64-bit universal2 installer | macOS | experimental, for macOS 11 Big Sur and later; recommended on Apple Silicon

By the way, 3.10 universal installers are also for macOS 10.9 and later
So I conclude the best strategy is to leave out the python3.8 from the default args (leaving python3.9 python3.10) so that the universal installer will work for macos10.9 and later.

macOS 64-bit Intel installer	macOS	for macOS 10.9 and later

alexsavulescu · 2022-01-16T15:13:26Z

So I conclude the best strategy is to leave out the python3.8 from the default args (leaving python3.9 python3.10) so that the universal installer will work for macos10.9 and later.

That looks sound to me

nrnhines · 2022-01-16T22:47:12Z

I was able to build on the Apple M1 a universal2 macos10.9 of python3.8.12 from sources with

./configure --prefix=$HOME/soft/python3.8 --enable-universalsdk --with-universal-archs=universal2 --enable-shared --with-openssl=/opt/homebrew/opt/openssl

But as the openssl option did not suffice to get a working pip3.8, I also had to

cp -R /Users/hines/Library/Python/3.8/lib/python/site-packages/numpy /Users/hines/soft/python3.8/lib/python3.8/site-packages

Anyway the resulting nrn-8.0a-736-g93c9d91aa-macos10.9-arm64-x86_64-py-38-39-310.pkg works for
neuron.test() and neuron.test_rxd() for both /Users/hines/soft/python3.8/bin/python3.8 and /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8

I'll upload to Release 8.0a artifacts

nrnhines · 2022-01-16T23:16:55Z

I'm now thinking that it would be best to have the decision about universal2 and MACOSX_DEPLOYMENT_TARGET based on the target Python(s) be determined within CMake. Seems like that would pave the way for making universal2 wheels.

alexsavulescu

@nrnhines I think this can be merged?

alexsavulescu · 2022-02-18T12:56:50Z

Seems like that would pave the way for making universal2 wheels.

Indeed. We could maybe try to do the universal2 wheels soon or at least m1 wheels.

nrnhines · 2022-02-18T14:58:50Z

I only have a concern about the proper use of MACOS_DEPLOYMENT_TARGET and how safe it is when the Python run on the user machine is different.

nrnhines · 2022-02-22T13:23:49Z

Re-exploring MACOS_DEPLOYMENT_TARGET=10.9, I'm seeing that it works in the sense of -c import neuron; neuron.test for all python3.8, 3,9, 3.10 on x86_64 and arm64. That includes the M1 python3.8.10 which is macosx-11-universal2 and brew installed python3.9, python3.10 which is macosx-12-arm64. So it appears that, although the build has to be consistent with the python build, thereafter it works for pythons with any greater macosx target. By the way, to do a universal2 build for python3.8, I had to build python3.8 from sources (see earlier comment #1570 (comment) )

nrnhines · 2022-02-22T13:28:25Z

The final change to this PR that I think should be done is to change the file nameing convention to use the sysconfig.platform() name. I.e instead of nrn...-macos10.9-arm64-x86_64-... it would be nrn...-macosx-10.9-universal2-...

alexsavulescu · 2022-02-22T13:38:49Z

IMO having 3.8.12 built from source so that we can fully be consistent wrt MACOSX_DEPLOYMENT_TARGET=10.9 would be appropriate for the pkg installer.

nrnhines added 4 commits January 1, 2022 17:40

On mac arm64 can do a universal2 build for "x86_64 arm64"

76a270b

fix some typos. bldnrnmacpkgcmake.sh works for universal2

f4d0212

doc for universal2 install packages.

48616e9

macOS,arm64 stderr message if mismatched arch for dlopen.

e81e026

nrnhines requested review from pramodk and alexsavulescu January 2, 2022 22:25

nrnhines added 2 commits January 2, 2022 20:35

Some doc details about notarization.

6e80a61

Update mac_pkg.md

d29db7a

nrnhines changed the title ~~macOS unversal2 package installer~~ macOS universal2 package installer Jan 3, 2022

nrnhines added 2 commits January 4, 2022 09:39

macOS x86_64 machine can also build a universal2 package.

0de4d4b

Merge branch 'master' into hines/universal2-macos

abbe04a

alexsavulescu reviewed Jan 5, 2022

View reviewed changes

src/mac/CMakeLists.txt Outdated Show resolved Hide resolved

CMakeLists.txt Outdated Show resolved Hide resolved

CMakeLists.txt Outdated Show resolved Hide resolved

nrnhines and others added 2 commits January 5, 2022 06:33

Apply suggestions from code review

9c4cb48

Co-authored-by: Alexandru Săvulescu <[email protected]>

fix brew install instructions

be6255a

MACOSX_DEPLOYMENT_TARGET=11

54e9009

nrnhines added 3 commits January 7, 2022 15:06

macos version appears in installer name

b9a811a

update macos instructions

2e4e3a2

Merge branch 'master' into hines/universal2-macos

16cd0ca

Merge branch 'master' into hines/universal2-macos

fb05a84

nrnhines added 2 commits January 16, 2022 16:11

Choose MACOSX_DEPLOYMENT_TARGET consistent with all Pythons

93c9d91

Merge branch 'master' into hines/universal2-macos

cacb3ee

nrnhines mentioned this pull request Jan 20, 2022

Add support to release Linux AArch64 wheels #1516

Closed

Merge branch 'master' into hines/universal2-macos

2be921f

alexsavulescu approved these changes Feb 18, 2022

View reviewed changes

Merge branch 'master' into hines/universal2-macos

ced504a

nrnhines added 3 commits February 23, 2022 12:47

use sysconfig.get_platform() to determine name of pkg file.

11e50b1

Merge branch 'master' into hines/universal2-macos

ea91228

Latest mingw changed location of crtbegin.o

a09fe20

nrnhines merged commit 83b7831 into master Feb 24, 2022

nrnhines deleted the hines/universal2-macos branch February 24, 2022 01:52

nrnhines mentioned this pull request Feb 25, 2022

Add support for macOS arm64 wheels #1649

Merged

alexsavulescu mentioned this pull request Feb 25, 2022

Python version support policy #1097

Closed

alexsavulescu mentioned this pull request Mar 22, 2022

NEURON 8.1.0 Release #1719

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macOS universal2 package installer #1570

macOS universal2 package installer #1570

nrnhines commented Jan 2, 2022 •

edited

Loading

codecov-commenter commented Jan 2, 2022 •

edited by codecov bot

Loading

pramodk commented Jan 3, 2022

nrnhines commented Jan 4, 2022

alexsavulescu left a comment

alexsavulescu commented Jan 5, 2022

alexsavulescu commented Jan 5, 2022

nrnhines commented Jan 5, 2022

alexsavulescu commented Jan 5, 2022

nrnhines commented Jan 5, 2022

alexsavulescu commented Jan 6, 2022

alexsavulescu commented Jan 12, 2022

nrnhines commented Jan 12, 2022 •

edited

Loading

nrnhines commented Jan 16, 2022

alexsavulescu commented Jan 16, 2022

nrnhines commented Jan 16, 2022

nrnhines commented Jan 16, 2022

alexsavulescu left a comment

alexsavulescu commented Feb 18, 2022

nrnhines commented Feb 18, 2022

nrnhines commented Feb 22, 2022

nrnhines commented Feb 22, 2022

alexsavulescu commented Feb 22, 2022

macOS universal2 package installer #1570

macOS universal2 package installer #1570

Conversation

nrnhines commented Jan 2, 2022 • edited Loading

codecov-commenter commented Jan 2, 2022 • edited by codecov bot Loading

Codecov Report

pramodk commented Jan 3, 2022

nrnhines commented Jan 4, 2022

alexsavulescu left a comment

Choose a reason for hiding this comment

alexsavulescu commented Jan 5, 2022

alexsavulescu commented Jan 5, 2022

nrnhines commented Jan 5, 2022

alexsavulescu commented Jan 5, 2022

nrnhines commented Jan 5, 2022

alexsavulescu commented Jan 6, 2022

alexsavulescu commented Jan 12, 2022

nrnhines commented Jan 12, 2022 • edited Loading

nrnhines commented Jan 16, 2022

alexsavulescu commented Jan 16, 2022

nrnhines commented Jan 16, 2022

nrnhines commented Jan 16, 2022

alexsavulescu left a comment

Choose a reason for hiding this comment

alexsavulescu commented Feb 18, 2022

nrnhines commented Feb 18, 2022

nrnhines commented Feb 22, 2022

nrnhines commented Feb 22, 2022

alexsavulescu commented Feb 22, 2022

nrnhines commented Jan 2, 2022 •

edited

Loading

codecov-commenter commented Jan 2, 2022 •

edited by codecov bot

Loading

nrnhines commented Jan 12, 2022 •

edited

Loading