Add macos support in build.rs for linking to Intel MKL

[coreylowman]

MacOS support needs REDIST_DIRS, LINK_DIRS, and LINK_LIBS.

Originally posted by @coreylowman in #64 (comment)

[coreylowman]

Root issue #18

[yerke]

Oops. I didn't that there was a separate issue for macOS and opened #72.

I was able to run all examples (except for mnist since I don't have the input files) successfully on macOS 12.4 with MBP 16 M1 Max.

I wasn't able to install Intel oneAPI Base Toolkit. It breaks during installation on Distribution for Python component with the error below. Do I need the whole Toolkit or only parts of it?

07/13/2022 17:51:04:545 : 247755 : ERROR : Installation of component has failed.
Component id: intel.oneapi.mac.python3, name: Intel® Distribution for Python*, version: 2022.1.0-213.
During the execution of the application '/var/intel/installercache/downloadcache/intel.oneapi.mac.python3,v=2022.1.0-213/exec_script.sh' with arguments '/opt/intel/oneapi' errors were received:'+ xargs /bin/sh
'
Error: Sequence execution failed.

[joseph-x-li]

I doubt we can install Intel MKL on M1 macs. I have access to an Intel Mac, will get back with an attempt on that one. By the way, I am "YRUDancin" from reddit.

[coreylowman]

@yerke you only need one part of the install. i think it's called something like "Intel oneAPI Math Kernel Library"? You shouldn't need to install anything for python.

[coreylowman]

I doubt we can install Intel MKL on M1 macs. I have access to an Intel Mac, will get back with an attempt on that one. By the way, I am "YRUDancin" from reddit.

Can you expand on this? I had assumed since the mkl install page had macos as one of the dropdown options that it would be possible.

[joseph-x-li]

I assume Intel MKL uses the x86-64 ISA, so it will only work with Macs with an Intel chip. M1 macs are all arm64e. You will likely have to fall back to a different BLAS implementation for M1 Macs.

Anyways, I have MKL and mkl_link_tool, what command would you like me to run?

[yerke]

Running mkl_link_tool on macOS 12.4 with MBP 14 Intel:

Configuration
=============

OS:                     mac
Architecture:           intel64
Compiler:               clang
Linking:                dynamic
Interface layer:        lp64
Parallel:               yes
OpenMP library:         iomp5

Output
======

Compiler option(s):
 -m64  -I"/opt/intel/oneapi/mkl/2022.1.0/include"

Linking line:
 -L${MKLROOT}/lib -Wl,-rpath,${MKLROOT}/lib -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm -ldl

MKLROOT is /opt/intel/oneapi as on Linux.

% ll compiler/latest/mac/compiler/lib
.rw-r--r-- 1.8M root 26 Feb  6:16 libiomp5.dylib
.rw-r--r-- 206k root 26 Feb  6:16 libiomp5_db.dylib
.rw-r--r-- 176k root 26 Feb  6:16 libiompstubs5.dylib

% ll mkl/latest/lib
drwxr-xr-x    - root 13 Jul 18:47 cmake
.rwxr-xr-x  52M root 11 Mar 22:03 libmkl_avx.2.dylib
.rwxr-xr-x  50M root 11 Mar 22:04 libmkl_avx2.2.dylib
.rwxr-xr-x  66M root 11 Mar 22:04 libmkl_avx512.2.dylib
.rwxr-xr-x 497k root 11 Mar 22:04 libmkl_blacs_mpich_ilp64.2.dylib
.rw-r--r-- 951k root 11 Mar 21:44 libmkl_blacs_mpich_ilp64.a
lrwxr-xr-x   32 root 29 Mar  3:02 libmkl_blacs_mpich_ilp64.dylib -> libmkl_blacs_mpich_ilp64.2.dylib
.rwxr-xr-x 290k root 11 Mar 22:04 libmkl_blacs_mpich_lp64.2.dylib
.rw-r--r-- 578k root 11 Mar 21:43 libmkl_blacs_mpich_lp64.a
lrwxr-xr-x   31 root 29 Mar  3:02 libmkl_blacs_mpich_lp64.dylib -> libmkl_blacs_mpich_lp64.2.dylib
.rw-r--r-- 545k root 11 Mar 20:40 libmkl_blas95_ilp64.a
.rw-r--r-- 546k root 11 Mar 20:39 libmkl_blas95_lp64.a
.rwxr-xr-x 166k root 11 Mar 22:04 libmkl_cdft_core.2.dylib
.rw-r--r-- 187k root 11 Mar 21:44 libmkl_cdft_core.a
lrwxr-xr-x   24 root 29 Mar  3:02 libmkl_cdft_core.dylib -> libmkl_cdft_core.2.dylib
.rwxr-xr-x  69M root 11 Mar 22:04 libmkl_core.2.dylib
.rw-r--r-- 415M root 11 Mar 21:09 libmkl_core.a
lrwxr-xr-x   19 root 29 Mar  3:02 libmkl_core.dylib -> libmkl_core.2.dylib
.rwxr-xr-x  26M root 11 Mar 22:04 libmkl_intel_ilp64.2.dylib
.rw-r--r--  39M root 11 Mar 21:09 libmkl_intel_ilp64.a
lrwxr-xr-x   26 root 29 Mar  3:02 libmkl_intel_ilp64.dylib -> libmkl_intel_ilp64.2.dylib
.rwxr-xr-x  30M root 11 Mar 22:04 libmkl_intel_lp64.2.dylib
.rw-r--r--  44M root 11 Mar 21:09 libmkl_intel_lp64.a
lrwxr-xr-x   25 root 29 Mar  3:02 libmkl_intel_lp64.dylib -> libmkl_intel_lp64.2.dylib
.rwxr-xr-x  68M root 11 Mar 22:04 libmkl_intel_thread.2.dylib
.rw-r--r--  77M root 11 Mar 21:09 libmkl_intel_thread.a
lrwxr-xr-x   27 root 29 Mar  3:02 libmkl_intel_thread.dylib -> libmkl_intel_thread.2.dylib
.rw-r--r-- 6.7M root 11 Mar 20:45 libmkl_lapack95_ilp64.a
.rw-r--r-- 6.6M root 11 Mar 20:42 libmkl_lapack95_lp64.a
.rwxr-xr-x  11M root 11 Mar 22:04 libmkl_rt.2.dylib
lrwxr-xr-x   17 root 29 Mar  3:02 libmkl_rt.dylib -> libmkl_rt.2.dylib
.rwxr-xr-x 7.8M root 11 Mar 22:04 libmkl_scalapack_ilp64.2.dylib
.rw-r--r--  10M root 11 Mar 21:49 libmkl_scalapack_ilp64.a
lrwxr-xr-x   30 root 29 Mar  3:02 libmkl_scalapack_ilp64.dylib -> libmkl_scalapack_ilp64.2.dylib
.rwxr-xr-x 7.7M root 11 Mar 22:04 libmkl_scalapack_lp64.2.dylib
.rw-r--r--  10M root 11 Mar 21:47 libmkl_scalapack_lp64.a
lrwxr-xr-x   29 root 29 Mar  3:02 libmkl_scalapack_lp64.dylib -> libmkl_scalapack_lp64.2.dylib
.rwxr-xr-x  33M root 11 Mar 22:04 libmkl_sequential.2.dylib
.rw-r--r--  34M root 11 Mar 21:09 libmkl_sequential.a
lrwxr-xr-x   25 root 29 Mar  3:02 libmkl_sequential.dylib -> libmkl_sequential.2.dylib
.rwxr-xr-x  44M root 11 Mar 22:04 libmkl_tbb_thread.2.dylib
.rw-r--r--  75M root 11 Mar 21:09 libmkl_tbb_thread.a
lrwxr-xr-x   25 root 29 Mar  3:02 libmkl_tbb_thread.dylib -> libmkl_tbb_thread.2.dylib
.rwxr-xr-x  15M root 11 Mar 22:04 libmkl_vml_avx.2.dylib
.rwxr-xr-x  14M root 11 Mar 22:04 libmkl_vml_avx2.2.dylib
.rwxr-xr-x  14M root 11 Mar 22:04 libmkl_vml_avx512.2.dylib
drwxr-xr-x    - root 13 Jul 18:47 locale
drwxr-xr-x    - root 13 Jul 18:47 pkgconfig

[yerke]

I tried to update build.rs myself by replacing LD_DIR = "DYLD_LIBRARY_PATH" and using

#[cfg(all(target_os = "macos", target_pointer_width = "64"))]
pub const LINK_DIRS: &[&str] = &[
    "compiler/latest/mac/compiler/lib",
    "mkl/latest/lib",
];

But when I try to use mkl-static-iomp feature, I get

% cargo r --example classification
   Compiling dfdx v0.8.0 (/Users/yerke/dev/rust/dfdx)
    Finished dev [unoptimized + debuginfo] target(s) in 12.11s
     Running `target/debug/examples/classification`
dyld[72161]: Library not loaded: @rpath/libiomp5.dylib
  Referenced from: /Users/yerke/dev/rust/dfdx/target/debug/examples/classification
  Reason: tried: '/opt/intel/oneapi/tbb/2021.6.0/env/../lib/libiomp5.dylib' (no such file), '/opt/intel/oneapi/mkl/2022.1.0/lib/libiomp5.dylib' (no such file), '/opt/intel/oneapi/ippcp/2021.6.0/lib/libiomp5.dylib' (no such file), '/opt/intel/oneapi/ipp/2021.6.0/lib/libiomp5.dylib' (no such file), '/opt/intel/oneapi/dnnl/2022.1.0/cpu_iomp/lib/libiomp5.dylib' (no such file), '/opt/intel/oneapi/dal/2021.6.0/lib/libiomp5.dylib' (no such file), '/Users/yerke/dev/rust/dfdx/target/debug/build/bzip2-sys-d1e56fd9059d5e57/out/lib/libiomp5.dylib' (no such file), '/Users/yerke/dev/rust/dfdx/target/debug/build/zstd-sys-71ec340096cb4b20/out/libiomp5.dylib' (no such file), '/Users/yerke/dev/rust/dfdx/target/debug/deps/libiomp5.dylib' (no such file), '/Users/yerke/dev/rust/dfdx/target/debug/libiomp5.dylib' (no such file), '/Users/yerke/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libiomp5.dylib' (no such file), '/Users/yerke/.rustup/toolchains/stable-x86_64-apple-darwin/lib/libiomp5.dylib' (no such file), '/Users/yerke/lib/libiomp5.dylib' (no such file), '/usr/local/lib/libiomp5.dylib' (no such file), '/usr/lib/libiomp5.dylib' (no such file)
zsh: abort      cargo r --example classification

I am not sure how to provide this rpath in dfdx.

[yerke]

I found this issue on rpath that seems to be relevant: rust-lang/cargo#5077

[yerke]

As suggested in rust-lang/cargo#5077 (comment), I tried to add .cargo/config with

[target.x86_64-apple-darwin]
rustflags = ["-C", "link-arg=-Wl,-rpath,/opt/intel/oneapi/compiler/2022.1.0/mac/compiler/lib"]

and it worked!

Let me try to check all other features and make a PR.

[yerke]

Created PR #73.
@joseph-x-li do you want to test it on your machine?

[joseph-x-li]

I just tested all four MKL configurations on my machine and it was successful.

[coreylowman]

Thank so much both of you for your work on this! 🎉 🚀