Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Change the list of MKL libraries we redistribute #2126

Closed
droberts195 opened this issue Nov 10, 2021 · 0 comments · Fixed by #2238
Closed

[ML] Change the list of MKL libraries we redistribute #2126

droberts195 opened this issue Nov 10, 2021 · 0 comments · Fixed by #2238
Labels
:ml

Comments

@droberts195
Copy link
Contributor

In our build setup instructions we specify the MKL libraries that we will redistribute, for example on Linux:

ml-cpp/build-setup/linux.md

Lines 280 to 285 in e262592

sudo cp /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so /usr/local/gcc103/lib
sudo cp /opt/intel/mkl/lib/intel64/libmkl_core.so /usr/local/gcc103/lib
sudo cp /opt/intel/mkl/lib/intel64/libmkl_def.so /usr/local/gcc103/lib
sudo cp /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so /usr/local/gcc103/lib
sudo cp /opt/intel/mkl/lib/intel64/libmkl_avx*.so /usr/local/gcc103/lib
sudo cp /opt/intel/mkl/lib/intel64/libmkl_vml*.so /usr/local/gcc103/lib

Based on the information in the "Computational Layer" section of the table in http://portal.nacad.ufrj.br/online/intel/mkl/common/mkl_userguide/GUID-C823F752-DDA3-4EFB-B673-222C2720FAFA.htm, the subset of libraries we are shipping is not quite correct.

We claim that our code will run on processors that support at minimum SSE4.2 instructions. This means that we should be shipping libmkl_mc3.so in addition to the libraries we are currently shipping. Additionally, since we won't run on processors that support a maximum of SSSE3 nor Hi-k Core 2 processors (from 2009), we don't need to ship libmkl_vml_mc.so nor libmkl_vml_mc2.so.

We should correct this next time we upgrade PyTorch and have to revisit the PyTorch dependencies.
droberts195 added a commit to droberts195/ml-cpp that referenced this issue Mar 22, 2022
@droberts195
[ML] Upgrade to PyTorch 1.11 on Linux
a8bf300 
Together with elastic#2233 and elastic#2235 this completes the upgrade to
PyTorch 1.11 for all platforms.

A small change to the way things are done is that we now copy
all MKL libraries into the Docker images, then copy the required
subset into our release distribution. This means we can change
the subset of MKL libraries that we redistribute without having
to rebuild our Docker images.

Fixes elastic#2126
@droberts195 droberts195 mentioned this issue Mar 22, 2022
Merged
droberts195 added a commit that referenced this issue Mar 30, 2022
@droberts195
[ML] Upgrade to PyTorch 1.11 on Linux (#2238)
c7c3683 
Together with #2233 and #2235 this completes the upgrade to
PyTorch 1.11 for all platforms.

A small change to the way things are done is that we now copy
all MKL libraries into the Docker images, then copy the required
subset into our release distribution. This means we can change
the subset of MKL libraries that we redistribute without having
to rebuild our Docker images.

Fixes #2126
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:ml
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] Upgrade to PyTorch 1.11 on Linux droberts195/ml-cpp
1 participant
@droberts195