LLVM error with 'prange' and 'cache=True'

[leohaim]

First let me thank for making available the prange feature again! I have implemented it at several places in my code and it mostly works as expected.
There is one issue, however: It seems that some cached functions are overwritten when prange is used

If I execute the script below, it works for the first time but when I execute it a second time, I get:

LLVM ERROR: Program used external function 'do_scheduling' which could not be resolved!

At other occasions I get an LLVM error that the external function kernel.349734392 could not be resolved.

Leo

from numba import njit,prange
import numpy

@njit(cache=True,parallel=True)
def ncat(a,b,c):
for i in prange(a.shape[0]):
pncat(a,b,c,i)

@njit(cache=True)
def pncat(a,b,c,i):
for j in range(a.shape[0]):
a[j]=0.

@njit(cache=True,parallel=True)
def ncatx(a,b,c):
for i in prange(a.shape[0]):
pncat(a,b,c,i)

@njit(cache=True)
def pncatx(a,b,c,i):
for j in range(a.shape[0]):
a[j]=0.

a=numpy.random.randn(10)
b=a+a
c=a+b
ncat(a,b,c)
ncatx(a, b, c)
print 'finished'

The output of numba --sysinfo on my system follows below:

System info:

Time Stamp
2017-06-28 22:06:54.432007

Hardware Information
Machine : x86_64
CPU Name : sandybridge
CPU Features :
aes avx cmov cx16 mmx pclmul popcnt sse sse2 sse3 sse4.1 sse4.2 ssse3 xsave
xsaveopt

OS Information
Platform : Linux-2.6.32-696.3.1.el6.centos.plus.x86_64-x86_64-with-centos-6.9-Final
Release : 2.6.32-696.3.1.el6.centos.plus.x86_64
System Name : Linux
Version : #1 SMP Wed May 31 19:16:33 UTC 2017
OS specific info : CentOS6.9Final
glibc info : glibc 2.2.5

Python Information
Python Compiler : GCC 4.4.7 20120313 (Red Hat 4.4.7-1)
Python Implementation : CPython
Python Version : 2.7.13
Python Locale : en_US UTF-8

LLVM information
LLVM version : 4.0.0

CUDA Information
Error: CUDA device intialisation problem. Error at driver init:

CUDA driver library cannot be found.
If you are sure that a CUDA driver is installed,
try setting environment variable NUMBAPRO_CUDA_DRIVER
with the file path of the CUDA driver shared library.
:
Error class: <class 'numba.cuda.cudadrv.error.CudaSupportError'>

Conda Information
conda_build_version : 2.1.5
conda_env_version : 4.3.22
platform : linux-64
python_version : 2.7.13.final.0
root_writable : False

Current Conda Env
_license 1.1 py27_1
abstract-rendering 0.5.1 np111py27_0
accelerate_cudalib 2.0 0
alabaster 0.7.9 py27_0
anaconda custom py27_0
anaconda-client 1.6.0 py27_0
anaconda-navigator 1.4.3 py27_0
argcomplete 1.0.0 py27_1
astroid 1.4.9 py27_0
astropy 1.3 np111py27_0
atom 0.3.9 py27_0
babel 2.3.4 py27_0
backports 1.0 py27_0
backports_abc 0.5 py27_0
basemap 1.0.6
basemap 1.0.7 np110py27_0
bcolz 1.0.0 py27_0
beautiful-soup 4.3.2 py27_0
beautifulsoup4 4.5.3 py27_0
binstar 0.11.0 py27_0
bitarray 0.8.1 py27_0
blaze 0.10.1 py27_0
blaze-core 0.9.0 py27_0
blz removed 0
bokeh 0.12.4 py27_0
boost 1.57.0 4
boto 2.45.0 py27_0
bottleneck 1.2.0 np111py27_0
cairo 1.14.8 0
casuarius 1.1 py27_0
cdecimal 2.3 py27_2
certifi 14.05.14 py27_0
cffi 1.9.1 py27_0
chaco 4.5.0 np110py27_0
chardet 2.3.0 py27_0
chest 0.2.3 py27_0
click 6.7 py27_0
cloog 0.18.1 1 serge-sans-paille
cloudpickle 0.2.2 py27_0
clyent 1.2.2 py27_0
colorama 0.3.7 py27_0
conda 4.3.22 py27_0
conda-build 2.1.5 py27_0
conda-env 2.6.0 0
conda-manager 0.3.1 py27_0
conda-verify 2.0.0 py27_0
configobj 5.0.6 py27_0
configparser 3.5.0 py27_0
contextlib2 0.5.4 py27_0
cryptography 1.7.1 py27_0
cubes 1.1 py27_0
cudatoolkit 7.0 1 numba
curl 7.52.1 0
cycler 0.10.0 py27_0
cython 0.25.2 py27_0
cytoolz 0.8.2 py27_0
dask 0.13.0 py27_0
datashape 0.5.4 py27_0
dateutil 2.1 py27_2
dbus 1.10.10 0
decorator 4.0.11 py27_0
dill 0.2.5 py27_0
docutils 0.13.1 py27_0
dynd-python 0.7.2 py27_0
ecmwf-api-client 1.3
enable 4.5.1 np110py27_0
enaml 0.9.8 py27_1
entrypoints 0.2.2 py27_0
enum34 1.1.6 py27_0
et_xmlfile 1.0.1 py27_0
execnet 1.3.0
expat 2.1.0 0
expressions 0.2.3 py27_0
f90nml 0.9.1
fastcache 1.0.2 py27_1
filelock 2.0.7 py27_0
flask 0.12 py27_0
flask-cors 3.0.2 py27_0
fontconfig 2.12.1 2
freetype 2.5.5 2
funcsigs 1.0.2 py27_0
functools32 3.2.3.2 py27_0
future 0.13.1 py27_0
futures 3.0.5 py27_0
gcc_49 4.9.1 6 serge-sans-paille
Genshi 0.7
geos 3.3.3 0
get_terminal_size 1.0.0 py27_0
gevent 1.2.1 py27_0
glib 2.50.2 1
gmp 5.1.2 2
gmpxx 5.1.2 5 serge-sans-paille
grako 3.10.0 py27_1
greenlet 0.4.11 py27_0
grin 1.2.1 py27_3
gst-plugins-base 1.8.0 0
gstreamer 1.8.0 0
h5py 2.6.0 np111py27_2
harfbuzz 0.9.39 2
hdf4 4.2.11 0
hdf5 1.8.17 1
heapdict 1.0.0 py27_1
icu 54.1 0
idna 2.2 py27_0
imagesize 0.7.1 py27_0
iopro 1.7.2 np111py27_p0
ipaddress 1.0.18 py27_0
ipykernel 4.5.2 py27_0
ipython 5.1.0 py27_0
ipython-notebook 4.0.4 py27_0
ipython-qtconsole 4.0.1 py27_0
ipython_genutils 0.1.0 py27_0
ipywidgets 5.2.2 py27_1
isl 0.12.2 0
isort 4.2.5 py27_0
itsdangerous 0.24 py27_0
jbig 2.1 0
jdcal 1.3 py27_0
jedi 0.9.0 py27_1
jinja2 2.9.4 py27_0
jpeg 9b 0
jsonschema 2.5.1 py27_0
jupyter 1.0.0 py27_3
jupyter_client 4.4.0 py27_0
jupyter_console 5.0.0 py27_0
jupyter_core 4.2.1 py27_0
kiwisolver 0.1.3 py27_0
launcher 0.1.5 py27_0
lazy-object-proxy 1.2.2 py27_0
lcms 1.19 0
libdynd 0.7.2 0
libffi 3.2.1 1
libgcc 4.8.5 2
libgfortran 3.0.0 1
libiconv 1.14 0
libpng 1.6.27 0
libsodium 1.0.10 0
libtiff 4.0.6 3
libxcb 1.12 1
libxml2 2.9.4 0
libxslt 1.1.29 0
llvm 3.3 0
llvmlite 0.19.0.dev py27_8 numba
llvmpy 0.12.7 py27_0
locket 0.2.0 py27_1
lxml 3.7.2 py27_0
Magics 2.31.0
markupsafe 0.23 py27_2
matplotlib 2.0.0 np111py27_0
mistune 0.7.3 py27_0
mkl 2017.0.1 0
mkl-rt 11.1 p0 [mkl]
mkl-service 1.1.2 py27_3
mock 1.0.1 py27_0
mpc 1.0.1 0
mpfr 3.1.2 0
mpi4py 2.0.0 py27_2
mpich2 1.4.1p1 0
mpmath 0.19 py27_1
multipledispatch 0.4.9 py27_0
nbconvert 4.2.0 py27_0
nbformat 4.2.0 py27_0
netcdf4 1.2.2 np110py27_0
networkx 1.11 py27_0
nltk 3.2.2 py27_0
nose 1.3.7 py27_1
notebook 4.3.1 py27_0
numba 0.34.0.dev np111py27_505 numba
numexpr 2.6.1 np111py27_2
numpy 1.11.3 py27_0
numpydoc 0.6.0 py27_0
odo 0.5.0 py27_1
openblas 0.2.14 4
openpyxl 2.4.1 py27_0
openssl 1.0.2k 0
pandas 0.19.2 np111py27_1
partd 0.3.7 py27_0
patchelf 0.8 0
path.py 10.0 py27_0
pathlib2 2.2.0 py27_0
patsy 0.4.1 py27_0
pcre 8.39 1
pep8 1.7.0 py27_0
pexpect 4.2.1 py27_0
pickleshare 0.7.4 py27_0
pillow 4.0.0 py27_0
pip 9.0.1 py27_1
pixman 0.34.0 0
pkginfo 1.4.1 py27_0
ply 3.9 py27_0
prompt_toolkit 1.0.9 py27_0
psutil 5.0.1 py27_0
ptyprocess 0.5.1 py27_0
py 1.4.32 py27_0
pyasn1 0.1.9 py27_0
pycairo 1.10.0 py27_0
pycosat 0.6.1 py27_1
pycountry 16.11.8
pycparser 2.17 py27_0
pycrypto 2.6.1 py27_4
pycurl 7.43.0 py27_2
pyface 4.5.2 py27_0
pyflakes 1.5.0 py27_0
pygments 2.1.3 py27_0
pygrib 1.9.9
pylint 1.6.4 py27_1
pymysql 0.6.6 py27_0
PyNIO 1.4.1
pyopenssl 16.2.0 py27_0
pyparsing 2.1.4 py27_0
pyqt 5.6.0 py27_2
pyshp 1.2.0
pytables 3.3.0 np111py27_0
pytest 3.0.5 py27_0
pytest-xdist 1.11
python 2.7.13 0
python-dateutil 2.6.0 py27_0
pythran 0.6.0
pythran 0.7.2 0 serge-sans-paille
pytz 2016.10 py27_0
pyyaml 3.12 py27_0
pyzmq 16.0.2 py27_0
qt 5.6.2 3
qtawesome 0.4.3 py27_0
qtconsole 4.2.1 py27_1
qtpy 1.2.1 py27_0
readline 6.2 2
redis 3.2.0 0
redis-py 2.10.5 py27_0
requests 2.12.4 py27_0
rope 0.9.4 py27_1
ruamel_yaml 0.11.14 py27_0
runipy 0.1.3 py27_0
scandir 1.4 py27_0
scikit-image 0.12.3 np111py27_1
scikit-learn 0.18.1 np111py27_1
scipy 0.18.1 np111py27_1
seaborn 0.7.1 py27_0
setuptools 27.2.0 py27_0
simplegeneric 0.8.1 py27_1
singledispatch 3.4.0.3 py27_0 numba
sip 4.18 py27_0
six 1.10.0 py27_0
snowballstemmer 1.2.1 py27_0
sockjs-tornado 1.0.3 py27_0
sphinx 1.5.1 py27_0
sphinx_rtd_theme 0.1.9 py27_0
spyder 3.1.2 py27_0
spyder-app 2.3.8 py27_0
sqlalchemy 1.1.5 py27_0
sqlite 3.13.0 0
ssl_match_hostname 3.4.0.2 py27_1
statsmodels 0.6.1 np111py27_1
subprocess32 3.2.7 py27_0
sympy 1.0 py27_0
system 5.8 2 https://conda.binstar.org/numba
terminado 0.6 py27_0
theano 0.8.2 py27_0
tk 8.5.18 0
toolz 0.8.2 py27_0
tornado 4.4.2 py27_0
traitlets 4.3.1 py27_0
traits 4.5.0 py27_0
traitsui 4.5.1 py27_0
ujson 1.33 py27_0
unicodecsv 0.14.1 py27_0
unixodbc 2.3.1 1
util-linux 2.21 0
wcwidth 0.1.7 py27_0
werkzeug 0.11.15 py27_0
wheel 0.29.0 py27_0
widgetsnbextension 1.2.6 py27_0
wrapt 1.10.8 py27_0
xlrd 1.0.0 py27_0
xlsxwriter 0.9.6 py27_0
xlwt 1.2.0 py27_0
xz 5.2.2 1
yaml 0.1.6 0
zeromq 4.1.5 0
zlib 1.2.8 3

If requested, please copy and paste the information between
the dashed (----) lines, or from a given specific section as
appropriate.

=============================================================
IMPORTANT: Please ensure that you are happy with sharing the
contents of the information present, any information that you
wish to keep private you should remove before sharing.

[stuartarchibald]

Thanks for the report and also for being an early adopter/tester. I can replicate what you see.

I think this is down to the way parfors has to loop back into the compiler during lowering to create temporary gufuncs kernels which are not cached (hence the external function .kernel.<random_number> could not be resolved). The other problem about missing do_scheduling is down to the parallel functions needing that symbol from the gufunc_scheduler library which does the thread scheduling and wouldn't necessarily be in scope when loading from disk.

Will have to have a think RE resolution. I did have a small hack at the problem, it seems quite complicated!

[stuartarchibald]

@leohaim as a temporary measure #2448 makes it such that users cannot get into this situation. If the combination parallel=True, cache=True is used, Numba now warns the user this is not supported and continues with the cache set to false. Thanks again for reporting this and trying it out. I shan't close this ticket until a proper fix is in.

[stuartarchibald]

This is a general caching problem that's still present. Also the raised warning message makes no sense in this context, e.g.:

NumbaWarning: Cannot cache compiled function "ncatx" as it uses dynamic globals (such as ctypes pointers and large global arrays)

[stuartarchibald]

Cannot replicate in 0.45. Caching improvements were made in #4155