Warning and slowdown for multithreaded Julia code

[kpamnany]

Running our application (Celeste) with 8 threads or more produces the following warning (5 of them, sometimes 4):

WARNING: could not attach metadata for @simd loop.

After which performance gets completely hosed. This message doesn't appear (and performance doesn't get trashed) with 4 threads or less.

Searches for this issue suggest this has something to do with inlining functions, but most of what I found related to package coverage testing which is not what I'm doing.

Is there some issue with threads and this loop metadata?

[kpamnany]

Any suggestions? @vtjnash, @yuyichao, @JeffBezanson?

[tkelman]

what commit are you on? consequence of #15717?

[kpamnany]

Don't think so?

julia> versioninfo()
Julia Version 0.5.0-dev+3108
Commit d72842a* (2016-03-11 20:19 UTC)
Platform Info:
  System: Linux (x86_64-suse-linux)
  CPU: Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, haswell)

[yuyichao]

From the version it seems that this should be affected by #15717 although that won't print a warning like this...

I'm not really familiar with the threading front end but it may be helpful to see the typed ast (@code_typed) of the function for 4 threads and 8 threads.

[kpamnany]

I'll confirm this, but there should be no difference in the AST -- the loop iteration space is divided by the number of threads given by Base.Threads.nthreads() and that is dynamically determined (by the environment variable JULIA_NUM_THREADS).

[kpamnany]

It isn't just with 8 threads; doing a multi-process run with MPI, each process having 4 threads, I saw this warning show up on one of the processes. :(

[JeffBezanson]

It's likely this is fixed now. If not, the workaround to try is to declare the types of variables used both inside and outside a @threads loop. Please confirm when you get a chance.

[kpamnany]

This seems to be gone.